Splitting knowledge out doesn't mean you need to drop weights. Just that it needs to become independent / attachable.

To some extent we've already seen this with MoE and Frankenstein models.