Teams have both implicit and explicit working agreements. The ones the team does without thinking, and the ones that they've agreed on and written up. Explicit agreements can become implicit over time as the team internalizes those agreements.
What makes us think that adding or removing people from that dynamic won't affect the team?
Whether they are implicit or explicit, working agreements can be invalidated if the new team membership don't all agree. The entire team is meant to agree and not impose their rules on the new membership. The team has to find it's own path again.