Hey - yeah, I've been going back in time here on the forums for this one, and there are a few threads that touch on the question. Back in 2009 there was a bit of a discussion about it (https://forum.atlas-games.com/t/new-herbam-and-other-spells/157/1), where the conclusion seemed to be that they DO penalize each other (aside from Favored Abilities, which may or may not penalize anything but themselves, depending on the realm), but by 2014 your position seems to have solidified to the statement in the OP.
Thanks for the update!