What does it take to establish a new House?

Indeed so. I can only imagine a new house being formed if some epoch-making event causes a different magical tradition joining the Order of Hermes and their magic to be integrated into Hermetic magic, i.e. a breakthrough. I don't see traditions within a house wanting to break away from each other, e.g. Magi Trianomae and Magi Bonisagi, because it hasn't happened since Pralix broke away from the Order to found the Ordo Ex Miscellenea. Likewise, the lineage of Apromor still consider themselves to be followers of Flambeau even though they specialise in Perdo rather than Ignem. In some cases I can see magi of a particular lineage joining another house, as Myanar and his supporters joined House Bjornaer after being cast out of House Merinita, but not starting a new house.