First of all the exactly reach of each tribunal is not defined by borders but by covenants. Wherever a tribunal's covenants claimed areas cover is the area of the covenant. So they can change through time if for example two neighbor covenants from different tribunals win and loose vis sources.
Vis and magic auras are obvious and sufficient reasons, I think.
Forming a new tribunal should be extremely hard. The minimum requisite ruled by Grand Tribunal are that there must be at least 4 covenants with 12 magi. Then you should go to the Grand Tribunal and ask to get your new tribunal status. But even to go there is hard, as only the primi of each houses and three representatives from already existing covenants are given invitations to attend. Expect all kind of opposition from neighboring tribunals, which will probably want all these vis sources for them. Far away tribunals may be interested. Tribunals with internal secessionist covenants will also strongly oppose. Think that since the hermetic conquest of the British Islands after Dav'nalleous defeat, only one new tribunal had ben recognized, Novgorod, and it was stablished right after the Schism War, when nobody was looking, so to speak, and with the strong opposition of the Rhine Tribunal.
As for how an existing tribunal may pressure what it may see as a rogue covenant not behaving, also consider that existing covenants are organized and powerful, and have a lot of magi. You could expect a lot of simultaneous Wizards' Wars to claim what you discovered or obtained.
There was a topic some months ago discussing a bit in depth about that, as its name suggested, Creating a New Tribunal, and also in Through the Aegis there is mentioned that the Northern Seas covenant's founder is trying to set up a new tribunal over the northern seas, and some plans on how to do so in the long term.