Not sure what the difference between Empire nobility and other nobilities but it is not that simple, I am probably completely wrong but here goes:
A Count does not necessarily have any barons sworn allegiance to him, he might just have a large number of knights. Take the example Count in the Triamore set, he has allegiance from a dozen knights over as many fiefs, but also a hermitage of the templars who own a single fief and an abbey who own three, as well as receiving some tribute from the covenant in that setting who own three fiefs.
A knight does not necessarily have only one manor, a rich knight might have three or four. You can have a manor that you do not directly control run by a servant managing it for you without ownership given away, the name for that is a senechal. If you give the manor to a knight instead then that knight gets to take the taxes for himself, and merely tithes to the church its share and gives the higher noble tribute.
An Empire noble who would be a Count in sheer terms of land owned could be only a Baron in one Duchy and a Baron in another, and merely a knight in the third. And would give allegiance to each lord seperately for the land held. even though each of those Counts or Earl all owe allegiance to the Emperor. This comes about from marriages between nobles who live far off. It was also how the Netherlands was formed internally to the Holy Roman Empire as the Hapsburg family married into more and more titles and land, their head was Emperor, as well as being seperately King of Hungary, King of Spain, Count of Flanders, Graf of something else, etc.
An example of this from England, which you know most about, is the King of England also holding Normandy from France and swearing allegiance to the King of France for those holdings only. The higher the noble the more convoluted the web of mutually idiotic alliegances, where a noble in two countries has to decide which side in a war he fights for.
One of the reasons for the Netherlands eventual rebellion is that each fief he inherited had different laws and a different set of agreements with its Lord, which was ignored by the him.
It is likely any high noble in the Empire with fiefs held seperately has the same problem, he might have the right to impose taxes as he wishes, conscript troops at will, impose tariffs etc, in one domain. But in another it could be highly curtailed and he could make no decisions without confirmation from the head burgers of the city. Remember that not all cities get to be independant, they must have a licence to be so, from their overlord.