Okay, then I'm not suffering juvenile senility (good!)
A "tower" of any sort is generally seen as a military installation - defensive if not offensive. It says "ya can't tax me if I don't wanna!" to a noble, even if that's not the intent.
But the same spell (more or less) could create a manor house or other similar effect. Of course, even better is underground - find some bedrock, and a small PeTe (or even ReTe, if you have a use for the roughcut stone) will dig out a magi warren in a few weeks, and without the vis expenditure, and with a formulaic spell that can be used for other things ("dig a pit" with a level of complexity for control and another for volume.)
As for Specialists, nothing says they have to live with - you could hire them, set them up in their own workshop as "patroned" artisans somewhere, and they do all their work for you, even if not living with you. A valid option.
Okay, so a "minimum footprint" covenant.
I'm going to assume the magi don't want to spend any seasons on work a mundane could do - ever. And I'll take some cues from your list.
By the Covenants book, you need 1 servant/magi* just to keep them living as they are accustomed - fed, rooms cleaned, labs swept, etc. Call this...
Cook
Maid
Scullery (misc)
(* This is not how it's phrased - a mage is "5 pts", and you need 2 servants/10 pts.)
I think a Scribe is too important to exclude, so that, and he does double-duty as the librarian. Unless you have a HUGE lib', maybe one season a year (probably less) he does conservation on your library, and it's all good.
Scribe/librarian
(Note that he won't be binding books or making parchement or illuminating, not unless you're very lucky in finding a multi-skilled scribe. Most "craftsmen" saw no need to be multi-skilled - they learned one thing, they did one thing, they did it well, that's what they did, end of discussion.)
1 laborer - a strong back and weak mind. Tote that thing, lift that whatever. (Technically optional, but it feels right.)
a stableboy, only because you implied you needed one.
1 more servant for the above
grogs - as few as you can stomach.
& 1 more servant for those, & maybe 1 more laborer. (You don't "need" laborers, but they save money and just make sense. Someone to cut wood, lift heavy loads, carry stores, dig a ditch, shovel the shit (literally, ahem), that stuff. You're either keeping them on staff or hiring them regularly from nearby - might as well keep them under thumb.)
And that's it.
Total:
1 specialist
6 servants
2 laborers (technically optional)
the grogs
As for any "steward", your total "household" is now only 1 specialist, 8 "service types" and the grogs. Find a cook who can be "large and in charge", trust them with the majority of the provisioning and overseeing the household staff. Pick one grog to be the sergeant, and trust them with the arming/armoring (or splurge and get a turb captain). And trust the librarian to handle the parchments/inks and correspondence. (And by "trust" I mean "liberal use of Pose the Silent Question" & etc.)
So that's now a footprint of 9+ the grogs, or maybe a tight 6+ without the laborers. Not what I'd call an "typical" covenant, but certainly a functional one.
That better?
(For truly minimal, you dump everyone but 1 servant/mage. That's all that's "needed", but then you're spending many, many pounds/year on extra per-job hires who may vary from week to week.)