While not able to completely remove all of your staff, going with a form of Arthur's suggestion can significantly reduce the total you need. The cost savings from Enchanted Items can directly reduce the need for Craftsmen, which indirectly reduces the need for Servants, Laborers, and Teamsters. Not taking any Specialist will have a similar reduction. You can further reduce things by going with a minimal Turb (say only Shield Grogs and a couple of guards at minimum).
What you cannot reduce is the number of VIPs (Magi, Companions) and their required Servants/Teamsters. Every Magi requires a Servant unless you want a "Pauper" reputation and you will need a lesser number for the Companions (60%). Any Turb you have will require 1 Servant for every 10. You are also going to require some mix of Laborers and Teamsters (somewhere around ~60% of the number of Servants at minimum).
So say you have 5 Magi and 5 Companions, with 5 Shield Grogs and 5 Guards. You would also need 10 Servants and some mix of Laborers/Teamsters. Lowest would be 6 Teamsters. That would put you at 36 total people, which is a pretty damn small Covenant.
My Covenant isn't a good example because we actually have more people than we need and there are a lot (592 adults, 249 babies/children). However we are way larger than we need to be, since we have a lot of cost savings from enchanted items that are not being used. 19.25 ppy (pounds per year) of Provisions cost savings, 99 ppy of Consumables cost savings, and 34 ppy of Building cost savings from enchanted devices are not being used.
[Side Note: If you are wondering what these are, they are all in my "A Covenant's Collection of Enchanted Items" thread]
All the unused cost savings could allow us to reduce our population by a great deal. In fact our unused savings from enchanted items are almost equal in Points of Inhabitants to our actual total Points of Inhabitants. If we doubled our total population the only cost would come from Wages and the shortfall of Provisions.
Going the opposite way, we could cut 100% of our Consumable Craftsmen. 71% of our Building Craftsmen, and 27.5% of our Provision Craftsmen from our wasted cost savings. That would have further knock on effects of reducing Wages, needed Servants, needed Laborers, and the actual maintenance cost of those three (Consumables, Buildings, Provisions) by a lot.
Most likely we could half our total population and still support all the combat forces, Specialist, Companions, and Magi. If we dropped 90% of our combat forces so we only had Shield Grogs and a couple of guards then we could most likely drop our total population to ~20% of what it currently is, roughly a one to one ratio of VIP/Combat to Staff. If we dropped most of our Specialist (the ~80% we use for book copying, teaching children, and medical) then we could get down to ~10% of our current population.
While we would not do this since we like having the spare capacity, someone wanting to go the other way could get by with very little support people for their VIPs (Magi, Companions, key Specialist) and combat forces.