On re-reading, it appears you are looking for more practical and detailed advice. I don't really know of any place where low-magic high-Myth Ars Magica is discussed at length. Here are my thoughts, for what it's worth:
As I said, I see three major problems here.
(a) The power level of starting magi is too high. The starting Flambeau character can whip out a Ball of Abysmal Flame with approximately 11 penetration, and with just a little bit of experience (or just better design) can raise that to 2-3 per round (with Spell Mastery) and be all-but-immune to mundane attacks (with some protective magics). This a very splashy, nigh-unstoppable, killing machine in a low-magic (even if high-Myth) setting.
(b) Magi progress too fast. The canon (see the book Coveanants) implies source Qualities of about 10, with with just 2 seasons of study per year would yield 200 XP in a decade; put these in two Arts initially at 10, add two Affinities, and you reach Art scores of 19 (205 XP each). Five decades out of gauntlet, you have two Arts at 39 (805), nearly breaking the Later Life advancement assumptions on what is possible in the setting, and fully 12 decades post gauntlet you have two Arts at 60 (1855 XP), shattering said assumptions to smitherings. Now add in a Magic Focus, a few lab assistants, a specialized lab, and other perks and you easily get Lab Totals exceeding level 200 and spell and effect levels exceeding level 100, which is enough to do just about anything to entire kingdoms, continents, and oceans. This isn't such a big problem for NPCs, since you can just ignore it and keep them to lower power levels. But denying your PCs options to study is just not fun, and eventually they WILL start accumulating lots of XP and high Art scores - perhaps not as high, but high. I believe the only reason that this isn't a bigger problem for the Ars fan community is that most sagas aren't that long (in-game).
(c) The Magic Resistance of powerful creatures is a joke. This is partly because magi just have high Art scores and can be content with relatively low-level spells, and partly because magi can seriously boost their penetration with Arcane & Sympathetic Connections, raw vis, and Wizard's Communion. Another way to look at it is that MR scores are just too low. At the end, it's just not very Mythic to have the great Dragon serve as this-month's-hunt-quarry for your local Flambeau club lodge.
The best thing to do is to go over the rules and especially the spell guidelines, and "massage" them until you get what you want. But that's way too much work. So here are a couple of ideas.
A related issue is the availability of raw vis and, for that matter, books. I'm not keen on not allowing players to get what they want. If their character wants to study, I'm generally in favor of him finding books or raw vis to study from. It may take some adventuring, but a single adventure should suffice for seasons if not years of rewards (study time). I often won't even require adventures at all, allowing instead book and vis trade; there are plenty of other reasons to go out adventuring, reasons more personal and meaningful than needing yet-another-tractatus on Creo. This is a gamestyle decision, and if you want to limit book trade and vis sources and trade that is up to you. Be aware that limiting vis sources makes the Order more competitive and generally means that magi will hardly ever boost spells, and will be skimpy on casting rituals and making magic items, which means they will want to spend even more time reading books and inventing spells. Suppressing book trade works best in an insular, suspicious Order (that I personally don't like), and I'd strongly recommend insisting that books on magic are really minor magic items and as such can only be produced (written or copied) by magi. This is more colorful, too, and can apply to Lab Texts to give your magi even more things to do with their time that except inventing spells or studying Arts. If both raw vis and books are hard to find, progress can grind down to the point where the fast advancement of magi is not really a concern; I just find that such severe limitations are not reasonable and make the setting one of deprivation and poverty. It certainly takes it away from the society of semi-academics that you imagine.
Double the magic resistance of creatures. No, triple it. That would take care of all that high-penetration these Hermetic wizards are packing. To maintain the setting's integrity, you may want to only triple it against Hermetic magic - with high power, comes his resistance... apparently...
One important resource to get a Mythic theme is Realms of Power: Magic. It has rules and ideas on how raw vis can be used in Mythic ways - from raw vis that also has magical abilities, to using Magic Lore to figure out elaborate rituals that turn particular sources of it into magic items of sort. It also has rules on aligned-auras (a place aligned not just to Magic in general, but to, say, Fire) and on changing aura strength which I feel are very appropriate for a low-Magic high-Mythic setting. I think that such a saga can really benefit from this book. Of course, the above mentioned books (City and Guild, Art & Academe and Lords of Men) are perhaps even more useful, but would largely be relevant to companion characters and background in my opinion. RoPM talks about raw vis and auras, things magi interact with, so is more directly relevant.
Another thing you might want to consider is scrapping Ars Magica's magic system and going instead all-hedgie - do a saga on hedge wizards, or have the Order made up of hedge wizards. This feels much more Mythic and certainly low-magic, but sucks because Ars Magica's magic system is great and there is no good hedge wizard magic system. (Some are presented in Hedge Magic, such as the Learned Magicians or Folk Witches, but I found them unsatisfying.) You can always do a saga about companions, perhaps with some magical abilities (the various Realms of Power: X books would greatly expand on your options here), but again - this is no Ars Magica. So the only way to move forward is to lower the power level of the Hermetic magic system, which leads me to...
Arts as Abilities
A simple solution I am infamous for advocating is to have Arts cost as much XP as Abilities. This is a major house rule, but I think such a change is needed to make the type of change you seem to be after.
This cuts down the insanity, but won't eliminate it completely. Under this house rule, starting magi will tend to have lower level spells (the Flambeau will rely on Pilum of Fire, not Ball of Abysmal Flame), and as a result will also have more of them (which I consider a boon). Accepting this lower power-level will require some adaptation from experienced Ars Magica players, but the resulting magi are still powerful enough. In particular, spontaneous magic is still useful, given some ingenuity, and longevity still effective. Even under this house rule, ancient magi would still be ridiculously powerful; 12 decades post-gauntlet, the above assumptions still lead to two Arts at 26, which is enough to do some seriously setting-upsetting magic.
Another effect of this house rule is that small bonuses become more prominent, so magi will tend more to pursue them - especially for initial characters, but this can set the tone of the character so it's important later on as well. Players will use their talismans and tend to milk more out of virtues such as Cyclic Magic or Special Circumstances, and of course aura. I would also recommend supplementing this house rule with further house rules that promote such small bonuses that you want - rewarding things that will give the right "feel" to how spells are supposed to be cast. This can include free access to Potent Magic (see The Mysteries Revised Edition; it's in Houses of Hermes: Mystery Cults too, if I remember correctly), which would encourage the use of casting-items when casting spells - if you want your magi to use sympathetic-materials when casting spells, which has a nicer low-magic feel to it. You can also consider encouraging Ceremonial Magic, which feels more "mythic", especially if described as elaborate ceremonies; this can be done by adopting the Ceremonial Casting spell mastery ability (first described in Houses of Hermes: True Lineages, but also reprinted elsewhere), or the further rules on making ceremonial casting easier in Houses of Hermes: Societates. If you're into this sort of thing, you can institute an astrology-system to provide appropriate astrological bonuses at certain times; The Mysteries Revised Edition can help with that. Or, of course, you can just leave things as they are if you want magi to just will and speak their magic into being.
There are three principal ways to boost spell rolls in the core rules - arcane and sympathetic connections, raw vis, and wizard's communion. All are colorful in their way, and can contribute to that high-Mythic feel. Under this house rule I would suggest eliminating the penetration multiplier bonus provided by an arcane connection, or at least keeping it at "+1" even for fixed arcane connections, so as to keep the bonuses lower (in-line with the lower Art totals). The other two methods are tied to the Art and spell levels and as such are self-correcting; indeed, I think they even work better under the Arts as Abilities variant.
A major criticism against this house rule is that it means that Magic Theory becomes even more valuable than it already is. Just about any magus will invest lots of XP in raising it, often keeping it above or near his highest Art. I'm not sure if this is really that unfortunate, but do keep in mind that this means magi will be able to enchant complicated magic items and that the power level would be somewhat higher than what you might naturally consider under this house rule (as high Magic Theory scores will increase spell levels and so on). Perhaps most importantly, specialization will be less rewarding under this rule.
If you do decide to adopt this house rule, you should also make a few other changes to fit the rest of the rules to it.
- The score needed to train an apprentice is 2 (15 XP) in each Art, instead of the RAW 5 (15 XP) in each Art.
- You can write down a number of tractatus on an Art equal to the Art/2, instead of the RAW Art/5.
- Puissant Art grants a +2 bonus, instead of the RAW +3 bonus.
- Maximum Art scores at character generation follow the guidelines for maximum Ability scores.
- The highest Arts in the Order's history are assumed to be around 20, so maximum summa level is 10.
- Summa quality is still increased by +1 per dropped level for Arts, just like in RAW. (Otherwise, the quality gets too high.) Summa maximum quality is hence 11+(10-Level) [this provides a natural maximum of 20, below the 22 absolute maximum].
- The build point worth of summa on Arts are the same as those for Abilities (because of their lower Levels).