I've many thoughts on this, as I've played Ars magica for over 20 years since a teen in the 90s.
Simplify, Simplify, and Simplify. The game has far too many fiddly bits. It will always be a high crunch game, yet it can still do with more less. This is a huge barrier to entry. For those who complain, frankly, you have 5th edition.
Fundamental principle: Remove multiplication and division as much as possible.
i. Abilities and Arts using two different scales. Pick one.
ii. Advancement times need to be less fiddly.
iii. Spell Invention Time = Spell Level / (Lab Total - Spell Level) - this is awful to explain to people not great at arithmetic.
iv. Magic Resistance and Lab Totals are a vast array of formulas.
What is certaman, and does it do, vs what should it actually do? Does Certamen work?
What is the purpose of virtues and flaws? Some games have these as optional.
what is the point of having virtues and flaws balance at character creation? What is the purpose of this that is of a long term benefit to the game? Am I the only one with 5 players who finds that maybe one or two of a PC's multiple flaws comes up on a regular basis - and that keeping track of two dozen flaws with is a bit much of a time sink?
Vis. 15 different kinds is a MAJOR accounting headache. Have one. Have the flavors of vis matter "If your troupe decides to" -
This reminds, throw in some more OPTIONAL rules on how to approach things.
Troupe style - this is one of the most difficult things about the game. It's one of the most innovative ways to approach the linear fighters quadratic wizards problem in D&D, and it's just sort of there... But not much is done with it one way or another.
Done wrong, this is a MAJOR barrier to entry. You have new groups thinking that they have to make a Magus, Companion, and 3 grogs BEFORE THEY EVEN PLAY. They do this for two weekends and give up in disgust.
Setting:
This is something I've not seen touched on much (admittedly, I have not read all 246 replies). The setting needs to vastly handle better how magi are integrated, OR NOT, into Mythic Europe.
0. The magic is too strong, RAW, in 5th ed for the OoH not to have changed history already.
There needs to be some serious historical details ironed out, such as the Catholic and Orthodox church. I've heard sources that the Catholic church charged people with the crime of pretending to be witch, because magic isn't actually real. That shit needs to be handled, and handled well. If some subtle/major changes need to made to the catholic church, then for god's sake lets be upfront about that.
Many people have modeled the growth of the OoH and find it's actually too small by 1220 and the Schism war too minor to have done enough population damage.
Get rid of some of the houses. They're boring and have silly names.
Decide how much interference is allowed. By the RAW 5th ed, history should have been radically changed much earlier from interefering magi.
Roll the clock back and pick a different year to base things in. The 800s is too early, but say, 1050, after the schism war when things have been shattered, and is an era of rebuilding. Whole new supplements can be written because 1050 was quite different than 1220. It's before the Norman invasion of England, it's before Manzikurt shattered Byzantium.
Change the history, why worship what's already been written? Maybe the schism war DID create a true schism, and now there's two rival organizations, separated perhaps by geography (east vs west?) Who have different legal approaches and ways of organizing themselves.
I mean ultimately, the game needs a much stronger, MUCH bolder new vision of where it's going to go. Some tepid changes around the edges are just a waste of time, it's not like we're talking about a new edition or anything!