The best fix I've found is to make it actually like duelling. That is, it's about reputation. You can refuse if you like, but personal honour may demand you fight even though you are almost certsain to lose.
The points about duels of honor that are important are:
It's treated as absolutely serious by everyone who doesn't treat them as absolutely stupid. There's no middle ground on what you think about pistols at dawn. Sports dueling with swords, sure...some people in the "duelling rocks" camp like it, but generally people who fatally duel don't do it for fun, they do it because they'd really like to carve someone up.
People don't certamen over things where flipping a coin would do. People don't certamen their close friends or allies to actually decide things. Certamen is hitting people in the face with a glove, or possibly a sport, but it's not anything in between.
Certamen is effectively a mutually declared wizards war, but under the polite rule that people are not going to cripple each other afterwards. If you lose, you are socially treated as not being dead only because the person couldn't be bothered to have a war with you. People who use certamen lightly are encouraging wizards wars, and those sorts of reckless tools need to have their heads stomped on, Scaramouche-style. Mutually declared is important. This is you and your rival agreeing not to try and kill each other for real. Yes, certamen allows the guy who might have put a lightning bolt through your tower to bully you, but only to the extent of your personal embarrasment, and he's probably even going to tell you in advance that he'll chanllenge you unless you do X, so you have a chance to get out of it. You'd be a fool to go to certamen with an evenly matched young magus except on ther most vital of issues, because if you lose, afterward you are kind of his puppy.
Certamen is stressful and random, so even a master duelist going into it needs to be aware that there's a chance that a lucky shot will take him out.
If you are offered ceertamen and refuse, and continue to annoy a powerful magician, in a really serious way, you are leaving him only fatal force, or harrassment under cover of WAr, as an option. He'll give you a month to get your affairs in order, but certamen really is the gentlemanly way of telling younger magi not to stand on your lawn.
I'm not sure that this is how Tremere took over the Order, or at least the only tool he employed. As Timothy points out Certamen is a duel, and is therefore, by no means is the challenger assured victory. Using Certamen as a linchpin strategy to take over the Order and then the world seems fraught with difficulty and undue risk.
I think you do hit the nail on the head, though, that Certamen contests are boring to those not involved in them (as would any duel such as the Dimicatio contest). The key is to provide the other PCs not involved in Certamen something that keeps them engaged in what is going on. Perhaps the NPC Certamen contestant is employing some kind of dirty tricks, accomplices in the crowed designed to break the PC's concentration. You might need to switch off between the Certamen duel and whatever is going on around it, but it is one way I can think of to bring all the PCs into the actual duel.
In the games I've had, certamen was engaged a decent amount early on, often 'for love' as the Tremere put it. The PCs were new to the game and were flexing their magical muscles. There were also plenty of similarly young NPC magi who had the same attitudes, and at one point one of the PCs did try to dominate the Covenant Charter through Certamen, but that fell apart when he was bested. It was a decent gambit on both sides. The aggressor was a specialist in a technique and form, so he was likely to always have a good score in the fight, but he was too prideful. He was beaten after he accepted a certamen challenge right after another and was still fatigued, but was goaded into accepting it rather than resting first.
But I see it as something that more of the young magi do fresh out of apprenticeship, just aching to cut loose a bit. The older magi see the long term ramifications of it easier. And Tribunals like the Rhine actively work to eliminate frivilous Certamens. I often based my sagas in Stonehenge, which IMO has a bit of a frontier mentality, so Certamen was a bit more common, with the powerful wizards having an uneasy detente with each other, and squashing upstarts that tried to challenge them. But as was pointed out, anything can happen.
For others finding it boring, the event can actually go pretty quick, but that takes some of the fun out of it. I encourage the participants to vividly describe their illusions and offer bonuses if they are particularly good. The other players will often place bets and kabitz. But if it's happening a lot it will get boring. It's better to have it for more important issues and then try to drive up the drama.
I often like to consider customs of the Order as the antecedents for mundane developments: Magi did it first and "good" ideas spread. So too with Certamen developing into a code of honorable dueling as a substitute for uncontrolled vengeance, retaliation and feuding.
Some rule changes to support this:
a) Each magus chooses his own TeFo. Yes, gaming for a shared TeFo is a cute mini-game but in no way reflects the relative abilities of wizards in a real conflict. (If you want a minigame, maybe a big 2-D matrix of TeFo vs TeFo advantages and disadvantages, and another rule allowing each magus to select a different TeFo each round, so that over an extended conflict a magus can adapt to his opponent while revealing his own relative capabilities.)
b) Instead of inflicting fatigue, a magus can choose to inflict a wound, soaked as usual. In most Certamen, magi agree to only inflict fatigue, but Certamen to first blood or even to the death becomes possible. (This leads to things like a possible Transylvanian Peripheral Ruling that all Wizard Wars shall be resolved in this way, traditions of seconds, and so on.)
And then it can be played like a duel, with magi demanding satisfaction or whatever.
I think that's the point. Tremere designed Certamen to be his own little hack. People accept it because people generally won't put their lives on the line, even though some people game the system. Now some dedicated War Wizards will respond to Certamen bullying with Wizard's Wars a lot more readily.
It's about the mechanics of Certamen, and how it is really very dry, not too different than combat. Where combat is superior to Certamen, is that in combat, by and large, nearly everyone at the table is involved in the events. With Certamen, it might be two PCs or an NPC and a PC. Hard to get involved and excited about that when you're not participating. That's not to say that there might not be stakes for the other PCs present, it's just that it is a relatively dry experience for them.
Certainly punching it up with vivid descriptions of the phantasms is one way to do it, as is my suggestion of creating tension, activity and something to react to in the crowd for the PCs not participating in Certamen.
From a mechanical standpoint, I think the Dimicatio is a much more interesting and exciting contest.
IMHO Certamen works if you want it to. Mechanically and socially. If you really want it.
We've had our difficulties over the many sagas. In the current one we're trying to make it usable. We play in Loch Leglean, loose based on Lion of the North, with the feel of being a frontier Tribunal and with an atmosphere of strife (inspired by The Contested Isle). We fewer and less strict rules (our last saga was Rhine - at the other end of the scale) but also try to find a balance so a few powerful bullies don't rule it all by might alone.
First of, we have no Tremere, Whether or not this really is integral to making it work we can't tell, but I think many players feel intimidated in advance knowing that Tremere use Certamen and ought to be good at it.
Second, we try to have Certamen used more casually for settling differences which aren't covered by any rules - of which he have and want few. So our covenant's internal roles and positions may change if one magus challenges another's right to be Keeper of the Arcane for instance. Or whether or not some problem left unresolved during one magus' formal watch changes hands to the new magus responsible or if the original magus still needs to take action.
Third, we try to make Tribunal traditions and Peripheral Code ensuring possible conflicts with other magi over resources, but in friendly strife not all-out kill-'em-all warfare. And with certain limitations to create some form of balance. Created with the primary reason being stories and drama, rational rules and integrity second. There seems to be plenty of possible loopholes. And Certamen is believed to be a possible solution for some of these conflicts, at least temporarily
These are brand new rules we invented last Monday, because we ran Tribunal. The rules are in-game meant to have been in existence for a long time. We don't know yet if it will work.'
Also, I'd like to see Certamen between to conflicting magi be combined with Grogs and Companions doing other things - fighting each other, trying to interrupt the other side's magus so he loses concentration. This last bit is not yet resolved, it is still just an idea in my head, but is must not be too easy to pull off.
Personally I'd like Certamen to be more flexible and varied, to work more like Dimicatio. Once the Arts are set little changes round for round - sure you can switch Arts between Atk and Dfn but the dice are still very significant (unless the magi are old and Arts are very high). I'd like Certamen to be a form of duel also good training for real fighting, which is why I like the Fast Casting bits from Dimicatio. So Duellist also often make for good Fighters, Never could make anything work, the grouo saw holes in t before any implementation, and also had a dislike of the idea in general.
Oh, it certainly can work. The more that is done to keep all players involved the better it is going to work. But face offs against two people can be boring to everyone not involved. The longer they take to resolve, the more boring they can become. It's fun to read about an epic Certamen duel that lasts 30 rounds. Playing it...not so much, unless, as I've said, the others are in the crowd causing or preventing shenanigans.
This is true, of course. But aside from a particularly unlikely set of dice rolls, it is difficult to see how a Certamen duel can last this long.
In our saga we certainly don't have Certamen duels every session, but we do have one every few sessions. They usually only last half a dozen to a dozen rounds, and once you have your totals sorted out it only takes 10s of seconds to resolve a round. Often the longest bit is the players procrastinating over their choice of duelling Arts.
Most certamen I played (as SG running NPC) ended with a botch, or a player doing a 1 1 1 1 roll.
So: i'd remove stress roll.
Other ideas certainly, but I think some person already posted it here. (Like using actual "fake" combat, doing fatigue damage : it let magi use their potent spells on other magi without having to harm them. yeah for anyone developing spells, and not only the tremere.)
In our saga, we use an attack total much like the casting total of spells (adding Presence) and a defense total much like the magic resistance (adding Perception), and we use the non-lethal damage of Lords of men for the effects of successful attacks. So, in our saga, in certamen you could suffer a "fatal" bruise which causes a residual incapaciting wound.
We also have rules for certamen against several opponents and group certamen.
The Tremere no longer have a focus in certamen, they have a choice between Puissant Finesse and Puissant Penetration. A magus could specialize in certamen for relevant abilities (penetration, finesse, concentration, order of Hermes lore, but not parma magica). Order of Hermes lore could give a magus informations about the adversary's strong and weak points.
For the social effects, certamen is considered a sport in most cases, and a means of solving disputes if both magi agree on the terms. Without any agreement on the duel terms, there is no certamen and the magi must find another way of solving their dispute (force or tribunal case probably). For many magi, honor dictates to accept certamen, but the weaker ones often retreat behind the protection of the tribunal, especially when they know the Code is on their side for the disputed matter.
We only had one certamen for now, it lasted a mere moment as the PC magus lost concentration the first time he was hit. It was in the tribunal certamen tourney. We had two tribunals sessions, but in the first one, none of our young magi dared to enter the tourney.
Certamen that isn't a sport is a substitute for war, and should be considered on that basis. If you're challenging someone to certamen, you have a right to set any stake you want and the other magus doesn't have the right to simply refuse unless their Hermetic rights would be violated, but they do have the right to declare Wizard's War on you instead of fighting the certamen. On the other hand, many Flambeau and Tremere magi will fight a certamen even when their rights are unambiguous, as a point of honor.
Just how prevalent certamen and WW are depends, as always, on Tribunal and saga. If I'm running a saga with a functional Order, I'll assume that using certamen to bully weaker magi ("I say that your unGifted daughter is my slave, and will prove it through certamen") will get the Redcaps to step in and politely ask you to quit it. And if you get too flagrant, then eventually a mass of magi will get together to declare Wizard's War and put a stop to your bad attitude - or in the most exceptional cases, the Quaesitores will bring you up on Endangering the Order. If the Order is descending into Winter, then certamen-bullying is commonplace, only restrained by the threat of war.
Despite all my best efforts over multiple campaigns, I have never been able to get players excited about Certamen. The closest I came was in HBO, when Samson of Flambeau learned a particular Certamen fighting technique, then trained for a couple of seasons to get his Finesse really high to take advantage of that technique, and he ended up dueling another PC in a confrontation over a group of young magi trying to found a covenant in Hibernia. But once he'd crushed his opposition, he basically retired, and even a Certamen tournament was not enough to bring him out.
My frustration with making Certamen fun enough to engage players is exactly why I want a Certamen card game.
I did not expect a necro of this thread. That said, I never used it much in my games. I think I tried it a couple times in the late 90s or early 00's before 5th edition. It wasn't fun and no one liked it. My most recent 5th ed game a few years ago, we tried it out, and it was just a gian yawn. It could make for great reading in a novel, but not in an RPG.
What might make it work better is to detach it from mechanics period, and make it a narratively resolved conflict. Though I'm doubtful this would work for very many groups.