Suppose that Exemplum Ex Misc has the Book Leaner virtue, an Affinity with Rego, and a Rego score of 10 with no XP toward Rego 11.

He then acquires an excellent Q12 Tractatus, and spends a season studying it.

Does Exemplum gain (12x1.5) + 3 = 18+3 = 21 XP, putting him 2 XP away from Rego 12, or does he gain (12+3) x 1.5 = 15 x 1.5 = 22.5 (round up) = 23 XP, putting him at exactly Rego 12?

In canon, the rules say round up, and that's a hard "round up", ie, 3.1 becomes 4. So I would say, unless you are running a really cutthroat game, 23 exp.

Right, but the question is, which do you apply first? If you have an additive XP bonus and a multiplicative XP bonus, the order is important.

If you add 3 XP and then multiply the total by 1.5, then you are going to end up with more XP than if you multiply by 1.5 and then add 3 - without rounding of any sort.