So, I gave this some thought.
I think adapting the Experimental Philosophy rules in Arts and Academe could be of some use here. If you wanted to apply the Hermetic Research rules, you need to have effects that can be generated to express potential avenues for solutions for the central research problem.
Having a specific new Experimental Philosophy table for mathematics could allow for the creation of effects, which then lead to the use of Hermetic Research rules- however, this requires some kind of practical or mechanical utility for mathematics within the context of the game.
Because the study of mathematics is incredibly useful and has helped numerous other fields of study, but it could be difficult to translate that into utility (I would recommend something that enhances the ability to teach, learn or use Artes Liberales or other forms of Experimental Philosophy, personally).
This would also require tweaking the experimental table (I did an alternate table for book writing here).
However, doing this requires some level of abstraction between what the actual work being done is and the impact it will have on the game- since an individual "breakthrough" may not have any immediate applications outside of its own field. I confess to be really bad with a lot of theoretical and abstract mathematics, so I'm afraid I can't help you there.
You could always develop a new system that's mostly self-contained, wherein there are problems that award you with experience or something similar when solved and that "breakthroughs" allow you to do different things with the type of experience or the speed of resolution- so that problem that would normally require a great deal of time to resolve becomes much faster, or perhaps that you can apply some proof you practice to Astrological affairs or similar.