Hi. Newbie-storyguide here, asking stuff.
Are there no rules (game mechanic-wise) for fear or sanity in Ars Magica?
Not really (unless I'm forgettng something).
To me, that's good - it always worked n reverse for me.
Why be scared of the story whn ou know that all it'll cost your character is 3 "fear resistance" points?
There are crude mechanics for fear, in that a Bravery (personality trait) roll can be made. What happens if the character loses the roll - or if she succeeds, for that matter - is left for roleplaying and the troupe to determine.
Loss of sanity is not really part of Ars Magica, neither as a response to fear or in general. I would suggest similar effects can be produced by demonic powers, giving character Obsessions, flaws such as Wrath, and otherwise twisting their minds, and by magic altering the perceptions and apprehensions of the characters as detailed in Art & Academy. Otherwise, I don't think insanity as we currently understand it really features in Mythic Europe.
Of course, Your Saga May Vary - play however suits your game. Add on insanity rules from Call of Cthulhu, if that what you like 
Yair
Check out the Mentem spells, especially
CrMe Rising Ire (p. 148), PeMe Black Whisper (151) and ReMe Vision of Infernal Terror (p. 152) to see how spells and emotions go together. Maybe something similar can be used for your situation.
Thanks for the replies, especially JeanMichelle. Yes, it was mainly the supernatural horror I was after. Sometimes I´m not good enough storyguide to convince players "You are very afraid. VERY afraid!" and then a mechanic is good.