Legitimate mages in the study rooms, scriptorum. Not necessarily a dozen legitimate magi in the actual shelves and rooms where the books are stored.
You can also make the defensive spells linked to some sort of spell taht magi going into the shelves has to lower parma to get access spell cast on them (crco to create hidden mark) and you scry for those without legitimate access.
Why does the tagline "Ocean ex Tytalus' Eleven" keep popping into my mind? 
Against the arrayed Magi of Durenmar and anyone who owes them a favor or wants to get on their good side, I'm not sure the Primus of Guernicus could push a charge through.
I'm not sure how much detail is given in GotF, but I know "no direct access, ask a librarian" has been used in many important libraries in sagas I have played. Then you can scry, block and screen all you want, since anyone actually among the shelf who isn't a librarian is trespassing. Only librarians (identified by the arcane connections inserted into the protecting magical items) can get in.
But again I think the most important security is a very strict control of where every single book is at any time. It's not like they haven't got enough apprentices to fix all the ACs, and the amount of vis they have incoming from the library use alone is rather big.
If a book isn't where it should be it is stolen. Try to retrieve it by magic. If this fails, locate it and dispatch the Defenders of the Order. IMS a call from Durenmar (signed by the head of Bonisagus and/or Philippus) that they need someone to hunt down "a thief depriving the Order of its magical knowledge" would land a small horde of Hoplites with the culprit.
Now, you might steal a book and plant it with someone else...
I think the level of protection will depend on what is being protected and what the consequences are if someone gets a hold of it who is not supposed to.
Texts on Parma Magica are going to have tighter security than books regarding anything else, since it is important that those outside the Order of Hermes don't get copies. Lab texts may well have Peripheral Code rulings making access to the texts something to be monitored (i.e. Touch of Midas) If Durenmar has a copy of a text but don't have rights to make copies of it under the terms of The Cow and Calf, then they are going to make it more expensive to steal that text than it would be to purchase a copy of it legally. If a text is something that existed before the Cow and Calf became widespread and multiple copies exist outside of Durenmar, then it is probably not worth the risk of getting caught and marched to steal one particular copy.
The safest way to steal from Durenmar is to come up with a fast and secret way to record the contents of a book so that you end up with a copy when you've only paid for a short period of study - target high level summae.
The safest way to steal from Durenmar is to make a deal with the Dark Fae, and steal when he strikes with The Black Fir...
When chaos reigns is the best time to strike!
That's your best chance of getting away with a big heist but first you have to survive performing it! Not to mention that the conspiracy* itself is another capital crime.
*What are the chances of a great dark fae not monologuing?
The University of Oxford requires the following oath from everyone who wants access to the Bodleian Library:
I hereby undertake not to remove from the Library, nor to mark, deface, or injure in any way, any volume, document or other object belonging to it or in its custody; not to bring into the Library, or kindle therein, any fire or flame, and not to smoke in the Library; and I promise to obey all rules of the Library.
(When I was an undergraduate you had to verbally give the oath in the presence of a librarian. Nowadays they just require you to sign it. Some people have no sense of tradition!)
I wouldn't think it unreasonable if Durenmar were to require a similar promise (excluding the bit about smoking) from people wanting access to its shelves and to enforce it with magic. Certainly the first hard problem ought to be giving that oath while under magical surveillance... Perhaps the creation of a temporary front personality, entirely ignorant of the plan to rob the library would be your first step.
But I don't actually see the way to profit from this unless there is a specific target you're after that the Bonisagi won't just share with the rest of the Order: the sealed stacks of Durenmar doubtless contain the spells that were deemed too dangerous for the Order. The Shield-Breaker spells, the spells that allow you to make enchanted Shields, the true Immortality Ritual (with the hideous price that so many people would be willing to pay)...
In that case not only must the plan contain a way in and a way out, it must ensure that no-one ever suspects that the crime has taken place...
Indeed, and when swore it I inadvertently missed out the "not" and the bat-eared librarian spotted it and made me do it again.
Make sure itis an audible alarm - the louder it is the more pissed off Philipus will be at the offender. I hear he dislikes loud noises.
Probably no visitors in the library itself - Covenants, p. 96ff has some (tiny) morsels on Durenmar.
Ofcourse that whole chapter is rather interesting.
I'd love to see some of these paired with more active defenses, if only for the magnificent effects -
Imaginem trickery could be wonderful for faking entire shelves of books or hiding others. Not to mention traps. For example, a room bewitched to look like the archives, but which is in fact where Durenmar keeps its scorpion collection. It was a gift from a Criamon. Best not to ask.
Many libraries at the time would not be open shelf - you would request a book and it would be brought to you, by a librarian. If the volumes are kept in a restricted area where all but three or so magi would never have any reason ever to tred, then you could have all sorts of fun and never even have to so much as skirt the code.
Overall, at the very least, defenses should be of many different kinds, using different forms, so that even inordinately gifted magi would face areas well outside their area of expertise. Mundane traps will be necessary for specialists in Vim.
I personally like the idea of spy-mice. A Bjornaer of mine used his strength in Animal, and special virtues, to turn house-hold pests or pseudo-pests into a very useful network of spies. Its amazing how few people - or players - check for that sort of thing.
For the sake of fun, give it an Achilles heel - preferably something human... an assistant with access has some sort of story or personality weakness.
Getting in and out successfully should be an epic success taking months, at least, to scope out, plan and execute.