I don't mean from the player's point of view, but from within the setting. I mean, how do magi know if a book's worth their time or not? And how long does that take? And what terms do they use to describe level and quality? I doubt they'd use our game-speak.
For that matter, how did we determine the level and quality of real-world books?
We say things like “that author is a good writer,” or “they wrote really clearly and concisely,” or “I found the meandering flow of the writing tedious and it was overburdened with terrible examples.” And all of these statements can be said by different people about the same book since there is not one style of writing that is any kind of similar thing to game system concept of Book Quality since how much someone can get out of a book differs not just on the book but the reader and their preferences and mood and willingness to try and struggle with difficult passages and any number of other reasons than the author.
But to actually answer your primary question, they roughly categorize the books into quality ranges of Damaged, Vain, Sound, and Excellent. This is explained in more detail on p95 of Covenants. Some folks have some issues with the categorizations but for Summae is is a function of the level and quality, just like BP, and tractatus they purely use the quality.
Magi are scholars and scholars throughout history have engaged in correspondence and peer-review. Modern day versions we have are the peer-review you find for published scientific papers (which will tell you if a paper is worth reading or a waste of time), the peer-review system for college textbooks, and the varies "professional" reviewers you find for non-scientific text. All of them work by knowledgeable people reading the text, giving it a grade (which varies by the type of text), and corresponding with others about it. The overall average of these grades is the generally accepted quality of the work.
Grades can range from a simple "Good/Bad", a "Statement of Quality" as used in AM Covenants (Damaged, Vain, Sound, Excellent), to more complex formula which work on a tiered grade system such as 5~0, 10~0, A~F, or any other possible variation (to include +/- or decimal numbers). The last is a more modern system which is unlikely to be found in ME.
Even if not played out, Magi will spend time sending out notes, questions, and insights to other Magi. Additionally the Redcaps will spread gossip they have heard which would be of interest to the Order, such as if they have heard several Magi comment on how good or bad a given newish text is. Magi who engage in Correspondence (the XP generating kind) should rapidly gain knowledge the other has of the quality of works read. Bits and pieces about varies works will be discussed and the quality of works not yet read can often be determined from this.
Magi who rarely if ever are visited by the Redcaps and tend to be isolationist will generally have little knowledge of the quality of newer works, while those visited often and who engage many of their fellows will often be abreast of newer works.
Magi judge the book by the cover.
If the book has been beautifully bound with resonant materials (in the skin of a fabulous beast, with gold and silver hinges with gemstones set in) then it was probably a really good book to begin with and deserved the extra attention, and must be worth studying.
If it's a pile of vellum lazily copied and then put into cheap wooden boards, they probably leave it to someone more desperate to study.
If they're a specialist and have run out of high quality books to read, then they will read any book on their favourite subject and then let everyone else know if it was worth the read or not.
Take a look at HoH:TL p.10f Colentes Arcanorum. So House Bonisagus has made an institution from the tradition of classical Greek and Latin literature to evaluate, excerpt, quote, reference and distribute earlier works.
In general, also in the middle ages books were commented upon and judged in other books - so scholars and in particular librarians had often read about a book before finding it. A famous medieval book is the Myriobiblos of Photios, listing and reviewing some 279 earlier books.
Ooo...
Almost half the books mentioned no longer survive. These would have disappeared in the Sack of Constantinople by the Fourth Crusade in 1204[.]
Very setting relevant...