Quaesitors in 5e can be of any house, including Mercere. In fact there has to be at least one from every house. Redcaps are for all legal respects, full members of the order. So is there any specific game reason one could not be a Quaesitor?
If one WERE to make such a character, what would he look like? I am guessing he would have magic items enabling him to produce "Forensic Magic". (Restore the faded threads, frosty breath of the spoken lie, etc.) Heck, I could see a lineage of these guys, the master passing on the magic tools to the student when he retires.
Maybe he is an expert for legal procedures and a good writer; wrote a weighty summa on Hermetic law.
Or as a investigator: He looks into the kinds of crimes that need the Mercere records for solving - like vis source registration fraud
I think he must be an indoor person, because he wouldn't stand a chance as a hoplite.
Redcaps can have a lot of potency. With sufficient backing, the only thing they really lack is Parma, which can be gotten around with wits and items that make them hard to detect. Or perhaps a magical "partner" to back him up. In fact, a hoplite Redcap would be an interesting character in a higher powered saga.
Still, I agree. That would actually be a really interesting lineage, and a fascinating character to play. There would have to be something extraordinary about him, a drive to be as good as his magical counterparts, and the ability to use his own brains where others rely (perhaps foolishly) on magic alone.
Or he could be one of the minority of Redcaps who are magical, tho' as far as story goes I like the challenge of the mundane investigation better, if one aided by magical devices.
"What would he look like?" Anything goes. Excel at few predictable skills, and from there it's dealer's choice.