Fair enough. Are they from a shared religious background? Like, using your example of Delphi, I could imagine a group that claims descent from the priesthood of Apollo at Delphi who look after the site and negotiate between, say, Sibyllic, Mithraean and Mercurian cults. They're very different cults but have some shared cultural DNA from the broader classical Mediterranean religious-cultural complex. So the caretakers might be a small-c 'cult' in the sense of a pagan religious organisation without being a full-on Mystery Cult in mechanical terms. They look after the site, handle bookings and ward off intruders, including cultists who aren't from the same 'family'. (E.g. if this was in an Ars context of classical pagans + Christians, Muslims, Jews + non-classical pagans, the caretakers' roles could include chasing off intruders from Christian/Islamic/Jewish mystery cults.)
Somewhat more of a stretch, but given the shared heritage, you could even have them as outer attendants in the lowest level of each of the associated cults, so e.g. if the Mithraeans are holding a ceremony but don't have enough choristers to make quorum, the caretakers can fill in (and don't stay around for the higher-degree ceremonies).
Alternatively, they could be the descendants of a family who were appointed by an ancient authority to look after the site, something like how the Joudeh and Nuseibeh families are custodians of the keys and doors of the Church of the Holy Sepulchre in Jerusalem (and were originally appointed by Saladin). (As I understand it, they aren't custodians of the actual church itself.) In this case, they don't have to be the same religion as any of the cultists at all, and being from a different religion actually helps their neutrality status when moderating between cultists from a particular religion.
So, just to make up a fictional example, say there were Sibyllines, Apollonians, Mithraeans and Pythians constantly getting into fistfights at Delphi in the early 300s. Constantine I heard about it, told them to settle down and gave physical custody of the site to a local Christian family. Their descendants don't take part in any of the cult activities, but they keep the outer parts of the site clean, ward off trespassers, negotiate admittance and so on. They might be the owners of the land the site is on as far as the secular authorities are concerned.
(EDIT: Or, to combine the two options, there's a family that claims descent from the last priest of Apollo at the site. Nowadays they're all Christians, but the cultists still recognise the family's authority as neutral caretakers of the site.)
(EDIT 2: A third option, just for laughs: the caretakers are actually fairies who claim to be the descendants of the original custodians of the site.)