I really don't know, having not grown up in such a situation nor knowing anyone who has. Said situation being part of the upper tier of society dwelling in a castle on a hill. Would it be too far away to let children wander on their own? I don't know. Gifted children are even more problematic. At the very least, i doubt they would be left unsupervised. And an apprentice, I would imagine, would be kept cooped up working.
As for a "lousy childhood"; they eat well and are dressed warm and get to sleep under a roof and have fire. Sounds like an excellent situation to me
It may be easier if we had an idea of what sort of children you are aiming for. Vibria's kids are 5 and under I think. Vocis's kids are all grown. The apprentices are a sticky situation. I have not yet considered what children may exist amongst the grogs and covenfolk. I myself am not too into creating child characters or interested in dealing with child matters. But others may well be! So maybe some of us should make some child characters for this one? Fortune points earned from stories go to the players and can be used by any character of theirs, so their is motivation