Sorry if this is something that is supposed to be known, but I couldn't find any particular mention in the books I've got.
When creating new magi following the character creation rules, is it allowed to invent new spells for their initial 120 levels, or are they restricted to the ones written in the rulebook? I would assume that since nothing is mentioned, you can indeed design them, but in my last game the storyguide didn't allow us to choose "custom" spells for this stage. I assume it was simply for convenience, but I'd like to clarify this.
ArM5 rules to create Hermetic spells are not always straightforward. Guidelines for their creation are spread over many books appearing over the years, and some spells contain/imply their own guidelines never written out separately. Some implicit, intended restrictions have been made explicit only in later books (like HoH:S p.68 about the creation of Abilities by Hermetic magic, or TME p.99 Limitations of Hermetic Theory).
Please bear with your SG, if he does not wish to dump the campaign and your characters in at the deep end. Allowing all the spells players come up with in a first creative burst of character design can cause serious trouble, if not all their consequences have been checked sufficiently at campaign start.
New characters can specialize in magical effects, which soon are found to be impossible for current Hermetic magic.
Some spells may be closely monitored or even banned by the Order of Hermes.
Other spells may have to conform to guidelines not seen at character creation, requiring later removal of that spell or rebuilding of the character.
So, being careful with characters' starting spells is necessary. Only allowing published spells is very defensive - but it may be the sign of conscientious beginners.
Keep in mind that the story effect of a character having a starting spell means that their master knew it and taught it to them or at least had a lab text for the spell and granted the character time during apprenticeship to learn the spell. So, generally, as an SG I don't allow many (any?) custom spells without a really good reason.
A wholly custom spell that requires working out its own base effect is probably not a good idea. A 'custom' spell that is taken from one of the splatbooks but does not require any additional virtues is probably okay. A minor adjustment to an existing spell (e.g. Endurance of Berserkers with D: Diam) is the kind of thing I'd let slide.
A good rule of thumb is 'do I need to adjudicate some icky ruling with this spell?' If the answer is yes, that's a good indicator that the spell is a bad idea.
There's also the question of why the spell is being selected in the first place. Players tend to pick spells for one of these reasons:
defensive selection - specifically picking spells / abilities to counter expected threats within a given conflict environment
offensive selection - specifically picking spells to be able to crush expected threats within a given conflict environment
flaw mitigation - picking a spell to make a flaw into a not-flaw, such as a spell that allows a blind character to see.
fun - picking a spell because it seems appropriate for the character's personality or it just sounds cool
Of these, #1 and #3 need the most consideration.
#1 means the player is looking for risk-free encounters, and is expecting these kind of challenges to make up the bulk of what they face. It's natural to do that, but it also changes the dynamic of the game. It makes the SGs job a bit like writing Superman; writing a story about punch-ups and martial conflict in which the hero is invulnerable. Heroes who are never at risk aren't really heroes.
#2 means the player wants to feel like a god in a given scenario. That's fine, and make sure to throw the odd scenario their way so they can feel godly. But it means being careful to ensure the limelight is shared around.
#3 is a bit of a no-no. It means the flaw is not a flaw. If a flaw is not a flaw, it shouldn't be worth the flaw points.
For the campaign I ran, I allowed my players to have one invented spell at Gauntlet. The rest had to come from either Core Rulebook or one of the HoHs. My reasoning was that, for my setting, the Core Rulebook spell list actually contains most of the spells that are in wide circulation within the Order. Anything beyond that would have to come from their parens.
I do believe the the method in my gaming groups is: Go nuts! Have as many or as few custom and variant spells as you want. In fact we encourage people to think outside the box, and perhaps want something else that what the core book lists. Often it is simply a change of a single parameter. The only requirement we have (apart from designing it correctly so it actually works, and have the numbers check out) is that variant spells need a distinctive name from the spell it resembles.
I think this is ok since your parens or someone from his lineage may have had made a variant of said spell and taught it or made a text. If your concept is a magus specifically using different magic that your parens I may ask how a custom spell could be known if it is not whtin the magus' capabilities in a lab to have made it himself. Never been in that situation though.
Also, I agree completely that the spell list in the core book can be used to represent the spells commonly known in the order and in easy circulation. Some more than others though, but we don't have a ready list. If need be we'll look at the "list of common spells" form CJ's questionnaire last year.
When a player announced his magus seeks a lab text on such-and-such spell, we decide the cost and time to deliver. Usually quite cheap and within a year if it is a spell from the core rues, it isn't high magnitude, and it doesn't seem 'rare'. Although this may depend on tribunal and saga. Right now we play in the Rhine so a lot of spells are (theoretically) available at Durenmar, if someone bothers making a copy. And if you are on good terms with a Redcap both the request and the delivery can be snappy. For General spells the buyer usually specifies a range of levels and the Troupe decides which ones are available. For even more specific things like Rego and Muto Vim of which there are a multitude of variables we must make even more consideration. Like: "I want Maintaining the Demanding Spell but at D: Sun and I need it to affect 5th magnitude spells" or "I want Wizard's Extension Ignem to affect 3rd mag spells". The Troupe may decide there is a text available but it is of lower level (and thus may be too weak) or higher level (and this present a challenge in inventing and perhaps also subsequently casting). Or it could require a journey somewhere to look through an old library, or to negotiate with another magus or covenant who in turn may want something. Because this is simply more fun that saying either "no" because this limits the character or "yes" because it may merely cost vis which a bookkeeping currency, while time and stories gives more drama.