It agree that should be preserved, but it does create a problem with the rules as written.
Consider a maga with Subtle Magic. She will normally cast spells without gestures. She probably hasn't used gestures for casting spells since her early days as an apprentice.
Now she has to Fast Cast a spell. Since she is in a hurry she must .... use the gestures she can just barely remember how they go? Doesn't make any sense that using gestures would be quicker for her than not using them.
Similar considerations for Quiet Magic, Deft Magic, and spellmastery options for casting without words or gestures.
(On a related note, Ritual Magic can't be cast with anything but standard voice and gestures either which creates similar problems, but that is probably a separate discussion.)