While A&A strongly suggests you have to have a (rational) soul to have Intelligence, that is only almost compatible with the rest of canon. The contradictions are:
Magic can create intelligent magical animals and faeries. (Per the Limit of the Soul text, some magi believe it only summons these.)
Cunning animals that become familiars gain Intelligence.
Marvelous Hound in Covenants.
Awakened Animals in Calebais. (Note that Calebais involves some adaptation/early-in-edition weirdness. For example, it also says "the animal’s Cunning becomes Intelligence, and it gains Presence and Communication scores." But in 5e animals already have such scores.)
The 4th level MuHe guideline "Awaken the consciousness of a plant (Mentem requisite)" as used by Stir the Slumbering Tree.
Were I writing 6th edition, my temptation would be to eliminate #1, #3, and #4, make #5 only produce plants with Cunning, and explicitly lampshade #2.
Does it really? Yep, to humans "The soul provides both reason (ratio) and intellect (intellectus)."
But it need not be the only possible source at least of reason, or kind of. Faerie minds are explained in RoP:F p.9f Cognizance, p.11 Pretenses and p.52: Faeries are able to mimic Abilities that would require reason for humans without having a human soul.
Faerie priests (like Monachielli) might perform explicit acts of reason also as parodies of human thought processes - and so might the awakened anima in a mechanica constructed following Heron's legacy (AM p.75ff).
Take a look at LE MALADE IMAGINAIRE Acte II Scène VII (Le Malade imaginaire/Acte II - Wikisource): Thomas Diafoirus reasons formally without understanding the purpose of the discourse. There you have a being with reason, but without intellect. As he is just a character in a play, he clearly needs no soul.
Yep. Since Medieval beliefs did not form a comprehensive, contradiction-less world-view, we have to accept there will be problems. The addition of magic doesn't really help that much.
Personally, I think it a big stretch to assume Hermetic magi have a particularly complete knowledge of the way things really work.
That's why I think it's much better to go into things picking and choosing what you want for a saga. There will be contradictions: What you can do is choose which you use to highlight the oddness of ME and/or dismiss with "that's just the way it works."
"Changing an animal into a human" is normally base 10, The Marvelous Hound might be base 15 because it transforms both mind and body.
The Cunning Characters sidebar on pg. 33 of RoP: Magic indicates that a Cunning character becoming Intelligent can just happen in addition to gaining a Virtue or Flaw through Warping.
I would expect that increase just for the physical part, though. It should be simpler to become a normal human shape than to become an anthropomorphized hound. That sounds a lot like MuAn 15. And then you need to transform the Cunning on top of that. Now, if this just changed the hound into a human, then at least 15 is bumped up for the mental part. But that's not the case.
It seems to me more like a cosmetic effect or even a limitation on the functional effect of having a human form. Like, it doesn't say anything about them retaining a bite attack or a hound's senses, so they probably don't have any functional benefit from it.
All it really does is potentially make your dgrogs bad at interacting with people.
Could not the Familiar Bond give a close enough connection the the Mage's soul that the Familiar gains enough benefit to convert Cunning to Intelligence?
The Familiar needs to be compatible enough with the mage to voluntarily enter the Familiar Bond.And the mage needs to respect and welcome the Familiar.
Need to know more of the nature of the Soul to say whether it can extend over the Familiar Bond. Which Hermetic Magi can't find out.
One conceivable interpretation, certainly, is that the intelligence of the familiar is drawn from the soul of the magus. This would also make a lot of sense if the familiar bond is the only Hermetic way to create intelligence.
Personally, I find every attempt to widen the scope of Hermetically creation towards mindful beasts and automata quite itchy. As was mentioned, we are really implying a lot of different features which should not be reducible to a single Intelligence characteristic, including soul, spirit, Aristotle's five modes of the soul, consciousness, and (free) will.
The most offensive creation of power gaming is the intelligent beast without will, allowing the magus an intelligent thrall under the complete control of their own will. I strongly believe that there is a level of intelligence which is impossible without that free will, and thus could not possibly bend completely to the magus' will.
Before we go so deeply into what aspects of Mentem can be created Hermetically, we should look very critically at what aspects would definitely be beyond Hermetic limit. The soul clearly is, but what other aspects fall under the purview of the soul?
Any real world situation will find contradictions. They exist in the modern world between models of relativity and quantum mechanics. They exist much more blatantly from our perspective in the medieval understanding of the world, with the simple reason that in the real world nobody has access to the rulebook.
The rules in AM conform to the medieval understand, contradictions and all, but just as in the real world the universe is not fully understood. There is surely a reason that humans require a soul for reason while faeries and magical creatures do not. It may be related in some way to why a human who becomes a magical creature loses their soul, or why conversely a magical entity that becomes human gains one. It is all in the big "we don't know" category.
Part of designing a spell to give an animal with cunning actual intelligence is buried in those contradictions and lack of understanding. Part of the question is going to be what approach the mage will take to model this intelligence- which can be buried so deep inside the story that it is never openly addressed. Maes will tend to build off what has worked for them, or failing that what has worked for others that they can find records of. It seems, from our perspective, that there are numerous forms of pseudo-souls that can provide intelligence or something indistinguishable from it, so it should be possible, but not easy, to accomplish. as with a lot of magic there are multiple ways to accomplish the same thing.