The closet thing I can find to a "rule" is that if the word pre-existed English - e.g. Gk. archangelos - then it continued into English with the "k" sound. If the word was coined in English, by attaching "arch-" to an existing English word - e.g. Eng. archenemy - then it is pronounced with a "ch" sound. This is because the standard pronunciation of the Greek shifted from "k" to "ch" over time.
So, as a creation of modern fantasy literature in English, "archmage" would be pronounced with a "ch" sound.
But, since we are imagining that in the ArM world archmages existed back into Greek usage as "archimagoi", it would presumably have come forward into ArM Latin with a "k" pronunciation rather than a "ch" one - pronounced as "arkmagus".