One of the issues am facing is what to do about layout.
I have some knowledge of working with Illustrator and InDesign, so it's not a matter of doing the actual Layout.
My issue is page size. As someone not from the US or Canada, the page size I usually work with, and what the printers here use is A4, A3, and such.
But in the US and Canada it seems the common size for RPG books is letter, which had completely different measurements.
Should I make it one or the other? I am inclined to make two PDFs, one for Letter and one for A4, but am not sure that's a good solution, since, because of the size difference, the layout between them will be different.
If it's exactly half the world in either case, than I could learn to live with that...
But in all seriousness, this is a PDF, and I expect it might get printed for reference, in which case, Europe uses A4 paper, and the US/Canada use Letter paper.
If you plan on having a print-on-demand option and publishing in English, go with the "imperial" formats. If you leave wide enough borders to the left and right to accomodate for the narrower DIN formats, printing it on that kind of paper doesn't even necessarily require rescaling (i.e. shrinking) when printing out at home (and even if, it's usually negligible).
American here. While I do have a ream of A4 size paper that I use to print reference pages from European RPG publishers such as Free League, I have successfully printed A4 formatted pages on letter-sized paper without issue and vice versa. The margins are not optimal, but it works fine for my purposes since I only print a few pages for reference and I'm not printing out whole books. I recommend DIN A4.
A4 and US letter are very close in size and proportions. It's not actually hard to design a page layout that will print on both, just with a different balance of blank margins. (A4 is slightly narrower and taller.)
US letter is… well, there's a hint in the name. Pretty much everywhere else uses A4, so if you personally are not in the US, I would say go with A4. This is particularly true for Ars Magica, which has a proportionately more non-US fanbase than most TTRPGs.
If you want to do PoD, you may need to reformat, but that will almost certainly be necessary in any case, because PoD tends to be fussy about the PDF you feed into it.
Do you know how I can find out the requirements for a PoD file? In case I want to offer something I author as PoD, it would be good to know in advance.
I second David's input. It must have been at least a decade since the last time I had a printer complaining about paper size, and I print a lot of stuff from all over the world.
Interestingly, Atlas uses a custom size. The Hibernia book is 213x276 mm which is neither letter nor A4, and it prints alright on A4, and I reckon it does on letter too.
Perhaps, but as I hinted at earlier - A4 is a somewhat awkward size for printed books. Too tall pages. You certainly can use A4 - especially for game books as opposed to other types of books - it just less than optimal.
Just pointing out once more (as a European myself): If you intend to put it up as PoD, most people would probably prefer the books be the same format as the rest of the line.