For the example given, I wouldn't allow a shift-over. Area Lore isn't just a mental map of the area, but also knowing the people and situations at work within the area.
Area Lore (Manor) means the person knows a lot about that manor. They know who the various people are, who is important, who isn't, who to go to for illegal stuff, etc.
They would also know the lands around the manor - so finding Manor B would be doable with a fairly easy area lore roll for area A. However, knowing anything about the people who live in Manor B (beyond those Manor A has regular contact with) would not be covered by Area Lore (Manor A).
Treating area lore a bit like languages makes sense. A -1 to -3 penalty for trying to use Area Lore (Manor A) on one of the neighbouring manors would make sense. I'd even be inclined to either let the second area lore ability be developed from the 'penalised rank', or grant the 7xp practice even if the person doesn't travel the area regularly until it reached that level to indicate existing familiarity.
2 miles is pretty close together. I wouldn't even require an area roll to find one from the other unless there was some serious mitigating factors, such as a pea-soup fog. Otherwise... 2 miles is close enough to cast sight range magics on individuals, so finding a building shouldn't be hard. 