Barring exceptional circumstances like Parma Magica or the Gentle Gift, the Gift creates an unfavourable emotional reaction in those in its "presence". What presence means was the subject of another thread; this one is, in some sense, about time rather than space.
What happens when you remember interacting with someone with the Gift, e.g. when you read a letter from him? Do you remember the animosity you felt back then (I'd say you do)? If you do, does it still affect you -- so now the interaction penalty applies to the written interaction too ("That bastard has the gall to write to me")? Or does it feel unjustified ("For some unfathomable reason I always end up arguing with that scholar, even though admittedly he's such a nice person -- let me make up for it")?
Yes, you'd remember your reaction to the Gifted person back when you met - assuming of course that you remember the encounter at all. It would be strange if you had completely forgot your impression of them.
But while you will remember your reaction, the memory itself won't trigger the effects of the Gift.
You remember thinking that person was a creep/untrustworthy/dangerous/treacherous/whatever, but thinking back on the encounter you probably won't be able to pinpoint exactly what made you think that.
So, no penalty when reading a letter or similar - regardless of if you had met them before or not.