Barring exceptional circumstances like Parma Magica or the Gentle Gift, the Gift creates an unfavourable emotional reaction in those in its "presence". What presence means was the subject of another thread; this one is, in some sense, about time rather than space.
What happens when you remember interacting with someone with the Gift, e.g. when you read a letter from him? Do you remember the animosity you felt back then (I'd say you do)? If you do, does it still affect you -- so now the interaction penalty applies to the written interaction too ("That bastard has the gall to write to me")? Or does it feel unjustified ("For some unfathomable reason I always end up arguing with that scholar, even though admittedly he's such a nice person -- let me make up for it")?