Lunch Money Rules Questions

Regarding Loss of Counters

The rules state : "If your attack succeeds, it is your responsibility to remove counters from the victim's pile and put them in the lost counter pool". If I am a victim and my opponent fails to remove my counters do I still lose them?

To be honest, we've almost always removed our own counters, usually with much theatrical moaning and wincing and carrying-on. But I guess it's fair to read that as saying it's ultimately the attacker's responsibility to make sure your counters get taken away.