Are you sharing with an active AI? If that's the case, they will order their own guys to harvest gold. That's simply part of the AI scripting. To fix this, you can remove the player from its slot, but keep the units there. However, the AI wont work then, so you would be essentially commanding idle units.
-VGsatomi