I get this problem a lot too, and honestly I think it's just a bug.
It only seems to happen to me when, my storage buildings are full (whatever appropriate storage for whatever resources) when the delivery men first try to deliver. Then, even if I add the right sort of storage building (road connected, staffed, the whole nine yards) sometimes those delivery men still insist "no one will accept these goods...".
My solution to this is either,
A) I ignore them (whatever resource area they are coming from seems to make more delivery men anyhow) or
B) I delete the road out from under them and *POOF* they disapear! (and of course you want to rebuild that square of road right away!)
I try real hard to not look at the problem as a bug, but more as a motivation for me to build new storage facilities BEFORE the old fill up! Before! heh, easier said than done though, sometimes...
Anyhow, if this is your problem too, deleting road squares should be much more affordable than deleting gem mines.
Good luck,
pewknee