What a sponge does
Sand and road dust are abrasives. When you run a sponge or a brush over the body, grit gets trapped between the foam and the paint and rides along the whole path of your hand.
That's where swirl marks come from: a web of fine scratches you see in sunlight, especially on dark cars. They don't wash off. The only fix is polishing, which means removing a layer of clear coat.