What is ironic about the narrartor's encounter with the blond man in Invisible Man?

In the book "Invisible Man" the man is a black man but he says that he is invisible because when one is black one has no visibility.  He explains the phenomena as occurring because the inner eyes of man doe not allow him to see who he really is.

One night he is walking and he accidentally runs into a blond man. The man calls him an insulting name and so he grabs his coat lapels.  He tells the man to apologize.  He butts the man in the head and the man tears up but does not apologize.  He continues to curse him.

The invisible man butts him again and again and the man falls down.  He kicked the man who continued to utter insults at him. He becomes so outraged that he pulls out his knife contemplating slitting the man's throat.  He realized that the man did not see him.  He feels drunken and ashamed and leaves the man on the sidewalk in pain.

The next day he reads in the paper that the man had been mugged.  He had known the true details.  The irony is that the man had never really seen him.  He had only seen a color to curse at a label.  He as a person had been invisible to the man.  The man had thus thought himself to have been robbed by a phantom.