Who's who?

Growing up, our dad told
us this riddle, which I didn't
understand even after he
gave us the answer.

A man sits in a jail cell.
He is visited by another man.
After the visitor leaves, the
guard asks him who his visitor
was.
The prisoner answers:
"Brothers and sisters
I have none.
But that man's father
Is my father's son."
Who was the visitor?

Ok, I understand the answer
now...

But back then, I just didn't get it.