I heard some accounts that said the hecklers called him "cracker". Mind you, when I grew up, we never said either word, I grew up around whites (but spent 3 years in East Brooklyn, that's were I heard the word ******, and the blacks were so surprised that I never heard the word, I did after living in klanville, after we moved down south, but I spent less then 3 years in grade school with blacks and that's were I heard the word ******).
Now, as an adult, I worked various jobs through high school and college, and I got to know a lot of blacks from all backgrounds. That's when I first heard the word "crackers" that had nothing to do with soup. And I remember thinking this word is so silly and I had to ask where did this one come from, and why "crackers", why not "milk" or what else is white?....well, you get my point. It's silly.
When I looked at the video, Kramer is pulling his ear, this body language MAY indicate HE CAN HEAR THEM, and he probably heard some things that really hurt his feelings, as he is Jewish, and Jewish people have had a close relationship with blacks, so, he responded back with the feelings he had about what those guys were THINKING and SAYING.