1. People make music videos out of it!!! What losers!!!!!!
2. I think a lot if it is racism that people don't notice. Asians, esp. Japanese people, are generally looked upon as nerds for some reason. For many people, when they think Japanese, they think guy with round glasses and a calculator pouch on their pants. That's how it is in America at least. It could be different in other countries.
3. As mentioned before, people take their obsession to a higher level than they do for other things.
4. It's a different and unusual language. Japanese is not a Romance or Germanic language (duh) and thus seems like it takes more effort to learn with the different alphabets (when in reality, it's a simple language compared to English). Also, the alphabets are VERY different. More effort=nerdy
5. Have you SEEN the people who write/draw anime?
6. Face it guys. Who here isn't a nerd? XD