In fact, compared with the Western countries, there aren't many uneducated people who will actually say they hate foreigners.  Note I say foreigners and not people of other races, because the two go hand-in-hand 99% of the time in Japan.  Although there are whites and blacks who have Japanese citizenship, the number is very well under 1% of the population, so to most Japanese people, Citizenship=Race=Language=birthplace.  Also, because the number is so small, and because most foreigners in Japan are there only temporarily, they are treated as being outside society and have little voice.  In countries like the US where minorities make up a significant portion of the population, people have to listen.  In Japan, though, it's different, they are small enough in number to be safely ignored most of the time.

Again, outright racism is rare, because Japanese are well educated and consider racism to be a bad thing.  Yet, Japanese also hold a national pride, like people of most countries.  More to the point, most haven't dealt with foreigners, or have done so only to a very limited extent - and so it is easy for stereotypes to prevail.  I have witnessed this in Western countries as well.  People who live in the cities where concentrations of minorities are higher are typically more accepting, as they have dealt with people of all races on a daily basis for some time.  People in more rural areas are more likely to be openly racist.  Japan is much like the rural areas overseas in that the concentration of minorities is very low, even in the metropolitan areas.  On a positive note, it is important to realize that there are very few outspoken or violent racists in Japan.  People simply see things from their perspective, which foreigners don't understand.  They see dealing with outsiders as risky, and many times they are right.  The closest thing to the KKK that exists in Japan is the black vans with loudspeakers blaring nationalistic speeches you will occasionally see parked in populated areas.  Even most Japanese people think they are crazy.  On the other hand, when positive things happen in Japan, they congratulate themselves, while when poor things happen, they often blame others - especially foreigners - for the situation at hand.  

Many tourists in Japan are not exactly polite, and thus make the situation worse.  When Japanese people witness foreign people doing untoward things, you can't blame them for assuming this is common-place.

The English Myth:

Most Japanese Know English right?  Wrong.  

Ahh... I know what you're thinking, "But they all study it in high-school, right?"

Well, yeah, sure, just like all people in the US used to study Italian, German, or French in in High-School.  Still, once they graduated high-school, how many people remembered it more than a year later.  What you learn in high-school classes is limited, and if you don't use it, you won't retain even that.  English isn't typically used in Japan for any serious purpose, and there is almost nothing that is available only in English, so there's little use for English in Japan.  In fact, there's really not even much chance to use English in Japan unless you work for the tourism industry, or go out of your way looking for a way to use English.  Estimates are that between 10%-30% of the population speaks English in Japan.  I would say that if the meaning is American high-School level English, then the number is closer to the 10% or lower.  The 30% probably represents the number of people who can speak or understand some limited amount of English at slow speeds.

Why is this important?  Because when most Japanese don't speak English, and most foreigners don't speak Japanese, then there is no real communication - and that contributes to the building of stereotypes.  Many foreign people don't realize the number is so low, because they of course spend time talking to the small percentage that do know English, but the overall numbers affect how society views foreigners in general.

So, without further delay, let me introduce some of the common stereotypes about foreigners in Japan:

Whites:

Japanese people have a love-hate relationship with white people.  On the one hand, the US won World War II (and yes, Japanese typically consider that Americans are all white, and all white people are Americans).  White people are seen as superior in some ways due to winning the war, and yet seen as inferior in many says.  After all, Western culture is seen as being the "Wild-Wild West" due to the relative lack of process (cowboy mentality), higher levels of violence, crime, homelessness, etc.  White people are often seen as glamorous, probably due to the Hollywood Effect.  American movies typically feature mostly white protagonists, who are strong, brave, etc., and Japanese people watch these movies.  (American movies feature a large percentage in the action genre compared with Japanese movies).  Thus, all white males are somehow tinged with the image of Brad Pitt by some Japanese.  On the other hand, foreign people in general are seen as unsophisticated.  Foreign people typically don't know Japanese customs or mores, and so appear to be impolite by Japanese on their home turf.  (It doesn't occur to most Japanese that the reverse is also true then they go overseas).  This is also the fault of foreign people who come to Japan and don't take the time to learn about where they are living, of course.  Foreign people are seen as "cool", which makes them desirable for dating, but also not good for being serious with.  Many Japanese people wouldn't want to take their foreign boyfriend/girlfriend to meet their parents, who might be shocked.  People who want to learn English assume that all white people know English, and may actually use you for that purpose even if they don't otherwise like you.  At a basic level, being white will make you more popular socially in informal situations, and make your life much more difficult in formal situations.  You will be seen as a play-thing or entertainment, but not taken seriously.  Even if you know Japanese and work a real job (which would put you ahead of most foreigners in Japan), things like renting an apartment or taking out a loan will be more difficult because banks and such will tend to look at you with suspicion.

Blacks:

The love-hate thing is even stronger with black people than white people.  Younger city people especially may love the hip-hip culture on the surface, and of course assume that all black people are the same and automatically rappers or such.  Thus black people may enjoy even more automatic popularity than white people in some circles.  This is countered by being taken even less seriously than white people in some formal situations.  Also, since when black people aren't being portrayed as hip rap stars, they are frequently portrayed as criminals in Western media, Japanese (and Asian cultures in general) unfortunately tend to believe that too.  I met one Chinese girl who had never dealt with a black person, who tended to avoid them because they were all "poor or illiterate".  She must not have met my black landlord who had an MBA.

Asians:

Perhaps the hardest hit by racism are other (non-Japanese) asians.  They have all of the suspicion leveled upon blacks and whites, but without the glamour.  Japan is economically superior to all other Asian countries so far (although China is closing in fast on the national level, the GDP is still much lower), and thus they tend to look down on the other asian countries in much the way that many Americans look down on Mexico.  To make matters worse, the history between Japan and the other Asian countries is less than pleasant.  Japan committed horrible acts during World War II, and has failed to apologize, preferring to sweep the issues under the rug rather than admit fault.  This means that man other Asians don't like Japanese because they feel Japanese are ruthless, unapologetic, or simply smug.  On the other hand, Japanese people frequently see other Asians as uneducated, poor, etc.  Many Chinese, and the majority of Filipino people in Japan are working at Hostess or Massage, which tends to lower their image a bit more.  On the other hand, a large percentage of Chinese and Koreans learn Japanese properly before moving to Japan, attend a Japanese University, and graduate to join the workforce with a real (corporate) job.  You will see many Chinese working in large Japanese companies these days compared with white or black people, because they have done a better job integrating with society.  As an aside, many Japanese men like Chinese and other Asian women, because they see White/Black females are large and intimidating.  Chinese girls are cute and exotic, while still not intimidating.

There are basically two ways someone who comes to Japan can go:

a. Use and abuse your foreigner status.  First of all, you can rack up as many girls (or guys) as possible, since many younger people are interested (in a superficial way) in dating foreigners. Secondly, you can get certain jobs (like English assistant teaching) almost automatically.  Although asians are expected to learn the language, many people will assume white/black people aren't capable of this.  Even if you know Japanese, you can of course pretend you don't when it's convenient.  You can blatantly do things that aren't allowed, break the rules, and ignore social mores.  This is of course bad, but you can always pretend you didn't know, and most people will believe you.  They will just shake their heads and say "those silly ignorant foreigners".  Insist on only eating McDonalds, and import foreign movies and such all the time.  Only hang out with the small percentage of people who know English, and only go to foreigner hang-outs like "Gas Panic" and "Hub."  If you want to be really annoying, you can even speak English all the time, but spice it up with words that even Japanese infrequently use like "Gaijin", in order to try to be a hipster.  In the unlikely event that you work at a Japanese company, make sure you head straight home when everyone else goes out after work.  You can probably get away with all of this, but it will be at the expense further the stereotypes of the idiot foreigner, and hurting things for everyone else.

b. Assimilate.  Learn Japanese, go to a Japanese University, get a job at a Japanese company, hang out with your co-workers like everyone else.  Learn to entertain customers, and live in the same kind of housing as other people (With a Japanese land-lord).  Learn to like Japanese food, and in general strive to understand and fit in with Japanese culture.  Some would see this as "selling-out", but nobody says you can't also do those things you want to do, but that don't fit the mold.  You can still eat your foreign food at home, you can still watch English (or whatever language) movies sometimes, etc.  The point is, by striving to fit in, you will learn rapidly, have more change of long term success, and raise the image of people like yourself from the "idiot foreigner" to at least "versatile foreigner".  The way things in Japan are right now, you will always be seen as a foreigner at some level if you look "different", but you will help yourself and those who come after you.  Then perhaps someday it will be the norm for those who come here to be an active part of society, instead of on the outside looking in wondering what's going on.

Nobody can stop racism overnight, but it's important to understand the causes, tolerate the situation as well as possible while pointing it out politely when appropriate.  Overall, the most important thing is to be a good ambassador, so that people will realize on their own that their stereotypes aren't always correct.

There seems to be a lot of talk about the word "gaijin", and its meaning, whether it's rude, racist or not, etc. (See one such discussion here.)  Many people feel it's harmless, while it bothers other people to no end.  Most Japanese people don't mean any harm in using it either.  I am going to agree with those who find it offensive for the following reasons:

1. Pointing out that someone is different when there is no point is discrimination, even if it's not negative.  I have one person who is eager to show that he is open-minded, so he often says unnecessary things like "Hey, my black friend Tom..." - which, ironically betrays his lack of sophistication.  Hotels don't have signs that say "Color TV" anymore, because color TVs are normal now.  Likewise, in normal conversation, you don't need to mention that your friend is White, Black, Asian, or other, because those should all be perfectly normal options.  If you separate people into two groups with no reason, you are still segregating.   

2. "Gaijin" is not a proper word.  At least say "Gaikokujin-sama" or something that seems more polite.  To be fair, I have almost never heard the word "Gaijin" come out of a Japanese person's mouth.  It's always foreigners who think the word is cute, and try to sound hip by saying things like "Dude, check out my Gaijin-card."  It reminds me of black people calling each other the "N" word, and then complaining when other people use it.  

3. Gaijin is often used to me "non-Asian", which is different from "foreign national."  People need to get used to the fact that not all non-Asian people are foreign (i.e. there are 100% legally Japanese people born in Japan who speak only Japanese, but don't look asian), and certainly not all foreign people in Japan are non-Asian (most of them are Chinese or Korean).  You can say "That's a technicality", but everything in this world is a technicality in some sense.  There is no clear line to divide people into groups, even if you wanted to.  The clearest line you can pick is the one defined by the law, and you can't see that one at a glance.

4. This is by far the most important:  In every instance I am aware of, it isn't the person using the terms who gets to decide if those terms are offensive or not.  If the people who the terms are used on feel uncomfortable, then the terms are not appropriate.  I might think that calling Japanese people "Japs" is perfectly appropriate because it's just a shortened version of "Japanese", but if Japanese people find it offensive, then it is offensive.  At the end of the day, I don't get to decide what's appropriate, they do.   Similar issues exist with other minorities who have gained such "nicknames".  Since at least some of the "target" population of the words "Gaijin" and "half" don't like those terms, or being called out as different at every possible opportunity, that means that the gratuitous use of the words is, by definition, offensive.

For a second opinion, check out the Japanese Culture page at the excellent Japan FAQ.