As a first hand example, when I was in college (in Japan), one of the foreign students just disappeared.  He didn't show up to class for weeks, and when the professors finally got worried and checked, he hadn't been to his dorm room either.  None of his friends had seen him recently, and his parents didn't know where he was either.  The school filed a missing persons report with the police, who eventually contacted the school and said "Don't worry about him."  That was all they would say, but the implication was that they had him, or at least did have him in custody at some point.  He never returned to school, and nobody saw him again.

There are only two things likely to get you into that much trouble in Japan, drugs, and guns.  Since it's nearly impossible to even get your hands on a gun (even the criminals are afraid to be caught with one!), it was probably drugs.  

That being said, there are some Japanese people who occasionally smoke up with a little marijuana.  Personally, I think they are crazy.  They might not get deported, but they can get imprisoned for a long time for any amount of any kind illegal drugs found on their possession at all.  It just doesn't seem worth the risk.  Even if they smoke in the privacy of their own home with a couple friends, smoking makes smoke, which smells.  It will only take one nosey neighbor saying "Hmm, I wonder what that smell is..." for everything to come crashing down.  If you know people who do anything like that, I suggest that you leave very quickly if anyone starts smoking something besides tobacco.  It won't make you the life of the party, and they probably won't get caught anyway - but if they do you sure don't want to be there when the police arrive.  This is doubly true if you are a foreigner.  Even if they don't arrest you on the spot, they may make a note of it that will be reviewed when it's time for your visa renewal!

Now, a while back CNN showed a special on people smoking "bath salts" in the Southern US.  At first, I thought people must be out of their minds to be smoking bath salts.  What's next, hand sanitizer?  Then I got it "oooh, they're actually not bath salts, they're 'bath salts'!"  That is to say, they are substances that will (supposedly at least) get you high if you smoke them, but aren't illegal.  It would be illegal to market them as something to smoke, so they market them as "bath salts", "incense", or whatever.  I wonder what would happen if you actually put them in your bath tub?

Anyway, I came to find out that a similar legal loophole exists in Japan. Enterprising chemists are taking advantage of the fact that the law moves slowly to create and sell "incense", etc. that contains chemicals very similar to those in marijuana, or other drugs.  They call it "Herb" (ハーブ), and the whole category is called 脱法ドラッグ, which means law-evading drugs.  According to news reports I have seen, products exist in leaf form (to look and smoke like tobacco or marijuana), power form (for smoking? snorting?), and liquid (presumably for drinking).  The leaf products may look natural, but they aren't.  The producers take tobacco or some other leaf and saturate it with the chemicals they have made.  Occasionally the legal system makes one chemical or class of chemicals illegal, and the producers move onto a new formulation.  In fact, even if a shop gets caught selling real drugs, they can apparently claim they didn't know, and that the supplier is at fault.  The news has also mentioned dozens of reports of people being rushed to the hospitals, so obviously the chemicals used are not that benign.  Since these aren't actually marketed for human intake, they don't have to list the ingredients, and they don't have any kind of safety approval.  

I strongly recommend against buying or taking any of these "legal drugs" for the following reasons:

1. There is a chance that what you bought isn't actually legal after all.  If you get busted with real drugs, it's big trouble for you.

2. Even if what you bought was legal, it might become illegal after a short while (when the law is updated) without you knowing.  

3. Even if what you bought was and still is legal, it might not be safe!  

4. Even if what you bought is legal and (relatively) safe, if you get caught with something that looks like illegal drugs, it will be a big headache.  Can you immagine you telling the police "But it's a legal drug!", as they arrest you?  Sure, after you have been in the police station for a week and the tests of the drug come back and show it's legal, they might let you go - but you will be on their watch list for sure.  (And again, this will probably be filed in your police record and reviewed when you apply for a visa or something).

5. Even if the drug would be mostly safe when used properly and you would never get caught, I don't think the shops would advise you on the proper dosage, etc. - since they need to maintain the official stance they they are selling incense, and not something for you to consume.  The products are also constantly changing in order to stay ahead of the law, which means what you buy this week might not be the same thing you bought last week - and what was a safe dose of last week's product might be an overdose this week.

Looking at all of the above - is it really worth it?

Alcohol and tobacco are both perfectly legal in Japan, but if you think you need more than that, Japan probably isn't the best place for you to be.

Update:

Starting in late 2014, there has been a crack-down on the semi-legal drugs, and many places which had been selling them have apparently been closed down.