Issue #1: Locked Phones

The first thing to find out is: Is it locked?  If it's locked to an overseas carrier it almost certainly won't work here.  If it has WiFi, of course you can use that when you are in range of a WiFi signal.  If your contract has expired and the phone is paid for, then it's possible your carrier will unlock it for you (or at least tell you the unlock code) if you explain the situation ask them nicely.  (Note that if the phone is physically compatible, you can of course use it on roaming in Japan, but that quickly becomes very costly, and wouldn't provide you with a local number).

Issue #2: Hardware Compatibility

Is the phone physically compatible?  If you have a phone from Verizon or Sprint, it may work on the AU network in Japan.  If you have a phone from AT&T or T-Mobile, it may work with Docomo, Softbank, or possibly even eMobile.  GSM only phones (i.e. 2G TDMA GSM handsets such as the Blackberry Pearl) will not work at all.  The frequencies used by the carriers here differ as well, so the more frequencies your phone supports, the better your chances are.  eMobile, in particular, while having very good data service, uses frequencies that many phones don't support.  Willcom service only works with PHS phones (which aren't popular outside of Japan).  I would recommend having a friend test it with their SIM card first.

Issue #3: No SIM Card-only Sales

Most of the carriers here in Japan won't just sell you a SIM card, so you may have to take a junk phone and swap the SIM card into the phone you really want to use.  You may be able to find someone who has a contract they would like to get out of and switch it to your name.  

Issue #4: Phone Specific Services

Finally, even if you have a SIM card and plan that works with your phone, your phone is physically compatible, and unlocked, you may have to change a number of settings to get things like email working.  Many of these settings can be found online with a bit of searching.  Note that, for example, if you don't have an iPhone SIM card, then Softbank won't give you an iPhone ("(i) mail") address.

Issue #5: Going Home.  

If you do buy a phone in Japan, and you want to use it overseas, realize that you will have to pay roaming fees unless you can rent or buy a SIM card overseas to use in the phone.  Some carriers here (like Docomo) will unlock your phone, while others (like Softbank) will not.  Others, like eMobile, have some models locked and others unlocked.

Issue #6: Japanese Only Features. 

You may decide that you want some of the features which are only offered by phones sold specifically for the Japanese Market, including: 1Seg TV, Mobile Suica (Train Pass and e-Money), etc.  Many phones sold overseas also have poor Japanese support, though that is starting to change with the popularity of Android and iOS.