It's a lot of fun and can lead to lots of adventures. All the bomb scare type stuff is just a bunch of power rich homeland security $ infused blowhards trying to sound and look important.
If you'd like to get started, the activity itself is free on the website; once you get into it, going premium makes it easier. To get started, a yellow etrex is all you need and that's easily under $100 new and probably much less used.
If you don't want to spend money to get started and don't have friends or family into it - look at waymarking.com
It's a different thing, but there you will find geocaching guides who will go with you to get started.
Such as http://www.waymarking.com/waymarks/WM54NA