I'm not a drug user at all, never have been, and I love the show as well as T&E, so I guess I'm yet another example proving those "only stoners" people wrong. I don't know if I'd call the show deep in the way that some Xavier fans might think (the fanmentary on the DVD ugh), but I do think you need to be above average intelligence and a huge comedy nerd to fully appreciate the show. But to be clear, I don't think you necessarily have to be smart to just like the show in general, there's plenty of shock humor and general craziness to satisfy anyone who likes that.
There are story lines and dialogue that are just nonsense. Playful, sarcastic, and absurd. A lot like Hitchhikers Guide to the Galaxy talk (heck the whole plot of the first show is a Hitchhikers ripoff). There's no hidden intellectual nuggets in the vast majority of Xavier's musings, and trying to look for them loses the fun of the show!
At the same time, certain story lines and certain parts of the dialogue are obviously ideologically pointed, and you kindof have to be in that head space already to find it funny, being prepared to detect sarcasm and not being a religious or spiritual individual helps... a lot.
So, I think it requires intellegence to distinguish between the parts that you need to engage in and the parts that you just let sink in. Also, of course, it requires a certain amount of intelligence to detect and appreciate all of the sarcasm. Like T&E, most of the show is sarcasm.