........Okay, given that you listed one of your favorite shows in another thread as Minoriteam, I think this about sums up that.
Xavier and Assy were good in their own unique way. Xavier is one of my favorite shows alongside Home Movies and Lucy, and the random stuff is not the meat and potatoes of the show. The underlying philosophy and wordplay was. Same with Assy McGee, the random humor wasn't the important stuff, it was a literal take on an arsehole cop drama. Assy McGee, the character, is a pure visual pun to the fullest degree.
Home Movies and Lucy don't play the random card, and have no reason to. They play it straight and take it slow. It's all about completely character driven dialogue. That being said, if it isn't for you, it isn't for you. I could write 10 pages worth of text describing how Home Movies and Lucy are wonderful programs, but it wouldn't change your mind. Especially if you are watching for randomness and not subtle humor.
While I don't expect you to change many of your opinions, I suggest you try to look at your favorite shows in a different light. Xavier is a completely different and thought-provoking show when you understand the philosophy that went behind every episode. It's way more than just random humor.