I feel you. I bailed on doing side gigs about a year ago. Even with good rates the tradeoff between that extra bit of cash and losing most of my spare time wasn't worth it.

These days I charge an inflated consulting rate to come off the bench. So far I haven't encountered a client willing to pony up, which is kind of nice since I have my spare time to work on renovating my house and practice my blacksmithing.