Absolutely agreed. Back in college I met some very sincere, thoughtful, and kind libertarians. I occasionally imagine how horrified they'd be to learn that their considered and nuanced political philosophy had become a popular fig leaf for "I do what I want without regard to harm for others".
The problem is libertarianism has always been this way. Typically when you're younger it sounds like a good idea. It would likely work if people weren't assholes. Then you grow up and realize people are assholes. Most people grow out of their libertarian phase, but the ones I worry about are those that don't. All I can think about when I see an old libertarian is "That asshole would install a toxic waste dump beside my house and make me spend the rest of my short life suing them for damages"
Could be. But for me there's a difference between "we have to limit harm, but let's do it in ways that otherwise lean toward freedom" and the maximalist "I DO WHAT I WANT! YOU'RE NOT MY REAL DAD" types. So I agree the absolutist libertarians are somewhere between dangerously naive and sociopathic. And I think the volunteer billionaire defense squad is ridiculous. But I have met reasonable ones who saw it more as a direction to lean, a counterbalance to authoritarian approaches.