I believe DuckDuckGo does (or at least they did) this with Bing. Starting a new scraper at a scale that users would need to be useful for what they're used to is such a huge jump. I'm sure if Kagi continue to grow they'd prioritize their own scraping too, but that's just not feasible at first.

Back in the day I'd suggest doing it via Alexa top sites, but now that Alexa is gone, I'm not sure what strategy I would use, but I would want to hit sites that are like the "top 10000 most popular" first, and scrape every inch I could.

I think Kagi is going in the opposite direction: https://blog.kagi.com/small-web

They try to highlight small, personal websites instead of the big mainstream sites.

(This was a HN submission 2 weeks ago)

I saw that, but that's kind of useless when you kind of want something like SO or similar results, something Google keeps failing at.

that's what that means yeah, but not necessarily present it in the same way