This seems like an oversimplification of Google’s value prop. Crawling the web is a somewhat trivial problem these days - ranking pages, removing spam, personalizing it to the viewer, and doing so in a matter of milliseconds from wherever you are in the world is a far greater problem and competitive advantage.
Pretty sure this is an oversimplification of Google’s value prop.
I didn't say anything about Google's value prop. My point is, whatever their value prop is, it's built on top of their ability to crawl the web at mass scale, and very quickly. So anybody who wants to compete with Google, by being better at "ranking pages, removing spam, personalizing it to the viewer," or whatever, will need to be able to do crawl in a similar manner. IOW, crawling is part of the "price of admission".
Yes, but I think the comment you responded to were saying that it is an insignificant part these days.
Maybe so. In which case, I'd say I disagree. Great crawling ability is definitely a necessary, but not sufficient, condition for building a competitive search engine. And while the technical aspects of building a large scale search engine have been at least partly trivialized by OSS crawling software, elastic computing resources in the cloud, etc., what is at issue is the possibility of site owners blocking anybody who isn't (Google|Bing|Baidu|etc).
In this context, that's my concern: being blocked from crawling, if you're not already on the "allowed" list. Hence my reference to the question quoted above, from TFA.
I think the terminology you are looking for (or at least that you could use that might trigger people to accurately infer what you are trying to express), is that search and crawling are the foundation of everything google has built, in multiple aspects.
In one aspect, it was literally their beginning, from which they were able to build a business and expand.
In another, much more on point aspect, it underlies the majority of their services, either directly of a few steps removed.
Like the foundation of a house, it may not always be the most visible aspect, and it may be taken for granted, but its contribution to the integrity of the whole can't be underestimated.
Web crawling is necessary, but not sufficient. You won't be able to build a better Google just because you can crawl the web, but you won't be able to build a better Google if you can't crawl the web.
You fail to understand that crawling the web is 0.5% of the picture. Google has the best treatment of that data to serve ads, personalise experience and that's what their value is. You think making a bit crawler would nullify google, but you'd have to do the other 99.95% to be better (which costs money most people don't have).
You think making a bit crawler would nullify google
No, I don't. I didn't say anything even remotely like that. What I'm saying is that crawling is a prerequisite for all of the "... best treatment of that data to serve ads, personalise experience ..." stuff. That's why this topic matters: because if people aren't allowed to crawl (or can't do it effectively for technical reasons, but that's a subtly different issue), then they can't do anything else.
Just to be clear, I am most emphatically not saying that just building a crawler is enough to compete with Google. What I'm saying is that you can't build a competitive search engine without that "0.05%" bit because it's required to enable the other 99.95%.
