Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Useless in the sense it doesn't really need to mention Bing.

Blocking /css and /js makes sense because they don't add value in being crawled. Not necessary, but doesn't hurt.

Best practice: Always have a robots.txt file, even if it's empty.



What is the reason for your Best Practice advice?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: