Our robots.txt file is not linked from anywhere on the site and contains just the following:
User-agent: *
Crawl-delay: 5
This only happens for some sites. Why is this happening and how do we stop it?
![[IMG]](proxy/64r1YmaKHeuUUqARGLFU3IbbIjwCxA6CFe1L5m1vSpSwoX5reTNpy0AExF8K34gJ1qoiu%2BcqU3tolCYY7ZEU3W%2FvNzazNteVwEB%2F/image.png)
Screenshot 1: Google Search console...
Why is Google indexing our robots.txt file and showing it in search results?
from Google https://ift.tt/30Sj6vi
via IFTTT

0 Comments