Verify the Robots.txt File
Verify that the robots.txt file is correctly placed for a robots.txt file.
- In your browser, enter the hostname of the
instance, a slash, and then robots.txt.
For example:
-
http://www.yourshophere.com/robots.txt -
http://staging.realm.company.demandware.net/robots.txt
-
- You see the contents of your robots.txt file appear. If you get a blank page, the file isn't being picked up.
Note: You might need to make sure that the static content page cache was invalidated
before you can see a new version of the robots.txt file.
To Verify That the Robots.txt File Is Behaving As Expected:
To check that your robots.txt file is behaving as expected, use the robots.txt analyzer tool in Google Webmaster Tools. You must have created a Google account to use this tool.