Salesforce B2C Commerce 24.7 > Merchandising Your Site > Search Engine Optimization > Sitemaps > Notify Search Engines About Your Sitemap

Use the Robots.txt File to Prevent Crawling of Development and Staging Instances

When a robot visits a web site, such as http://www.yourshophere.com/, it first checks for the robots.txt file and analyzes it to determine if it's allowed to index the site. For the robots.txt file to be used by Salesforce B2C Commerce, you must set up your hostname alias.

Here is a sample robots.txt file that prevents all robots from visiting the entire site:

User-agent: *# applies to all robots

Disallow: /# disallow indexing of all pages

The robot simply looks for a /robots.txt URI on your site, where a site is defined as an HTTP server running on a particular host and port number.

Note: Storefront Password Protection only password protects dynamic pages, not access to static content like images. Deploy a robots.txt file if you don't want images or other static content crawled.

Related Links

Generating a Robots.txt File

Assigning a Hostname Alias

Infocenter Retirement: On June 30, 2023, the Infocenter was retired, and documentation currently hosted on the Infocenter will be published to Salesforce Help, Commerce Cloud Developer Center, and Salesforce B2C Commerce Developer Documentation Resources. For more information, see the release note.