Set crawl rules and generate a ready-to-use robots.txt file.
What robots.txt does
This file tells search engine crawlers which parts of your site they may or may not access. Upload the generated file to your site's root directory (e.g. example.com/robots.txt).