How to Add robots.txt to WordPress, Shopify, and Webflow
Adding a robots.txt file to your website depends on which platform you use — the process is different for WordPress, Shopify, and Webflow, but the starting point is the same: generate a valid, correctly formatted robots.txt file using WikiPlus Robots Generator at wikiplus.co. This platform-specific guide covers every method for deploying your robots.txt file, from WordPress SEO plugins to Shopify theme code and Webflow publishing settings.
Adding robots.txt to WordPress
WordPress offers several methods. The easiest: if you have Yoast SEO installed, go to Yoast > Tools > File Editor. Click Edit robots.txt and paste your generated content. Yoast writes the file to your server automatically. If you use Rank Math, go to Rank Math > General Settings > Edit robots.txt. For a plugin-free approach, access your hosting control panel file manager, navigate to the public_html directory, and look for an existing robots.txt — if one exists, edit it; if not, create a new file named robots.txt in lowercase. A third option is to use the Insert Headers and Footers plugin to serve robots.txt content, though a physical file is more reliable. After any change, test at yourdomain.com/robots.txt and verify in Google Search Console.
Adding robots.txt to Shopify
Shopify handles robots.txt differently from other platforms. Starting in 2021, Shopify uses a robots.txt.liquid template file that gives you programmatic control. To edit it: go to Online Store > Themes > Actions > Edit Code. In the templates directory, look for robots.txt.liquid. If it does not exist, create it. Paste your WikiPlus-generated rules as Liquid directives or plain text. For straightforward allow/disallow rules, plain robots.txt syntax works inside the liquid template without any special Liquid tags. Shopify automatically adds default rules to protect checkout and account pages — your additions supplement rather than replace these. Deploy by saving the file; Shopify serves it at yourdomain.com/robots.txt immediately.
Adding robots.txt to Webflow
In Webflow, robots.txt is managed in Project Settings. Go to Project Settings > SEO > Indexing. There is a dedicated robots.txt field where you can paste your custom robots.txt content. Webflow provides a default robots.txt — review and edit it rather than replacing it entirely, as Webflow may add important default rules. After pasting your WikiPlus-generated content, publish the site to apply the change. Verify by visiting yourdomain.com/robots.txt. For Webflow sites with staging/preview URLs (*.webflow.io), Webflow automatically adds a noindex directive to prevent staging content from appearing in search results — your custom robots.txt applies only to your published domain.
Verifying robots.txt Across Platforms
After deploying on any platform, perform this three-step verification. First, open yourdomain.com/robots.txt in a browser — you should see plain text content with your rules. If you see a 404 error, the file is not in the correct location. If you see HTML, your web server is not serving the file as text/plain. Second, open Google Search Console > Settings > Robots.txt Tester and test a few URLs that should be allowed and a few that should be blocked. The tester shows which rule matches each URL. Third, use the URL Inspection tool in Search Console to check a page that should be crawlable — confirm Google can access it. Schedule a monthly reminder to check robots.txt after any major CMS update or migration.
Frequently Asked Questions
- Does Shopify have a robots.txt file?
- Yes. Shopify automatically generates a robots.txt file with default rules that protect checkout (/checkout), account (/account), and cart (/cart) pages. Since 2021, you can customise it via the robots.txt.liquid template in your theme code. Always start from Shopify default rules and add your customisations rather than replacing the defaults, as the defaults include important protections for your store functionality.
- Can I use Yoast SEO to manage robots.txt?
- Yes. Yoast SEO includes a robots.txt editor under Yoast > Tools > File Editor. It reads the existing robots.txt file and lets you edit it directly from the WordPress admin. The changes are written to the physical robots.txt file on your server. If you generate rules using WikiPlus Robots Generator, you can paste them directly into the Yoast editor — the two tools complement each other well.
- Why is my robots.txt file not being read by Google?
- Common causes: the file is at the wrong URL (must be exactly yourdomain.com/robots.txt — no subdirectory); the file returns a 404 or 500 HTTP status code; the file is too large (Google recommends under 500 KB); the file uses incorrect syntax. Check the HTTP status by entering the robots.txt URL in Google Search Console URL Inspection. Google Search Console also has a dedicated robots.txt report under Settings that shows the last fetch date and any errors.