How to edit robots.txt in WordPress

When a request comes in for /robots.txt, WordPress intercepts it through the do_robots hook and prints a small default set: it disallows /wp-admin/ while allowing /wp-admin/admin-ajax.php, and appends a sitemap line. Because this output is generated at request time, no file is stored anywhere on your server. That is why FTP and your host file manager show nothing at the root even though the URL returns content.

This matters because the moment a real file appears at the document root, WordPress yields to it completely. Your physical robots.txt takes over and the virtual defaults vanish. So the question is never how to edit the invisible file, but how to replace it, either by creating a real one or by hooking into the generator through a plugin that rewrites the virtual output for you.

The most direct method is to upload a real robots.txt to your site root, the same folder that holds wp-config.php and the wp-content directory. Connect with an FTP client such as FileZilla, or open the file manager in your hosting control panel. Create a new file named exactly robots.txt, all lowercase, and place it at the root, not inside wp-content or a subfolder. WordPress only defers to a file served from yoursite.com/robots.txt.

Paste your rules in, save, then visit yoursite.com/robots.txt in a private browser window to confirm your content is showing rather than the old default. If you generated your rules with robot.guard, you simply download the file and upload that exact file, or open it and copy its contents into the new file. Keep the Disallow for /wp-admin/ and the Allow for admin-ajax.php so plugins that rely on AJAX keep working, and keep your sitemap line so search engines can still find it.

If you would rather not touch FTP, both Yoast SEO and Rank Math include a robots.txt editor. In Yoast, go to Yoast SEO, then Tools, then File editor, and you will see a Create robots.txt button or an existing editable field. In Rank Math, open Rank Math, then General Settings, then Edit robots.txt. These editors write or manage the served file for you, so the result is the same as Option A without leaving the dashboard. Note that if a physical robots.txt already exists at the root, some plugins will detect it and ask you to edit that file directly instead.

Whichever editor you use, paste your full rule set in, save, and verify the live URL. A robot.guard set is a clean fit here: build your config in the editor, toggle on the search bots you want to keep, switch on blocks for the AI scrapers from the maintained list, preview it live, then copy the generated text straight into the plugin field. Blocking AI crawlers like GPTBot or CCBot here does not affect your Google ranking, because Googlebot is a separate user-agent that you leave allowed.