Use Robots.txt to Exclude Website From Google Search Results With Yoast

Use Robots txt to Exlude Website From Search Results With Yoast SEO Plugin JPG

In this video I want to show you how you can use the robots.txt file to exclude all the pages and posts from the Google search results. You can easily do this with the free Yoast SEO plugin.

In the last video I have also explained how you can exclude specific pages or posts from the search results. If you want to know how you can do that, make sure to check out the previous video as well.

Now let me walk straight to the steps that you must take to edit the robots.txt file.

Table of Contents

Install and activate Yoast SEO plugin

As you can see, I have now been logged in into the WordPress dashboard and the first thing you need to do is you have to make sure that you have to Yoast SEO plugin installed and activated. It’s a free plugin that you can install and activate in a few clicks.

Activate Yoast SEO plugin for WordPress

Go to robots.txt file in the File Editor

After you have activated the Yoast SEO plugin you need to go to the Tools option and click on it.

Select Yoast Tools option Activate Yoast SEO plugin for WordPress

Then you can see the File editor option over here and you need to click on it.

Select Yoast FIle editor Select Yoast Tools option Activate Yoast SEO plugin for WordPress

Now you can see this robots.txt file.

By default, it will be set up like this:

User-agent: * 
Disallow: /wp-admin/ 
Allow: /wp-admin/admin-ajax.php 

This means that the website can be indexed by crawlers from search engines for example.

edit robots.txt file with Yoast Select Yoast FIle editor Select Yoast Tools option Activate Yoast SEO plugin for WordPress

Change robots.txt file to exclude website from Google search results

If you want to make sure that is not the case, you want to exclude the complete website from search engines. Then you can replace this text. So, I’ll delete the old code and I copy and paste this code over in the field:

User-agent: * 
Disallow: / 

change code robots.txt file with Yoast edit robots.txt file with Yoast Select Yoast FIle editor Select Yoast Tools option Activate Yoast SEO plugin for WordPress

After that you need to click on Save changes and then the website cannot be crawled by crawlers from search engines. If you want to undo this then you have to replace the code in the field again to how it was before.

Everything is explained in this video tutorial that you can watch on YouTube over here.

In another tutorial I have explained how to optimize pages and posts with Yoast in combination with Elementor.

You can also exclude individual pages and posts with Yoast very easily.

hope this tutorial is helpful. It that’s the case make sure to subscribe to the channel and drop a like and comment.
Thanks, and I hope to see you back in the next tutorial!