The digital landscape thrives on visibility. However, not all content should be publicly accessible through search engines. Certain pages – like admin areas, thank you pages, staging environments, or content intended for members only – are best kept out of search results. Fortunately, WordPress, coupled with the powerful Yoast SEO plugin, provides robust tools to control indexing and ensure only the desired content appears in search engine results pages (SERPs). This guide will explore the various methods for removing specific pages, content types, and even entire sites from search engine indexes using Yoast SEO, providing a detailed understanding of the “what,” “why,” and “how” of this crucial SEO practice.
Understanding Indexing, Noindex, and Nofollow
Before diving into the practical steps, it’s essential to grasp the core concepts. Indexing refers to the process by which search engines like Google discover, crawl, and store information about web pages in their index. This index is then used to deliver search results. By default, search engines will index most publicly accessible pages on a website.
However, website owners often need to exert control over this process. This is where the noindex tag comes into play. The noindex meta tag is a directive that tells search engines not to include a specific page in their index. The page can still be crawled, but it won’t appear in search results.
Closely related is the nofollow attribute. nofollow instructs search engines not to follow the links on a page. While often used in conjunction with noindex, it’s a separate directive. Using both noindex and nofollow (noindex, nofollow) is often recommended for pages you want to completely exclude from search engine consideration.
These directives are communicated to search engines through the <meta name="robots" content="noindex, nofollow"> tag placed within the <head> section of a page’s HTML. Yoast SEO simplifies this process, eliminating the need for direct code editing.
Why Remove Pages from Search Results?
There are numerous reasons why you might want to prevent certain pages from appearing in search results. Here are some common scenarios:
- Duplicate Content: Search engines penalize websites with duplicate content. Removing duplicate pages improves your site’s overall SEO health.
- Thin Content: Pages with little or no valuable content can negatively impact your rankings.
- Private Content: Member-only areas, internal documentation, or checkout pages should not be publicly accessible.
- Staging/Development Sites: Preventing indexing of staging sites avoids confusion and potential ranking issues.
- Thank You Pages: These pages typically offer little value to search users.
- Archived Content: Outdated or irrelevant content can dilute your site’s authority.
Utilizing Yoast SEO for Page-Level Exclusion
The most straightforward method for removing a specific page from search results is through Yoast SEO’s individual page settings.
- Access the Page/Post Editor: Open the WordPress editor for the page or post you wish to exclude.
- Locate the Yoast SEO Metabox: Scroll down to the Yoast SEO section, typically found below the content editor.
- Navigate to the Advanced Tab: Click on the "Advanced" tab within the Yoast SEO metabox.
- Allow Search Engines Setting: Find the "Allow search engines to show this Post in search results" dropdown menu.
- Select "No": Choose "No" from the dropdown. This will add the
noindextag to the page’s HTML. - Should Search Engines Follow Links: It’s generally recommended to also disable the "Should search engines follow links on this Post" option. This adds the
nofollowattribute. - Save Changes: Update the page to save your changes.
This process effectively tells search engines to exclude the page from their index and not follow any links on that page. Note that by default, only Editors and Administrators have access to the Advanced tab.
Managing Noindex for Different Content Types and Taxonomies
Yoast SEO allows you to control indexing at a broader level, applying noindex tags to entire content types (Posts, Pages, Products) and taxonomies (Categories, Tags).
- Access Yoast SEO Settings: In your WordPress dashboard, navigate to SEO > Settings.
- Select Content Type/Taxonomy: Choose the content type or taxonomy you want to modify from the left-hand sidebar (e.g., Pages, Posts, Categories).
- Show X in Search Results: Locate the "Show X in search results" section.
- Toggle the Switch: Toggle the switch to the "off" position to exclude the content type or taxonomy from search results. The toggle will turn gray with an "X" mark.
- Save Changes: Click "Save changes" at the bottom of the screen.
This is particularly useful for excluding archive pages (category, tag, author archives) that may not offer unique value to search users. Google has become better at handling paginated series, so the feature to noindex subpages of archives has been removed from Yoast SEO.
Bulk Noindexing with Yoast SEO
For managing noindex tags across Custom Post Types (CPTs) or specific categories/tags, Yoast SEO offers a bulk editing feature.
- Access Yoast SEO Settings: Navigate to SEO > Settings in the WordPress dashboard.
- Choose Post Type/Taxonomy: Select the CPT or taxonomy you want to hide.
- Toggle "Show {your CPT/taxonomy term} in search results": Switch off the toggle to prevent indexing.
- Save Changes: Save your changes.
This method streamlines the process of applying noindex tags to large groups of content.
Noindexing Your Entire Site
In rare cases, you might need to prevent search engines from indexing your entire website. This is typically done during development or when migrating to a new domain.
- Access Reading Settings: In the WordPress dashboard, go to Settings > Reading.
- Discourage Search Engines: Check the box labeled "Discourage search engines from indexing this site."
- Save Changes: Click "Save Changes."
This adds a noindex meta tag to every page on your site. However, it’s important to remember that this is a request to search engines, and they may not always honor it.
Alternative Methods: Manual HTML Editing
While Yoast SEO provides a user-friendly interface, you can also manually add the noindex tag to a page’s HTML code.
- Access Theme Editor: Go to Appearance > Theme Editor.
- Edit header.php: Locate and edit the
header.phpfile. - Add Meta Tag: Insert the following meta tag within the
<head>section:<meta name="robots" content="noindex, nofollow"> - Save Changes: Save the file.
This method requires technical expertise and is generally not recommended for beginners.
Comparing Methods: Yoast SEO vs. Manual Editing
| Feature | Yoast SEO | Manual HTML Editing |
|---|---|---|
| Ease of Use | Very Easy | Difficult |
| Technical Skill Required | None | High |
| Risk of Errors | Low | High |
| Flexibility | High (Page-level, Content Type, Taxonomy) | Limited |
| Maintenance | Easy | Difficult |
| Scenario | Recommended Method |
|---|---|
| Excluding a single page | Yoast SEO (Page-Level) |
| Excluding all pages of a specific content type | Yoast SEO (Content Type) |
| Excluding an entire category archive | Yoast SEO (Taxonomy) |
| Temporarily preventing indexing during development | Yoast SEO (Site-Wide) |
| Making a quick, one-time change without a plugin | Manual HTML Editing (Not Recommended) |
Key Terminology
- Crawl: The process by which search engines discover web pages.
- Index: The database of web pages maintained by search engines.
- Meta Tag: A tag used to provide metadata about a web page.
- Noindex: A directive that tells search engines not to index a page.
- Nofollow: A directive that tells search engines not to follow links on a page.
- SERP: Search Engine Results Page.
- Taxonomy: A method of classifying content (e.g., categories, tags).
Final Thoughts
Controlling which pages appear in search results is a critical aspect of SEO. Yoast SEO provides a powerful and user-friendly toolkit for managing indexing, ensuring that only valuable, relevant content is visible to search engines and users alike. By understanding the concepts of indexing, noindex, and nofollow, and by leveraging the features within Yoast SEO, you can optimize your website’s visibility and improve its overall search engine performance. Taking the time to carefully manage your site’s indexability will contribute significantly to a healthier, more effective online presence.