Sometimes it is necessary that the pages of the site or the links placed on them do not appear in the search results. You can hide site content from indexing using a robots.txt file, HTML markup, or authorization on the site.
Prohibiting indexing of a site, section or page
If some pages or sections of the site should not be indexed (for example, from a service or confidential information), restrict access to them in the following ways:
- In your robots.txt file, specify the Disallow directive.
- In the HTML code of the site pages, specify the robots meta tag with the directive noindex or none. For more information, see the robots meta tag and the X-Robots-Tag HTTP header.
- Use authorization on the site. We recommend this method to hide the main page of the site from indexing. If home page is not allowed in the robots.txt file or using the noindex meta tag, but links lead to it, the page may end up in search results.
Prevent indexing of page content
Hide part of the page text from indexingIn the HTML of the page, add the noindex element. For example:
The element is not sensitive to nesting - it can be located anywhere in the HTML code of the page. If you need to make the site code valid, you can use the tag in the following format:
text to be prohibited from indexingHide from indexing the link on the page
In the HTML of the page, add the attribute to the a element. For example.
Greetings, dear friends! In today's article, I want to talk about how to check the indexing of a specific page on a site. Imagine that you have written an article and published it on a website. Before taking any action, you need to find out how the search engines rated this page at this stage of promotion.
To do this, we need to know if the page is indexed or not. If the page is indexed, we begin to improve it, select queries and analyze competitors, otherwise we speed up page indexing (). But how do you know if a page is indexed or not?
In general, you can use the most commonplace method. Open the webmaster's panel in Yandex and look at the indexed pages (similar to Google). Of course, the method is good, but, unfortunately, it has only one drawback, and it is very important for page promotion. The webmaster's panel is updated once a day, or even once every 2 days. This is not very convenient for a SEO optimizer, for whom it is very important to know the page quality score.
What does quality rating mean? Most SEO optimizers are promoted to low competitive search queries, so it is important to know what position the search engine will assign when the page is indexed for the first time. Depending on the position, you need to analyze the corresponding competitors standing higher in the search results.
Thus, I want to tell you about the tools that I personally use when promoting blogs.
Firstly, this is the YCCY program, which has been serving me for over 2 years. It is distinguished by its accuracy in checking page indexing. By the way, it is also one of the oldest SEO tools. Despite its age, it is still relevant and is used not only to check the indexing of pages, but also for a number of other purposes, such as batch checking all pages on the site, checking backlinks, and more. dr.
A search engine index is a database that stores information about web documents. The purpose of this "storage" is obvious - using a database, a search engine can quickly and accurately calculate the values of ranking factors, the numerical value of the relevance of a URL to a user's request, and generate search results.
For SEO specialists and site owners, this means one thing - how many pages are indexed, how many documents can be found in the search results for targeted queries, and the same number of URLs can attract traffic.
How to quickly check indexing in Yandex and Google?
There are a lot of ways and services for checking page indexing, and we have three tools that show the number of documents in the index by domain:
Just specify the domain in any of them. The tool will help check the site for indexing and tell you how many documents are in the database Google data and Yandex, with or without subdomains:
Search Operators for Checking Indexing
You can find out the indexing of a site using search operators. For Yandex, these are the operators "site:", "host:" and "url:".
For Google - "site:", "inurl:" and "info:".
In both search engines results are approximate, but you can use Extra options to find out how many pages were indexed, for example, in a week:
To check indexing a specific page site use the operator "url:" for Yandex and "info:" for Google.
Webmaster and Search Console
In the Webmaster, go to the "indexing" - "pages in search" tab. Here you can find statistics on documents added and removed from the search. You can also view the history of changes for the entire site for a specific date interval:
You can check the indexing of a site on Google using the Search Console. V new version the console, go to the tab "status" - "indexing of submitted URLs" and get a picture of changes, errors, excluded pages and a schedule of impressions:
Possible indexing control
It is important to understand that crawling or crawling a page on a site does not mean that it will be indexed. Sometimes it is necessary to prohibit indexing of pages that are not needed in the search, for example: URL with technical GET parameters, service or user documents (shopping cart, Personal Area and others), documents with UTM tags. Therefore, it is desirable for each site to have:
Using the Disallow command, you can prevent the indexing of specific URLs or sections of the site.
Crawl delay - sets the minimum time for a robot to contact the server to avoid loads on the site.
Clean param - allows you to specify CGI parameters that are not important for indexing in URLs, that is, various identifiers of users, sessions, and so on. Used only in Yandex.
Sitemap.xml - a file that helps search robots to crawl the site, taking into account the set priorities.
Robots.txt - a file that sets the rules and parameters for crawling for all search engine robots or individually for each:
Open pages should be in search, only then you can think about ranking. Do not forget to check the site for indexing, track changes and successful positions in the search results!
To understand whether a specific page of the site is in search and how many pages of your resource are generally seen search engines, you need not so much. Learn more about how to check the indexing of a site and get necessary information, we will explain in this article.Immediately, we note that when crawling a site, robots transfer all received information about the resource to separate databases. And then the search engine works on these bases.Note, however, that some inexperienced webmasters confuse the two concepts: crawling and indexing. Crawling is, in fact, a robot examining a site. It doesn't take much time to bypass the entire resource. But indexing - that is, adding information about a resource to databases - takes a relatively long time.
1) How to check the indexing of a site through the control panel in the service for webmasters
This method is basic. For Google service follow these steps:
- go to the service control panel;
- select the Search Console tab;
- then go to another tab "Google Index";
- click on the Indexing Status item.
For the Yandex service, the sequence of actions will be as follows:
- open Webmaster;
- find the Site Indexing item;
- select the line Pages in the search.
You can also go through another sequence of tabs - after indexing the site, click on the History tab, and then click on the Pages tab in the search. This will allow you to track how the number of indexed pages changes.
This method of verification is available only to those publishers for whom the corresponding access is open in the service toolbar.
The picture below shows a graph of a well-indexed site. As you can see, the number of pages in the index is constantly increasing.
And the picture below shows the case when there are problems with indexing or the pages of the sites are generally closed from indexing.
2) Checking by search operators
With their help, it is easy to clarify future results. For example, the "site:" parameter / operator is perfect for this purpose, by means of which you can see exactly how many pages are indexed.
For this in search bar(Yandex or Google, and better both there and there) the following combination is entered:site: site url
Note! A big difference in the results of the two search engines may indicate the presence of certain problems in the work of the site.
For example, the site cubing.com.ua, the data of which is shown in the screenshots below, is located under the AGS filter imposed by the Yandex search engine.
And here additional tools, which are in the search engines, make it possible to get a lot of interesting information. For example, how indexation has changed over a particular period of time.
3) Plugins and extensions
This is an automated verification method. In addition to plugins, you can use special extensions. They are also called bookmarklets, which are script programs. They are saved as regular browser bookmarks.
The appeal of scripts is that now you do not have to open the search engine again and again and enter some words or symbols. The program will do everything for you.
The picture above shows an example of how the plugin works. RDS bar ... The screenshot shows what data it shows when integrating into google browser Chrome. But in the browser Mozilla it provides even more data and capabilities.
In particular, such a plugin makes it possible to get data, both by separate page, and throughout the site as a whole.
Note! Plugins come in both paid and free versions. A clear flaw free versions- the need to constantly enter captcha. Terribly nervous!
The script / bookmarklet "Indexing Check" deserves good reviews. You just need to drag it to the panel of your browser. Then open the site you need and click on the script tab. This will activate a new browser tab showing all the data.
4) Special services
True helpers of all SEOs. It is possible to list such services for a long time - there are many of them. For example, a resource is allocated Netpeak Spider.
The main advantage of services is that you get a lot of additional information besides the basic data:
- data on the number of indexed pages;
- a complete list of all pages that are in the index;
- response codes;
- meta tags;
- robots.txt file and much, much more.
In addition, the service will provide information about errors on the site.
5) Checking the indexing of the site using Topvisor
Topvisor is the best service for monitoring and analyzing search engine promotion. To check the indexing, follow these steps:
After completing the registration, select the project you want to index.
Here you can create groups of keywords for convenience and use the service to determine the frequency of requests and see their basic, "phrasal" and "exact" values. In this case, "keys" can be entered manually, imported from a file, and also made a selection from Yandex.Wordstat and Google AdWords.
To check the positions, go to the "Positions" tab, select the desired search engine, for example Yandex, select the right group key queries and click on the green button in the upper left corner "update".
After a while, the data will be updated. In the report, you will see the positions of your site for certain requests and the frequency of requests for "keys". When any changes occur, for example, your site has moved up several positions in the search engine, then these changes will be displayed here. The option is also available automatic start checks at a specified time or depending on Yandex updates
Why there are problems with site indexing: a description of the main problems and ways to solve them
Let's take a look at the main reasons why a site might not be indexed. It often takes little time to fix errors, but after that the results search results improve significantly.
Young site
If your resource has very few days / weeks / months, you just need to wait. A little. This is how search engines work, that sites and pages do not get into the index immediately. Sometimes you have to wait several months.
Missing sitemap
It is impossible to achieve good results without a proper sitemap. Sitemap provides direct assistance to search robots, which significantly speeds up the speed of crawling a site, as well as getting its pages into the index. The easiest way to add a link to a sitemap is through the control panel in the webmaster service.
Presence of errors
Watch the alerts carefully when visiting the webmaster service. There, a message constantly appears about the presence of certain shortcomings, errors. Go through them and don't forget to fix them.
The robots issue
The robots meta tag is essential for proper site indexing. Unconsciously changing the settings on the hosting or your CMS leads to the appearance of the following entry in the code.
Which means that the page is simply not indexed.
Error with robots.txt file
Written correctly is a guarantee of good indexing. It is not uncommon to find tips and tricks that can be closed for search robots through this file all unnecessary.
However, a unique feature of the file is that only one incorrectly placed character can generally close the site for indexing by search engines.
Even if you were trying to close just one page, it may turn out that with one click you have covered several pages, etc.
Take a close look at your file. Is there a record in itUser-agent: * Disallow: /? If it is present, then the site is closed for indexing.
And a few more reasons why the site is not indexed
Among the main problems with indexing are:
- the presence of duplicate pages;
- use of non-unique materials when filling the site;
- low website loading speed and problems with access to it.
How to speed up website indexing in Yandex and Google?
The main goal of the site owner is to ensure that all pages open to search robots are indexed. To achieve such a result, frankly, is very, very difficult.
In addition, the entire process of adding pages to the index should be constantly monitored. Any changes - abrupt - whether positive or negative, signal the presence of specific problems that interfere with the normal indexing of the resource.
To check the indexing of the site, use any (or better sequentially each) method described in our article:
- the toolbar in the webmaster service;
- search operator / parameter site:;
- plugins and scripts;
- specialized online services.
Bonus for those who really want to achieve results
Introducing special table , which will help you when working with site indexing.
It is very, very simple to work with such a table. Here is a detailed sequence of actions:
- make a copy of the table with columns, as in the screenshot;
- select a domain zone;
- load the website URL into column A;
- wait until all URLs are loaded - the more there are, the longer it will take for the result.
Columns B and C can be copied, which will allow you to track the dynamics of page indexing.
You have not yet been trained on my author's course "How to create a website in 4 days"? Follow the link: and register right now, or you can use my instructions, where I show you step by step! Come, it will be interesting!
Good afternoon, dear reader ... Let's continue with my book on search traffic for bloggers and talk about indexing a resource today. In this article, you will learn how you can quickly and easily check the indexing of your website pages in Yandex and Google using regular search operators and various services. You will understand the importance of this process, you will see in different examples several options for defining documents for your blog. Of course, this post has practical value, but it is the first manual to study in order to further engage in search engine optimization of your web resource.
Checking the presence of a site in the index of search engines
Before we talk about various services and services for checking indexed pages in Yandex and Google, I suggest recalling the importance of the database itself. The fact is that any search engine works only with those documents that are viewed by its robot and entered into the database. This database is called an index, and the process of collecting all data from the pages of any site on the network is called indexing. This process is quite complex, but exciting and educational. And a correct understanding of the work of this process is one of the main conditions for a successful one both in Yandex and Google. Therefore, before further studying the material in this article, I recommend that everyone remember.
Checking the indexing of site pages in Yandex and Google
Yandex.... In order to find out the availability of pages in the Russian search engine, there are several operators. The operator site finds all the pages of your site in the Yandex database, and host provides information on indexed documents from domains and subdomains of the site. Here are the results of the site operator:
And here is the final result after using the host operator:
Now let's see how many indexed pages the host statement for my blog will show us not on the main mirror (without www). The result will of course be predictable:
If we follow the same scheme but for the site operator, then the number of pages in the index not according to the main mirror of my site will not change - it will remain 206. This is because, as I said, site finds all the documents of the site, and host finds the domain. Therefore, I will give you this advice - if you want to quickly estimate the number of pages of any web resource in the Yandex database, use the site operator (you can enter the site address with or without www in the query string - the result will be the same). If VM needs to know the exact number of pages in the index of any domain, use the following combination of operators (instead of my blog address in the example, put the desired domain address):
Google... The most popular search engine has one operator to identify which pages are indexed. Its name is site and it performs the same function as its colleague in Yandex - it gives a complete list of documents located in the Google database. This is what it looks like:
Unfortunately, this operator shows all pages that are in Google's index. But as you know, this search engine divides its database into a main index and an additional one. Basically contains all the documents on your blog that Google uses for its SERPs. The rest of the documents, which usually contain duplicate pages, technical sections from the CMS or URLs that are not suitable for ranking, are in an additional index called Supplemental Results (in SEO - "snot"). Therefore, to accurately determine the working indexed pages of your blog that are involved in ranking, the special command & is used:
As you can see, the results are very different in my case - the number of indexed pages in the main index is much less than the total. This suggests that at one time, during the first runs by a search robot on my blog, Google managed to add a bunch of garbage and duplicates to its database. And before I had time to discover it, their mass far exceeds the real pages of my site. Which of course affected the search traffic from Google. Therefore, it is so important, before the very first launch of your new web resource, to completely close all unnecessary sections of your site in the robots.txt file. But we'll talk about this another time.
Checking page indexing using services
RDS bar... The fastest way to check is to use (post on how to set it up and how to use it). This plugin that can be screwed to any popular browser, on the fly can give all the information about the presence of pages in the Yandex and Google databases (in the general index), as well as information about the date of indexing of the current page. Thanks to this extension, without wasting time on transitions to various services or to search, you can evaluate the indexing of any site. In general, I recommend it, it's very convenient:
XSEO Service... A set of tools for webmasters and bloggers, XSEO.in also gives clear information about the indexing of site pages in Yandex and Google. But unlike the previous service, it allows you to quickly enough, in a couple of clicks, see the number of pages in Google in different indexes - mainly and additional. Not every plugin, program or service contains such information. XSEO gives accurate data on the number of documents, such as for my blog:
To obtain such data for any web resource you are analyzing, it is enough to select the section "Checking site indexing" in the service.
PR-CY service... The well-known analytical service Pr-Cy.ru, which has a fairly good set of various reports and seo-tools. Like the previous sources, it also provides data on the indexing of any site. But it has a remarkable difference - it has interesting graphics in which it shows the history of page indexing in search engines. I rarely noticed such a thing anywhere, but this particular service has the most user-friendly interface that allows you to quickly see the change in the number of indexed pages. For example, this is the history of getting my blog documents into the Yandex database:
As you can see from this graph, the service clearly shows the number of indexed pages in the Russian search engine during the year. Thus, you can evaluate the indexing of any site, especially if you choose a good donor for the purchase of external links. For example, in, By the way, the graph clearly shows a jump in the number of indexed documents in May and then a return to the previous number. This is all because in April I changed the design of my site and my page structure completely changed. Yandex reacted quickly enough, but in due time the delivered redirect put everything back in place.
With Google, the story is a little different for me. Here is a graph of the indexing of pages on my blog, which began exactly a year ago:
As you can see, the growth was slow, but then I started writing more articles and started experimenting with tags. As a result, while I was deleting them, Google honestly scanned all the duplicates and sent them to their database. As a result, the positions of my blog fell and until the beginning of June this year the visibility of my site in this search engine did not reach even 2%. That is, there was no Google traffic on my blog. But as you can see, the number of pages in the additional index has already decreased. And this trend continues at the moment - it's all about one of my experiments to remove duplicate pages from the database. At the moment, the visibility of my blog on Google has risen to 21%, and the material on the experiment is increasing - for the future I have prepared my post on working with duplicates and excluding them from the Google index.
Checking site indexing in Yandex webmaster and Google Tools
For dessert, I left the most interesting services that can give complete information about the indexing of your web resource. Of course, I mean the webmaster toolbars from Yandex and Google. Only they have all the exact data on the presence of your site pages in their databases. So let's turn to the original source!
Service from Yandex... As I said, this is the best package of tools that will show you all the information about the indexing of your website pages (and not only). I use it from the very beginning, as soon as I sent my blog to be devoured by a Russian search engine -. Over the entire period of operation, I have repeatedly used the cool reports that this service provides.
In this article, I will not explain all the details of indexing reports - they are detailed in the. See the "Site Indexing" section. If something is not clear, the service has a wonderful help. As a last resort, if you don't understand the analytical data of the reports yourself, write your questions in the comments of this article.
Service from Google... Like Yandex,. And all the data in it provide the most reliable information, so to speak, first-hand. Therefore, to obtain the most accurate data on the indexing of site pages in this search engine, I recommend using the subsection "Indexing Status" in this service.
When you first enter this report, you will see a single chart, where its elements (individual points on it) are total numbers indexed total site pages in the Google index. This is the number of all documents that are in two indices - primary and secondary.
But of course, this is not all the information about indexing - for this you need to click the "Extended data" button and enable all empty check-boxes (check the boxes). As a result, three more are added to the first graph (the picture is clickable):
So, what do these graphs show and how we can benefit from their indicators:
indexed all time Is the number of all pages that it once found search robot Google. Typically, the values of the anchor points of this graph are much larger than the number of indexed entire documents. This suggests that in addition to periodically appearing new pages, Google ate during the scan all useless pages on your blog, and possibly duplicates. As in my case.
blocked by robots.txt file- the number of forbidden pages for indexing, for which the forbidding directives are registered in the robots.txt file. In theory, a number is indicated here that Google did not enter into its index due to the ban. But in fact, it is not accurate, because this search engine perfectly sees all pages of any site that do not give a 404 error. And even if a directive to prohibit crawling is spelled out for them.
deleted- the number of pages removed by request from the search results.
As you can see from the figure, each point on any graph provides all information on four indicators for a certain period of time (day - Sunday, when weekly data is updated in the reports of the Google webmaster). This information is shown on a separate number plate for each chart.
What you can see in the Indexing Status report
First, by looking at the data received on the "blocked in robots.txt" graph, you can assess whether your forbidden directives are working correctly. If the yellow graph constantly tends to the top, this indicates an obvious problem in the robots.txt file. In this case, it is worth carefully checking each directive. You may find errors or you will have to write new rules for the search robot.
Secondly, with properly configured indexing, the number of indexed pages should increase all the time. If this is not observed, then Google is not collecting information from your site correctly. There may be errors in the same robots.txt file, duplicate content appeared, Google began to collect unnecessary garbage due to imperfect CMS, etc. But in my case, the blue chart was at first jerky, and then, having gained a maximum, it began to decline rapidly at the end of July this year. This suggests that duplicates began to disappear from the index (I wrote about this above).
Thirdly, if there are jumps in the graph of indexed pages, which show a strong decrease in documents in the index, then you have problems accessing the web resource. Perhaps there is an overload in work due to the increase, or more pages with a 404 error have appeared (for example, with a sharp change in the structure), etc. In this case, you need to urgently take action, otherwise you can get a massive drop in indexed pages in Google. And this entails a collapse in search traffic.
Fourth, a large number of duplicate pages (a huge difference between the data of the red and blue graphs, if all other variants of errors are excluded) indicates that the canonical versions of pages are incorrectly configured on your site or there is duplicate content. If you do not take the necessary measures in time, Google collects a huge number of duplicates in its index, which are then very difficult to pull back. Plus, duplicate pages in no way contribute to increasing the visibility of the site, but rather the opposite - positions for many queries can drop.
On this my educational program on how to quickly evaluate the indexing of the pages of your site, I consider complete. As I said, if you have small questions, ask them in the comments. If you need a graph "Indexing Status" for the answer, leave a link to the screenshot with it. If your question requires the close attention of a seo-specialist, then my consultations will always come to the rescue.
Respectfully yours, Maxim Dovzhenko
