Crawl budget optimisation

Access and materials for crawl budget analysis are the set of data without which it is not possible to reliably assess where the bot wastes time and how to direct the crawl to the right URLs. The work requires, above all, server logs (if available), sitemaps and a list of the key sections of the site that are to be the indexing priority. Information about the rules for generating URLs is important too, because these are what often determine the creation of technical variants (e.g. parameters). In addition, access to the robots configuration and the ability to verify the implementation are necessary, in order to confirm that the settings are consistent with the change plan.

The range of input materials directly affects the precision of the diagnosis and the way the analysis is carried out.

When logs are missing, it is harder to confirm unambiguously the real distribution of the crawl at the level of URL types, and the recommendations are then based on indexing data and URL tests. That is why at the beginning we establish what data can be obtained and in what form it can be shared for analysis.

• Server logs (if available).
• Sitemaps.
• List of the key sections of the site and the indexing priorities.
• Information about the rules for generating URLs.
• Access to the robots configuration and the ability to verify the implementation.

Collecting crawl and indexing data consists of obtaining and organising information about the URLs that affect where the bot goes and what ultimately has a chance of being indexed. I organise the data on HTTP statuses, canonicalisation, indexing directives and which URLs are in the sitemaps. In parallel I analyse internal linking patterns, because they often determine whether the bot reaches priority pages or technical variants. The scope of this inventory grows with the size and complexity of the site, including the number of URLs, parameters, filters and language versions.

As part of the input data I create an inventory of page types and URL patterns, such as categories, products, pagination or URLs with filter parameters, together with an assessment of their role in indexing. I then compare the number and quality of URLs in the sitemaps with the URLs actually indexed, in order to identify sections with excess or insufficient indexing. I also verify whether the sitemaps contain only URLs that are canonical, indexable and have the correct HTTP status, and whether they are logically divided, e.g. by content type. Finally, I check the consistency of robots.txt, noindex, canonical and X-Robots-Tag headers with the sitemaps, and assess whether the linking generates parameterised URLs on a mass scale, creates paths that are too deep or leaves orphaned URLs.

The implementation plan and prioritisation of changes means arranging the recommendations in order of execution, together with the reasoning for “what to change, where and why”. I create the plan on the basis of the agreed policy for URL types (what is to be indexed and what restricted in the crawl), so that the technical and configuration actions are consistent with one another. For each change I also describe the risks and dependencies, so as not to cut off pages needed for indexing and not to introduce contradictory signals. I match the order of work to how the release process works on your site and to the technical dependencies.

The result of the planning is an implementation backlog, i.e. a list of tasks ready to be handed over for implementation and verification. Each task includes a priority, a description of the implementation, an acceptance criterion and a way of checking the effect, e.g. which URLs are to stop being crawled and which are to become canonical. At this stage I take into account that some changes may require development work or modifications in the CMS, and that the pace of delivery will depend on the availability of resources and the release cycle. The scope of the backlog may also grow with the size and complexity of the site, e.g. with a large number of URLs, parameters, filters or language versions.

• Change plan: what to change, where, why, with what risk and in what order.
• Implementation backlog: priority, description of the implementation, acceptance criterion and method of verification.
• Taking into account the technical dependencies, the implementation process and the availability of the team making the changes.