Site architecture optimisation: the key stages
Site architecture optimisation is a structured process running from defining the scope to quality control after implementation. The work begins with determining which sections of the site are covered and what the implementation constraints and the rules for approving changes are. Next come the inventory and the crawl, which feed the analyses of indexing, accessibility, navigation depth, URL structure, taxonomy, filters, pagination, internal linking, and canonicalisation and duplication signals. The result of these analyses is a diagnosis of the architecture problems and decisions on the target division into sections and hierarchy.
The final part of the process concentrates on designing the target architecture and translating it into implementation requirements. In practice, this produces URL and indexing rules and control over page generation, as well as an implementation specification for the dev team. If the structure changes, a redirect and URL migration plan is prepared, with old->new URL mappings and rules for handling removals and merges. After implementation, post-publication QA is carried out, and the results of the work go into the final report and the backlog of further actions, together with priorities.
- Start, and agreeing the scope and the approval workflow for changes
- Access and input materials: indexing and traffic data, and information about the CMS
- Site inventory and crawl (URLs, templates, page types, areas of scale)
- Analyses: indexing and accessibility, navigation depth, URLs, taxonomy, filters, pagination, linking, canonicalisation and duplication
- Diagnosis of the problems and decisions on the target hierarchy and the page types to be indexed
- Target architecture design, URL/indexing rules, implementation specification, redirect plan
- Implementation support and consultations, QA after implementation, final report and backlog
Start and defining the scope of work
Start and defining the scope of work means determining which sections of the site are covered (e.g. categories, blog, products) and what the approval workflow for changes will look like. At this stage the implementation constraints are also identified, as they can affect what can realistically be introduced in the templates, the navigation or the page generation rules. The scope and order of the work depend on factors such as the scale of the site, the number of language versions, the degree to which URLs are generated dynamically, the complexity of the filters, the availability of dev resources and planned changes to the content. The initial arrangements are the reference point for later decisions on what to develop, what to consolidate and what to restrict in indexing.
The scope can be defined once access and input materials have been confirmed, including access to indexing and traffic data, a list of key sections/URLs and information about the CMS and the page generation rules. If a test environment is available, it makes it easier to agree and verify changes safely before publication. The arrangements also clarify the typical risks of the cooperation, such as the lack of a test environment, limited influence over templates/the CMS, no possibility of bulk redirects or of modifying filter rules, and a long approval process. These conditions affect how implementation support, consultations and the later post-implementation QA are planned.
Access and input materials: the essential requirements
Access and input materials are needed so that the work on the site architecture can be based on indexing and traffic data and on the real rules by which the CMS operates. In practice this is about being able to check how crawlers reach individual page types and which areas of the site are key from the point of view of the structure. Information about how the site generates pages and URLs is just as important, because it affects the scope of the crawl and the later decisions on indexing control. If a test environment is available, it allows changes and their interpretation to be verified more safely before publication.
For this service, specific inputs are needed on the client’s side that directly make it possible to carry out the inventory and the crawl of the site. The input materials are used to build the list of URLs and page types and to establish which sections have the highest priority in the analyses. This makes it possible to work on a consistent data set and limits the risk of overlooking important areas generated dynamically by the CMS.
- Access to indexing and traffic data
- A list of key sections and/or URLs to be included in the work
- Information about the CMS and the rules for generating pages and URLs
- The possibility of working on a test environment (if one is available)
Site inventory and crawl: the baseline analysis
The site inventory and crawl is the baseline analysis, whose purpose is to collect a list of pages, templates and URL types and to indicate the areas of greatest scale. This stage makes it possible to establish which kinds of pages actually exist on the site and how they are generated. Particular attention is paid to the elements that usually increase the number of URL variants, such as filters, parameters and pagination. The result of the inventory is the starting point for the further verification of indexing, accessibility and the consistency of the structure.
During the crawl, URL patterns are identified, along with the places where the site can create large clusters of similar pages. On that basis it is possible to determine more precisely which URL types need separate rules for page generation and indexing control. The collected list of page types and URLs feeds the further analyses, including the verification of the readability and consistency of URLs and the assessment of the impact of filters, parameters and pagination on the structure. As a result, the following stages can focus on the priority areas instead of on random parts of the site.
Analysing the indexing and accessibility of pages
Analysing the indexing and accessibility of pages means checking which page types are indexed, which are blocked and where there are problems with crawler access. I verify errors, redirects and potential loops that can hinder the crawl and distort the interpretation of the site structure. The reference point is the previously identified URL types and templates, so the assessment concerns specific groups of pages and not individual cases. The results of this analysis are organised so that it is clear which problems require a priority response.
In this part of the work I compare how different areas of the site behave in terms of access and indexing, in order to point out the places that create a risk for the quality of the index. The irregularities identified are then aggregated into the diagnosis of the architecture problems together with their consequences, including in the context of wasted crawl budget, duplication and poor distribution of link equity. On that basis it is easier to establish which page types should be developed and indexed and which should be restricted. The conclusions from this stage are also the input for further decisions on indexing rules and control over page generation.
- Checking the indexing status of page types and URL patterns
- Identifying blocks and barriers to crawler access
- Detecting errors, redirects and loops that affect the crawl
- Organising the problems for further diagnosis and prioritisation
