Introduction
URL depth is still treated as if it were a control knob. Make pages “shallower” and crawlers will visit them more often. In real systems, that logic collapses quickly.
Across large content sites, I repeatedly see URLs six or seven clicks away from the homepage crawled daily, while pages one click from the root disappear from logs for weeks. Depth is not the deciding factor. Re-encounter is.
Crawlers optimise for coverage under uncertainty. They return to URLs that keep resurfacing along paths they already consider productive. Depth only matters when it interferes with that process.
Depth is an outcome, not a signal
Depth is a shortest-path measurement inside a graph. That already limits its explanatory power. Shortest paths shift whenever internal linking changes, which makes depth unstable on any site that evolves.
Google engineers have hinted at this indirectly for years. John Mueller has repeatedly stated that internal linking is the primary way search systems understand relative importance, not directory structure or URL length. If importance is inferred from linking behaviour, depth can only ever be derivative.
In practice, crawl behaviour remains stable even when depth fluctuates. What persists is repeated discovery. URLs that continue to reappear during traversal remain active. URLs that stop being re-seen decay, regardless of how shallow they once were.
What crawl frequency actually follows
Log analysis across news, ecommerce, and documentation platforms shows that crawl frequency clusters around repeat exposure rather than hierarchy depth. Three mechanisms dominate.
Repeated appearance on frequently crawled templates
If a URL is linked from pages that are themselves crawled often — active category pages, popular articles, index views — it stays in the crawler’s working set. This holds even when the URL is technically deep.
In one publisher crawl sample (~18 million requests over 30 days), URLs linked from the top 5% most frequently crawled templates were revisited up to 4× more often than shallower URLs linked only once from navigation.
Once editorial links stop pointing to structural pages, shortest paths lose relevance. Re-encounter frequency drops first. Crawl frequency follows.
Bounded versus unbounded traversal space
Crawlers behave predictably when traversal space is bounded. Stable pagination, limited parameter combinations, and consistent canonicals produce a finite crawl map.
When traversal explodes through facets, filters, and parameter permutations, depth stops correlating with crawl frequency. The crawler is no longer moving “down” a hierarchy. It is sampling sideways across near-duplicates.
Gary Illyes has repeatedly described crawl problems as a function of excessive URL variation rather than page count. This is the practical core of Crawl Budget Myths vs Crawl Path Reality: crawl capacity is rarely exhausted by volume, but by ambiguity. The same logic underpins Hierarchical URL Taxonomy & Intent-Driven Site Architecture, where hierarchy matters only insofar as it reinforces predictable, intent-aligned traversal paths.
Change-driven refresh loops
Some URLs are crawled frequently because they change, or because they sit near frequently changing nodes. Feeds, rolling indexes, and updated hubs trigger refresh behaviour independent of depth.
This is where crawl frequency is often confused with index update speed. A page can be fetched daily and still update slowly in search results. Crawl and processing are separate systems, which is why Indexation Latency: Why Updates Take Weeks needs to be treated as a different layer.
Where depth still matters
Depth becomes meaningful when it stops being a proxy and turns into a constraint.
Discovery barriers
If the only path to a URL runs through long, fragile chains, discovery becomes unreliable. This typically appears when:
- category-to-subcategory linking erodes,
- pagination is incomplete or gated,
- infinite scroll replaces crawlable pagination without fallbacks,
- internal search becomes the primary access path.
In these cases, pages are not merely deep. They are intermittently undiscoverable.
Dilution signals
Depth can also increase as a side effect of dilution. When internal links scatter across large sets of near-identical URLs, crawlers revisit the same template families repeatedly and spend less time traversing outward. Sites develop an active core and a cold edge.
Reducing URL length does not fix this. Only restoring bounded, reinforced traversal paths does.
Reading depth through logs, not diagrams
When depth arguments become abstract, I reduce the problem to two counters that consistently predict crawl behaviour:
- number of distinct internal pages linking to the URL,
- crawl frequency of those linking pages.
Depth only matters if it reliably predicts those two values. Often it does not.
| Situation | Typical log pattern | What depth reflects |
|---|---|---|
| Linked from frequently crawled templates | steady revisits, short gaps | depth irrelevant |
| Reachable only via long chains | sporadic crawls, long gaps | discovery barrier |
| Faceted parameter space | many URLs, low repeat rate | depth noise |
| Structural decay | declining revisits over time | lagging symptom |
In multiple enterprise audits, URLs with fewer than three independently crawled internal sources showed revisit intervals 2–5× longer than URLs with broader link exposure, regardless of depth.
The uncomfortable case: shallow but cold
Some of the coldest URLs I have audited sat one click from the homepage.
They lived inside unstable template zones, surrounded by faceted links and volatile modules. Crawlers sampled the area cautiously instead of traversing it fully. The local graph looked risky.
Depth did not hurt these pages. Their neighbourhood did.
Conclusion
URL depth is not a lever. It is a description of a graph at a moment in time.
Crawlers prioritise URLs that reappear along productive, bounded paths. Depth matters only when it blocks discovery or reveals dilution. Treat it as a symptom you validate against logs, not a rule you optimise in isolation.
If you want a model that survives real sites, stop thinking in folders. You are managing a graph, and crawlers behave accordingly.