Skip to contents

Discover a site's sitemaps as a tree

Source: R/sitemap-tree.R
sitemap_tree.Rd

Discovers a site's sitemaps and returns them as a tree: one row per evaluated sitemap, accepted or rejected, with the columns needed to explain the result. Two input modes select what x is:

Usage

sitemap_tree(
  x,
  from = c("root", "sitemap"),
  use_robots = TRUE,
  use_known_paths = TRUE,
  user_agent = default_user_agent(),
  limits = discovery_limits(),
  net_limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy()
)

Arguments

x

A single URL (character). With from = "root" a site-root URL (a bare host is accepted and normalized to https:// via the shared site-entrypoint policy); with from = "sitemap" an exact sitemap URL.

from

Input mode: "root" (default) treats x as a site root and runs discovery; "sitemap" treats x as an exact sitemap URL and fetches only that.

use_robots

When from = "root", fetch robots.txt and add every Sitemap: directive it lists (provenance "robots"), deduplicated against the guessed paths. Default TRUE. Robots rules (Disallow/Allow) are never applied — only the Sitemap: directive is read.

use_known_paths

When from = "root", try the guessed-path catalog (provenance "guessed-path"). Default TRUE. Set both use_robots and use_known_paths to FALSE for an empty tree.

user_agent

The User-Agent header for HTTP fetches. Defaults to the package User-Agent.

limits

Discovery limits (the candidate cap), as from discovery_limits(). Used only when from = "root".

net_limits

Network limits for the per-candidate fetches, as from fetch_limits().

index_limits

Sitemapindex-expansion bounds (recursion depth and per-index child-count cap), as from index_limits(). Defaults to index_limits().

policy

A request policy applied to every HTTP hop (root, robots.txt, discovery, redirects, and index children). Defaults to the no-op policy.

Value

A tibble with one row per evaluated sitemap (and per expanded index child) and columns depth, parent_sitemap, sitemap_url, page_count, gzip, status, reason, and provenance. Accepted rows carry a page_count/gzip; rejected rows carry their rejection reason and NA page count.

Details

from = "root" (default)

x is a site-root URL. sitemap_tree() reads robots.txt Sitemap: directives (provenance "robots") and tries the guessed-path catalog (every generic guess first, then WordPress/Shopify), classifying each candidate. A guessed path that 404s is a rejected not-found row, never a validation finding, and a single unreachable guess never fails the call. Robots rules (Disallow/Allow) are never applied — only the Sitemap: directive is read (ADR-006). Turn either source off with use_robots / use_known_paths.

from = "sitemap"

x is an exact sitemap or sitemapindex URL. sitemap_tree() fetches that one URL (no catalog, no guessing) and, if it is a sitemapindex, expands it. The root row carries provenance "seed". A fetch or parse failure yields a single rejected seed row.

In both modes each accepted candidate's page count and gzip flag come from the body fetched during discovery, so no source is requested twice, and an accepted sitemapindex is expanded recursively (cycle-safe, depth- and count-capped), contributing deeper rows parented to the index with provenance "child-of-index". To discover from sitemap bytes you already fetched yourself (no network for the root), use sitemap_tree_from_bytes().

See also

sitemap_tree_from_bytes() to discover from already-fetched bytes.

Examples

# Discover a site's sitemaps from its root URL as a tree of candidates.
# sitemap_tree("https://example.com")

# A bare host is accepted and normalized to https://.
# sitemap_tree("example.com")

# Fetch and expand one exact sitemap URL, skipping the guessed-path catalog.
# sitemap_tree("https://example.com/sitemap_index.xml", from = "sitemap")