Audit a sitemap in a single pass

Resolves each sitemap source — a sitemap URL or a local sitemap file — just ONCE and derives both the tidy URL table and the validation findings from that single resolved artifact, returning them together as a sitemap_audit() object. It is the single-pass equivalent of calling read_sitemap() and validate_sitemap() separately: it fetches every remote root (and every sitemap-index child) once instead of twice, and the two projections are guaranteed to describe the same snapshot of the source.

Usage

audit_sitemap(
  x,
  mode = c("strict", "non-strict"),
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  collect = TRUE,
  on_urls = NULL,
  max_active = NULL
)

Arguments

x: One or more sitemap URLs or paths to local sitemap files (.xml, .txt, .gz, or .tar.gz).
mode: "strict" (the default) or "non-strict", passed through to the findings projection exactly as validate_sitemap() uses it.
user_agent: The User-Agent header for HTTP fetches. Defaults to the package User-Agent.
limits: Network limits for HTTP fetches, as from fetch_limits().
index_limits: Sitemapindex-expansion bounds (recursion depth and per-index child-count cap), as from index_limits(). Defaults to index_limits().
policy: A request_policy() applied to every HTTP hop (root, robots.txt, discovery, redirects, and index children) — configure custom headers, authentication, a proxy, TLS options, retry/backoff, and per-host throttling there. Defaults to the no-op policy.
collect: When TRUE (the default) the tidy URL rows from every source are collected into the returned urls component, exactly as before. When FALSE the audit runs in STREAMING mode: each completed leaf sitemap's rows are handed to on_urls (if supplied) and then discarded, so peak retained-row memory is bounded to a single leaf rather than the whole hierarchy. This lets a valid protocol-scale sitemap index be processed without materializing one combined in-memory table.
on_urls: Optional streaming callback function(rows, source) invoked ONCE per completed leaf sitemap with that leaf's tidy rows and its per-source fetch-metadata record. Supplying it activates streaming mode (as if collect = FALSE). An error thrown by the callback aborts the audit cleanly with a classed sitemapr_stream_callback_error condition naming the leaf; the accumulator is not left half-updated.
max_active: Optional worker cap for opt-in bounded-concurrency expansion of a sitemap index (ADR-008). NULL (the default) keeps child fetches sequential; a larger value (a small, polite bound such as 4–6 is typical) fetches up to that many independent child sitemaps at once. Concurrency changes only when a child's bytes arrive, never the row order or budget-truncation point — the result is byte-identical to sequential mode.

Value

A sitemap_audit() object: a classed list with the components urls, findings, sources, problems, and tree. Access them with audit_urls(), audit_findings(), audit_sources(), audit_problems(), and audit_tree(). In streaming mode (collect = FALSE or a supplied on_urls) the urls component is the empty row schema — the rows were streamed out per leaf — and carries the total number of streamed rows as its "streamed_row_count" attribute; findings, sources, problems, and tree are UNCHANGED from a collected audit, so index-protocol findings stay complete because they are derived incrementally per leaf.

Details

audit_urls() on the result equals read_sitemap() on the same source, and audit_findings() equals validate_sitemap() on the same source and mode. The audit also carries the per-source fetch metadata, the non-fatal parse problems, and — for an expanded sitemap index — the discovery tree. A source-level failure (an unreachable source, a corrupt archive, an unsupported root) is recorded as a problem and a fetch-layer finding rather than aborting the whole audit, so every source stays attributable.

Examples

xml <- paste0(
  '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  '<url><loc>https://example.com/</loc>',
  '<lastmod>2024-01-01</lastmod></url>',
  '<url><loc>https://example.com/about</loc></url>',
  '</urlset>'
)
path <- tempfile(fileext = ".xml")
writeLines(xml, path)

# One pass yields both the URL table and the findings.
audit <- audit_sitemap(path)
audit_urls(audit)
#> # A tibble: 2 × 9
#>   loc    lastmod             changefreq priority images video  news   alternates
#>   <chr>  <dttm>              <chr>         <dbl> <list> <list> <list> <list>    
#> 1 https… 2024-01-01 00:00:00 NA               NA <NULL> <NULL> <NULL> <NULL>    
#> 2 https… NA                  NA               NA <NULL> <NULL> <NULL> <NULL>    
#> # ℹ 1 more variable: source_sitemap <chr>
audit_findings(audit)
#> # A tibble: 1 × 10
#>   code        severity layer subject_type subject_ref message evidence     mode 
#>   <chr>       <chr>    <chr> <chr>        <chr>       <chr>   <list>       <chr>
#> 1 PROTOCOL_L… info     prot… entry        sitemap://… <lastm… <named list> stri…
#> # ℹ 2 more variables: is_strict_only <lgl>, remediation_hint <chr>

# Streaming: consume each leaf's rows via a callback instead of collecting
# them. `urls` comes back empty, with the streamed row count as an attribute.
seen <- 0L
streamed <- audit_sitemap(
  path,
  collect = FALSE,
  on_urls = function(rows, source) seen <<- seen + nrow(rows)
)
seen
#> [1] 2
attr(audit_urls(streamed), "streamed_row_count")
#> [1] 2

Usage

Arguments

Value

Details

See also

Examples