Skip to contents

Read a sitemap into a tidy tibble of URLs

Source: R/read-sitemap.R
read_sitemap.Rd

Parses one or more sitemap sources — sitemap URLs or local sitemap files — into the tidy row tibble: one row per URL with loc, lastmod, changefreq, priority, the images/video/news/alternates extension list-columns, and source_sitemap provenance. Supported formats are XML urlset and sitemapindex, the plain-text format, transparent gzip (.xml.gz/.txt.gz), and — local files only — bounded .tar.gz archives.

Usage

read_sitemap(
  x,
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy()
)

read_sitemaps(
  x,
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy()
)

Arguments

x

One or more sitemap URLs or paths to local sitemap files (.xml, .txt, .gz, or .tar.gz).

user_agent

The User-Agent header for HTTP fetches. Defaults to the package User-Agent.

limits

Network limits for HTTP fetches, as from fetch_limits().

index_limits

Sitemapindex-expansion bounds (recursion depth and per-index child-count cap), as from index_limits(). Defaults to index_limits().

policy

A request policy applied to every HTTP hop (root, robots.txt, discovery, redirects, and index children). Defaults to the no-op policy.

Value

A tibble of URL rows with sources and problems attributes. An entry-point fetch failure or unsupported content raises a classed error condition.

Details

The result carries a sources attribute (the per-source fetch-metadata records) and a problems attribute (a tibble of non-fatal issues such as skipped archive members, unfetchable index children, or failed members of a submitted vector). read_sitemap() never returns a validation findings tibble; use validate_sitemap() for that.

A top-level sitemap index is expanded recursively (cycle-safe, depth- and count-capped) so every reachable child sitemap's rows carry per-child provenance; the bounds are configurable via index_limits.

When x contains more than one source, inputs are normalized, deduplicated, and capped using the submitted-list source-record policy. Per-source failures are recorded in the problems attribute and successful sources still contribute rows. Scalar calls keep the stricter historical behavior: an entry-point fetch failure or unsupported content raises a classed condition.

Examples

# Read a local sitemap file into a tidy tibble of URLs.
xml <- paste0(
  '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  '<url><loc>https://example.com/</loc>',
  '<lastmod>2024-01-01</lastmod></url>',
  '<url><loc>https://example.com/about</loc></url>',
  '</urlset>'
)
path <- tempfile(fileext = ".xml")
writeLines(xml, path)
read_sitemap(path)
#> # A tibble: 2 × 9
#>   loc    lastmod             changefreq priority images video  news   alternates
#>   <chr>  <dttm>              <chr>         <dbl> <list> <list> <list> <list>    
#> 1 https… 2024-01-01 00:00:00 NA               NA <NULL> <NULL> <NULL> <NULL>    
#> 2 https… NA                  NA               NA <NULL> <NULL> <NULL> <NULL>    
#> # ℹ 1 more variable: source_sitemap <chr>

# Read directly from a sitemap URL; a top-level index expands recursively.
# read_sitemap("https://example.com/sitemap.xml")