Skip to contents

Discover a site's sitemaps from already-fetched bytes

Source: R/seed.R
sitemap_tree_from_bytes.Rd

Builds the same discovery tree as sitemap_tree(), but from sitemap bytes you have already fetched yourself rather than from a site root. This is the explicit-seed entry point for sources sitemap_tree() cannot fetch directly — for example a site behind a TLS-fingerprint bot wall, whose sitemap you retrieved through a browser or proxy. The root document is parsed in memory (no network request for it); if it is a sitemapindex, its child sitemaps are still fetched and expanded over the network, cycle-safe and depth- and count-capped, exactly as sitemap_tree() and read_sitemap() do.

Usage

sitemap_tree_from_bytes(
  bytes,
  source_url,
  user_agent = default_user_agent(),
  net_limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy()
)

Arguments

bytes

The raw sitemap document: a raw vector, or a length-1 character string of the sitemap text. Transparent gzip is inflated automatically.

source_url

The URL the bytes came from (length-1 character). Used as the root row's sitemap_url, as the cycle-detection identity for expansion, and as the parent attribution for expanded children.

user_agent

The User-Agent header for the child fetches an index seed triggers. Defaults to the package User-Agent.

net_limits

Network limits for the child fetches, as from fetch_limits().

index_limits

Sitemapindex-expansion bounds, as from index_limits(). Defaults to index_limits().

policy

A request policy applied to every index-child HTTP hop. Defaults to the no-op policy.

Value

A tibble with the same 8-column contract as sitemap_tree(): depth, parent_sitemap, sitemap_url, page_count, gzip, status, reason, and provenance.

Details

A leaf urlset/text seed performs no network access at all. The root row carries provenance = "seed"; expansion children carry "child-of-index". Content that cannot be parsed yields a single rejected seed row rather than an error.

See also

sitemap_tree() for discovery from a site root or an exact URL.

Examples

# Parse an already-fetched leaf sitemap with no network access.
xml <- paste0(
  '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  "<url><loc>https://example.com/</loc></url>",
  "<url><loc>https://example.com/about</loc></url>",
  "</urlset>"
)
sitemap_tree_from_bytes(xml, source_url = "https://example.com/sitemap.xml")
#> # A tibble: 1 × 8
#>   depth parent_sitemap sitemap_url     page_count gzip  status reason provenance
#>   <int> <chr>          <chr>                <int> <lgl> <chr>  <chr>  <chr>     
#> 1     0 NA             https://exampl…          2 FALSE accep… NA     seed