Construct a request-policy for the fetch boundary

A request-policy is the single, constrained extension seam for customizing the httr2 requests sitemapr issues for every hop (root, redirect, discovery, robots, and index-child requests). It carries typed fields for custom HTTP headers, authentication, a proxy, and TLS/curl options, plus a generic prepare hook for anything not covered by the typed fields. Every field defaults to NULL, so the default policy is a byte-identical no-op and existing callers behave exactly as before.

Usage

request_policy(
  prepare = NULL,
  headers = NULL,
  auth = NULL,
  proxy = NULL,
  tls = NULL,
  retry = NULL,
  throttle = NULL,
  max_active = NULL
)

Arguments

prepare

Optional function(req, ctx) receiving the built httr2 request plus a hop-context list (currently the resolved hop url) and returning the possibly-modified request. Applied last of the caller-facing steps, so it composes on top of the typed fields.

headers

Optional named list or named character vector of HTTP headers to add via httr2::req_headers().

auth

Optional authentication object from request_auth_basic() or request_auth_bearer(), applied via httr2::req_auth_basic() / httr2::req_auth_bearer_token().

proxy

Optional proxy: either a proxy URL string or a request_proxy() object, applied via httr2::req_proxy().

tls

Optional named list of curl/TLS options (e.g. list(ssl_verifypeer = 0L)) passed through httr2::req_options(). Cannot override sitemapr's redirect controls, which are re-asserted afterwards.

retry

Optional request_retry() object enabling bounded retry with exponential backoff. NULL (the default) means a single attempt per hop — byte-identical to the pre-retry behavior. When set, only TRANSIENT failures retry: the retryable HTTP status set (default 429, 500, 502, 503, 504) and, when retry_on_failure = TRUE, transient transport errors. Retries happen inside a single hop's req_perform(), so they never inflate the redirect/hop count or consume the redirect budget. Deterministic failures are NEVER retried: SSRF rejections (raised before the request), malformed input (classified downstream), and resource-ceiling aborts (raised after the response is buffered) all sit outside the retried code path. Any Retry-After header is honored but BOUNDED by the configured max backoff.

throttle

Optional request_throttle() object enabling host-aware request pacing. NULL (the default) means no pacing, byte-identical to the pre-throttle behavior. When set, requests are keyed by canonical host:port, so different origins pace independently and a redirect onto another host pays that host's pace, not the origin's. Pacing runs around each request (after the per-hop SSRF guard and sitemapr's re-asserted transport controls), so it never weakens those safety semantics.

max_active

Optional worker cap enabling opt-in bounded-concurrency expansion of a sitemap index (ADR-008). NULL (the default) keeps child fetches sequential — byte-identical to the pre-concurrency path — and max_active = 1 is observably identical to that default. A larger value (a small, polite bound such as 4–6 is typical) fetches up to that many independent child sitemaps at once while the per-host throttle still paces each origin. Concurrency changes only when a child's bytes arrive, never the row/finding/tree order or the budget-truncation point: the output is byte-identical to sequential mode.

Value

An object of class sitemapr_request_policy.

Details

What callers may set. headers, auth, proxy, tls, and prepare are all applied to each hop's request AFTER the per-hop SSRF guard and BEFORE sitemapr asserts its own transport controls.

What sitemapr always owns (callers cannot override). Redirect control (followlocation = 0, maxredirs = 0) and the non-2xx error policy are re-applied after all caller customization, and the structural SSRF guard re-runs on every redirect hop before any network call. A policy therefore cannot re-enable automatic redirect following, defeat the per-hop SSRF re-check, or turn a non-2xx status into a transport error — even via tls/prepare — because sitemapr's controls win last.

See also

Other request-policy: request_auth_basic(), request_auth_bearer(), request_proxy(), request_retry(), request_throttle()

Examples

# (1) A custom header (e.g. to satisfy a staging gateway).
request_policy(headers = list("X-Env" = "staging"))
#> $prepare
#> NULL
#> 
#> $headers
#> $headers$`X-Env`
#> [1] "staging"
#> 
#> 
#> $auth
#> NULL
#> 
#> $proxy
#> NULL
#> 
#> $tls
#> NULL
#> 
#> $retry
#> NULL
#> 
#> $throttle
#> NULL
#> 
#> $max_active
#> NULL
#> 
#> attr(,"class")
#> [1] "sitemapr_request_policy"

# (2) Basic and bearer authentication.
request_policy(auth = request_auth_basic("user", "secret"))
#> $prepare
#> NULL
#> 
#> $headers
#> NULL
#> 
#> $auth
#> $scheme
#> [1] "basic"
#> 
#> $username
#> [1] "user"
#> 
#> $password
#> [1] "secret"
#> 
#> attr(,"class")
#> [1] "sitemapr_request_auth"
#> 
#> $proxy
#> NULL
#> 
#> $tls
#> NULL
#> 
#> $retry
#> NULL
#> 
#> $throttle
#> NULL
#> 
#> $max_active
#> NULL
#> 
#> attr(,"class")
#> [1] "sitemapr_request_policy"
request_policy(auth = request_auth_bearer("a-token"))
#> $prepare
#> NULL
#> 
#> $headers
#> NULL
#> 
#> $auth
#> $scheme
#> [1] "bearer"
#> 
#> $token
#> [1] "a-token"
#> 
#> attr(,"class")
#> [1] "sitemapr_request_auth"
#> 
#> $proxy
#> NULL
#> 
#> $tls
#> NULL
#> 
#> $retry
#> NULL
#> 
#> $throttle
#> NULL
#> 
#> $max_active
#> NULL
#> 
#> attr(,"class")
#> [1] "sitemapr_request_policy"

# (3) A corporate proxy.
request_policy(proxy = "http://proxy.internal:3128")
#> $prepare
#> NULL
#> 
#> $headers
#> NULL
#> 
#> $auth
#> NULL
#> 
#> $proxy
#> $url
#> [1] "http://proxy.internal:3128"
#> 
#> $port
#> NULL
#> 
#> $username
#> NULL
#> 
#> $password
#> NULL
#> 
#> $auth
#> [1] "basic"
#> 
#> attr(,"class")
#> [1] "sitemapr_request_proxy"
#> 
#> $tls
#> NULL
#> 
#> $retry
#> NULL
#> 
#> $throttle
#> NULL
#> 
#> $max_active
#> NULL
#> 
#> attr(,"class")
#> [1] "sitemapr_request_policy"
request_policy(proxy = request_proxy("proxy.internal", port = 3128))
#> $prepare
#> NULL
#> 
#> $headers
#> NULL
#> 
#> $auth
#> NULL
#> 
#> $proxy
#> $url
#> [1] "proxy.internal"
#> 
#> $port
#> [1] 3128
#> 
#> $username
#> NULL
#> 
#> $password
#> NULL
#> 
#> $auth
#> [1] "basic"
#> 
#> attr(,"class")
#> [1] "sitemapr_request_proxy"
#> 
#> $tls
#> NULL
#> 
#> $retry
#> NULL
#> 
#> $throttle
#> NULL
#> 
#> $max_active
#> NULL
#> 
#> attr(,"class")
#> [1] "sitemapr_request_policy"

# (4) Bounded-concurrency index expansion (opt-in; sequential by default).
request_policy(max_active = 6)
#> $prepare
#> NULL
#> 
#> $headers
#> NULL
#> 
#> $auth
#> NULL
#> 
#> $proxy
#> NULL
#> 
#> $tls
#> NULL
#> 
#> $retry
#> NULL
#> 
#> $throttle
#> NULL
#> 
#> $max_active
#> [1] 6
#> 
#> attr(,"class")
#> [1] "sitemapr_request_policy"