Skip to contents

Report non-fatal diagnostics for each URL under a standard profile

Source: R/accessors.R
get_url_diagnostics.Rd

Companion helper for the url_standard selector: reports the non-fatal validation/safety facts rurl observed while parsing each URL (for example an IPv4 host written in a numeric or non-decimal shorthand, or a path segment carrying an encoded reserved byte). Diagnostics are facts, not policy: they are emitted keyed to host/path shape in both standard modes so a security-sensitive consumer can reject a footgun URL regardless of which selector it chose, while a link-graph consumer can ignore them. The complete token vocabulary is enumerated below under Diagnostic vocabulary (canonical).

Usage

get_url_diagnostics(
  url,
  url_standard = NULL,
  scheme_policy = c("infer", "require"),
  scheme_acceptance = c("web", "general")
)

Arguments

url

A character vector of URLs.

url_standard

Standard profile governing interpretation: NULL (default; no diagnostics), "rfc3986", or "whatwg".

scheme_policy

Controls whether scheme-less, host-shaped input is accepted (an input-acceptance axis, distinct from protocol_handling, which only controls how the scheme is presented, and from url_standard, which controls interpretation). Defaults to "infer".

  • "infer": (Default) Fabricate http:// for scheme-less host-shaped input (e.g. example.com parses as http://example.com), a browser-omnibox-style affordance. This is the historical behavior.

  • "require": Reject scheme-less input — a scheme-less host-shaped value becomes parse_status = "error" rather than gaining a fabricated scheme. Use this for a strict, pure-parser posture. Note this governs only bare host input; scheme-relative //host input is governed separately by scheme_relative_handling.

scheme_acceptance

Which scheme tokens may enter parsing (a scheme-acceptance axis, distinct from scheme_policy, which governs scheme-less input, and from url_standard, which governs interpretation). Defaults to "web".

  • "web": (Default) Only the curated web-scheme allowlist (http/https/ftp/ftps/file) is admitted; a scheme-bearing input outside it is parse_status = "error". This is the historical, byte-for-byte compatible behavior.

  • "general": Admit any syntactically valid scheme token and parse opaque (mailto:x), non-special (foo://host), and RFC-generic URLs. Requires an explicit url_standard ("rfc3986" or "whatwg"), which decides the interpretation; general with url_standard = NULL is an error. Non-special / opaque hosts receive no www-stripping, no domain/TLD derivation, and are never run through the IDNA/punycode helpers. A non-special scheme with no // is an opaque path: it has no authority, so host, user, port and the domain/tld columns are all NA and the entire remainder is the path (query/fragment are still split off). This includes mailto: — the recipient's @ never re-triggers authority parsing. To decompose a mailto: recipient, use the accessors (get_host() / get_domain() / get_user(), ADR 0012 D7) or get_mailto_recipients(); those deliberately return a recipient's parts where this table presents NA, because a recipient domain is extraction metadata, not the URL's authority.

Value

For a length-1 url, a character vector of zero or more diagnostic tokens for that URL. For a length-n url (including n == 0), a list of length n whose i-th element is the character vector of that URL's tokens (character(0) when it has none).

Details

A single URL can carry several diagnostics, so the return shape is not a plain scalar-per-URL vector (see Value). parse_status stays coarse; diagnostics are never encoded into it.

Selected facts, not a conformance oracle

The diagnostics are deliberately a selected set of facts, not a complete validator. The absence of a diagnostic never implies the URL conforms to its scheme's specification or to WHATWG/RFC 3986. Full per-standard conformance validation is out of scope (ADR 0012 D5).

Two WHATWG-generic facts gate on the interpreting standard, not the acceptance axis, so they are reported whenever url_standard = "whatwg" — including the default "web" acceptance path (they are route-independent, string-level facts; RURL-sgjzbqzk):

  • invalid-URL-unit / invalid-credentials — WHATWG validation errors (WHATWG-verbatim names): a malformed %-escape or a non-URL code point, and any credentials (userinfo) present. Bounded detection.

The default combination ("web" acceptance with url_standard = NULL) emits no diagnostics at all, so it is unaffected.

With scheme_acceptance = "general" (the general-parser posture) a further set of selected facts is reported. These fire only under "general"; the default "web" acceptance never emits them:

  • unicode-outside-rfc3986-uri — under "rfc3986", a directly-written non-ASCII scalar value accepted by the sole RFC 3986 generic-grammar tolerance (not RFC 3987/IRI conformance).

  • transform-skipped-ineligible-scheme — the scheme is non-HTTP(S) and so ineligible for the SEO/semantic Stage-B transforms.

  • scheme-specific facts: ws-fragment-forbidden / ws-userinfo-forbidden (RFC 6455), mailto-fragment-discouraged (RFC 6068), tel-missing-phone-context (RFC 3966), data-missing-comma (RFC 2397), and, under "rfc3986", file-non-absolute-path, file-userinfo-extension (userinfo, permitted by RFC 8089 Appendix E.1's non-normative extended grammar), and file-component-outside-rfc8089 (a query or fragment, which RFC 8089's grammar does not mention and which are therefore inherited generic RFC 3986 components).

Diagnostic vocabulary (canonical)

This section is the single authoritative enumeration of the diagnostics vocabulary. It is held to the runtime registry (.URL_DIAGNOSTICS) in both directions by tools/diagnostics-doc-consistency.R, a CI gate: a token cannot be added, renamed, or removed without this list moving with it. Earlier design documents (including the v1 selector PRD's section 7 table) are historical records of what the vocabulary was when they were accepted — they are not registries and do not track it.

Every token below is emitted only when url_standard is not NULL. Tokens marked general additionally require scheme_acceptance = "general"; the rest fire under both acceptance postures and, unless noted, under both "rfc3986" and "whatwg".

Host — IPv4 shape. Facts about a host written as, or coerced to, an IPv4 address; security filters typically reject all of them.

  • ipv4-number-form — numeric IPv4 shorthand instead of dotted decimal.

  • ipv4-non-dotted — a whole-host number parsed/coerced to IPv4 in WHATWG mode.

  • ipv4-short-form — fewer than four dotted parts.

  • ipv4-non-decimal — hex or octal notation participated in IPv4 parsing.

  • ipv4-octal — octal interpretation changed the apparent address value.

  • ipv4-leading-zero — a dotted decimal-looking part had a leading zero.

  • ipv4-out-of-range — a dotted part exceeds 255 (fatal under "whatwg"; flags a numeric-looking reg-name under "rfc3986", e.g. 256.1.1.1).

Host — DNS length, UTS-46 and charset. Probed against the resolved host; IP literals are excluded.

  • domain-label-too-long — a label exceeds the DNS 63-byte limit.

  • domain-name-too-long — the whole name exceeds the DNS 253-byte limit.

  • domain-empty-label — the host contains an empty label (a ".." run, or a leading dot).

  • domain-hyphen-violation — a label breaks the UTS-46 hyphen rules (leading/trailing hyphen, or "--" in positions 3–4 of a non-xn-- label).

  • domain-std3-violation — a label carries a code point outside the STD3 LDH set.

  • host-charset-shimmed — the host carries one of the 15 code points WHATWG keeps but libcurl rejects, accepted by the shim (ADR 0009: ! $ & ( ) * + , ; =, plus the ASCII quotation mark, apostrophe, grave accent, and the two curly braces). "whatwg" only.

Path.

  • encoded-dot-segment — an encoded-dot segment (%2e / %2e%2e, any hex case) that the profile's dot handling acted on.

  • encoded-reserved-path-byte — the preserved path still carries an encoded reserved byte (%2F, %3F, %23) held as data rather than as a separator.

Port. Facts about the raw port versus the resolved scheme's WHATWG default, independent of the port_handling knob.

  • explicit-default-port — the port was written out and equals the scheme's default.

  • non-default-port — a port is present and is not the scheme's default (including any port on a scheme with no defined default).

Input shape — WHATWG cleanup. All three are "whatwg" only; "rfc3986" has no strip or rewrite step.

  • invalid-reverse-solidus — a literal \ was reinterpreted as / (special schemes only).

  • control-char-stripped — an ASCII tab/LF/CR was removed from the interior of the input (step 1, second half).

  • leading-trailing-stripped — a leading and/or trailing run of C0-control-or-SPACE was removed (step 1, first half).

Layer 5 — selected per-standard and per-scheme facts (ADR 0012 D5). Described in full under Selected facts, not a conformance oracle above.

  • invalid-URL-unit — WHATWG validation error: a malformed %-escape or a non-URL code point. "whatwg" only.

  • invalid-credentials — WHATWG validation error: credentials (userinfo) are present. "whatwg" only.

  • unicode-outside-rfc3986-uri — general; a directly-written non-ASCII scalar value under "rfc3986".

  • transform-skipped-ineligible-scheme — general; a non-HTTP(S) scheme, ineligible for the Stage-B transforms.

  • ws-fragment-forbidden — general; a fragment on a ws:/wss: URL (RFC 6455).

  • ws-userinfo-forbidden — general; userinfo on a ws:/wss: URL (RFC 6455).

  • mailto-fragment-discouraged — general; a fragment on a mailto: URL (RFC 6068).

  • tel-missing-phone-context — general; a local tel: number with no phone-context (RFC 3966).

  • data-missing-comma — general; a data: URL with no "," separator (RFC 2397).

  • file-non-absolute-path — general; a non-absolute file: path under "rfc3986".

  • file-userinfo-extension — general; userinfo on a file: URL, permitted by RFC 8089 Appendix E.1's non-normative extended grammar.

  • file-component-outside-rfc8089 — general; a query or fragment on a file: URL, inherited from generic RFC 3986.

See also

get_host_type, safe_parse_url

Examples

get_url_diagnostics("http://example.com/", url_standard = "rfc3986")
#> character(0)
get_url_diagnostics(
  c("http://example.com/", "http://2130706433/"),
  url_standard = "whatwg"
)
#> [[1]]
#> character(0)
#> 
#> [[2]]
#> [1] "ipv4-number-form" "ipv4-non-dotted" 
#>