Report practical host validation verdicts for a set of URLs

Tabular companion to is_valid_host: for each URL, reports the parsed host, its get_host_type classification, a logical column for each requested policy rule, and a reasons list-column naming the host facts observed. Useful for auditing a URL set before choosing what to keep — see why a host is not a practical web/SEO host, not just that it is not.

Usage

check_hosts(
  url,
  rules = c("url", "dns", "web", "registrable", "seo"),
  url_standard = "whatwg",
  scheme_policy = c("infer", "require"),
  scheme_acceptance = c("web", "general")
)

Arguments

url

A character vector of URLs.

rules

A character vector of one or more rules to score, drawn from "url", "dns", "web", "registrable", "seo" (all five by default). See is_valid_host for each rule's meaning.

url_standard

Standard profile governing host interpretation: "whatwg" (default) or "rfc3986".

scheme_policy

Controls whether scheme-less, host-shaped input is accepted (an input-acceptance axis, distinct from protocol_handling, which only controls how the scheme is presented, and from url_standard, which controls interpretation). Defaults to "infer".

"infer": (Default) Fabricate http:// for scheme-less host-shaped input (e.g. example.com parses as http://example.com), a browser-omnibox-style affordance. This is the historical behavior.
"require": Reject scheme-less input — a scheme-less host-shaped value becomes parse_status = "error" rather than gaining a fabricated scheme. Use this for a strict, pure-parser posture. Note this governs only bare host input; scheme-relative //host input is governed separately by scheme_relative_handling.

scheme_acceptance

Which scheme tokens may enter parsing (a scheme-acceptance axis, distinct from scheme_policy, which governs scheme-less input, and from url_standard, which governs interpretation). Defaults to "web".

"web": (Default) Only the curated web-scheme allowlist (http/https/ftp/ftps/file) is admitted; a scheme-bearing input outside it is parse_status = "error". This is the historical, byte-for-byte compatible behavior.
"general": Admit any syntactically valid scheme token and parse opaque (mailto:x), non-special (foo://host), and RFC-generic URLs. Requires an explicit url_standard ("rfc3986" or "whatwg"), which decides the interpretation; general with url_standard = NULL is an error. Non-special / opaque hosts receive no www-stripping, no domain/TLD derivation, and are never run through the IDNA/punycode helpers. A non-special scheme with no // is an opaque path: it has no authority, so host, user, port and the domain/tld columns are all NA and the entire remainder is the path (query/fragment are still split off). This includes mailto: — the recipient's @ never re-triggers authority parsing. To decompose a mailto: recipient, use the accessors (get_host() / get_domain() / get_user(), ADR 0012 D7) or get_mailto_recipients(); those deliberately return a recipient's parts where this table presents NA, because a recipient domain is extraction metadata, not the URL's authority.

Value

A data.frame with one row per input URL (input order preserved): url, host (the parsed host, or NA), host_type, one logical column per requested rule (NA when the URL has no host to judge; the "url" rule's column is named url_valid so it does not shadow the input url column), and reasons — a list-column whose i-th element is a character vector of the host facts observed for that URL (character(0) when none). The reasons evidence is reported for the row as a whole, independent of which rules were requested, and combines the host-shape diagnostics that fired (see get_url_diagnostics) with the policy tokens "ip-literal", "not-registrable", and "underscore-label".

Details

Like is_valid_host this is a policy layer, not parser conformance and not a conformance oracle: it never changes how a URL parses and the absence of a reasons token is not a validity guarantee (see is_valid_host's “A policy layer” section).

Examples

urls <- c(
  "http://example.com",         # registrable domain
  "http://_dmarc.example.com",  # valid DNS owner name, not a web hostname
  "http://a+b.example",         # valid RFC reg-name, neither web nor dns
  "http://-example.com",        # hyphen hygiene failure
  "http://a..com",              # empty label
  "http://localhost",           # web hostname, but not registrable
  "http://192.168.0.1",         # IP literal
  "http://xn--nxasmq6b.example.com" # IDN (A-label)
)
check_hosts(urls)
#>                               url                     host host_type url_valid
#> 1              http://example.com              example.com    domain      TRUE
#> 2       http://_dmarc.example.com       _dmarc.example.com  reg-name      TRUE
#> 3              http://a+b.example              a+b.example  reg-name      TRUE
#> 4             http://-example.com             -example.com  reg-name      TRUE
#> 5                   http://a..com                   a..com  reg-name      TRUE
#> 6                http://localhost                localhost  reg-name      TRUE
#> 7              http://192.168.0.1              192.168.0.1      ipv4      TRUE
#> 8 http://xn--nxasmq6b.example.com xn--nxasmq6b.example.com    domain      TRUE
#>     dns   web registrable   seo
#> 1  TRUE  TRUE        TRUE  TRUE
#> 2  TRUE FALSE       FALSE FALSE
#> 3 FALSE FALSE       FALSE FALSE
#> 4 FALSE FALSE       FALSE FALSE
#> 5 FALSE FALSE       FALSE FALSE
#> 6  TRUE  TRUE       FALSE FALSE
#> 7 FALSE  TRUE       FALSE FALSE
#> 8  TRUE  TRUE        TRUE  TRUE
#>                                                        reasons
#> 1                                                             
#> 2     domain-std3-violation, not-registrable, underscore-label
#> 3 host-charset-shimmed, domain-std3-violation, not-registrable
#> 4                     domain-hyphen-violation, not-registrable
#> 5                          domain-empty-label, not-registrable
#> 6                                              not-registrable
#> 7                                                   ip-literal
#> 8                                                             
# Score a single rule, or a subset:
check_hosts(urls, rules = c("web", "dns"))
#>                               url                     host host_type   web
#> 1              http://example.com              example.com    domain  TRUE
#> 2       http://_dmarc.example.com       _dmarc.example.com  reg-name FALSE
#> 3              http://a+b.example              a+b.example  reg-name FALSE
#> 4             http://-example.com             -example.com  reg-name FALSE
#> 5                   http://a..com                   a..com  reg-name FALSE
#> 6                http://localhost                localhost  reg-name  TRUE
#> 7              http://192.168.0.1              192.168.0.1      ipv4  TRUE
#> 8 http://xn--nxasmq6b.example.com xn--nxasmq6b.example.com    domain  TRUE
#>     dns                                                      reasons
#> 1  TRUE                                                             
#> 2  TRUE     domain-std3-violation, not-registrable, underscore-label
#> 3 FALSE host-charset-shimmed, domain-std3-violation, not-registrable
#> 4 FALSE                     domain-hyphen-violation, not-registrable
#> 5 FALSE                          domain-empty-label, not-registrable
#> 6  TRUE                                              not-registrable
#> 7 FALSE                                                   ip-literal
#> 8  TRUE                                                             
# RFC 3986 reg-name semantics instead of whatwg:
check_hosts("http://2130706433", url_standard = "rfc3986")
#>                 url       host host_type url_valid  dns  web registrable   seo
#> 1 http://2130706433 2130706433  reg-name      TRUE TRUE TRUE       FALSE FALSE
#>                                              reasons
#> 1 ipv4-number-form, ipv4-non-dotted, not-registrable