rurl

rurl is a lightweight, vectorized toolkit for URL parsing, normalization, extraction, and matching in R.

Current package capabilities include:

Robust parsing via safe_parse_url() and safe_parse_urls()
URL normalization with fine-grained controls for protocol, www, case, trailing slashes, index pages, path normalization, scheme-relative URLs, host encoding, and path encoding
Opt-in query-string handling: drop trackers, keep contentful params, and audit params across a URL set with query_param_summary()
URL component extractors (get_* helpers)
URL-based joins with canonical_join()
Built-in memoization caches with introspection and configuration (rurl_cache_info(), rurl_cache_config(), rurl_clear_caches())

Installation

# From CRAN
install.packages("rurl")

# Development version from GitHub
# install.packages("remotes")
remotes::install_github("bart-turczynski/rurl")

Function Overview

Parsing and normalization: safe_parse_url(), safe_parse_urls(), get_clean_url()
Accessors: get_scheme(), get_host(), get_subdomain(), get_domain(), get_tld(), get_path(), get_query(), get_fragment(), get_port(), get_user(), get_password(), get_userinfo(), get_parse_status()
Query introspection: query_param_summary()
Matching/joining: canonical_join() for deterministic canonical-key joins
Cache control: rurl_cache_info(), rurl_cache_config(), rurl_clear_caches()

Quick Start

safe_parse_url() is the core workhorse. It returns parsed components and a normalized clean_url.

library(rurl)

parsed <- safe_parse_url(
  "HTTP://www.Example.com/a//b/../index.html?x=1#frag",
  protocol_handling = "https",
  www_handling = "strip",
  case_handling = "lower_host",
  trailing_slash_handling = "strip",
  index_page_handling = "strip",
  path_normalization = "both",
  host_encoding = "idna",
  path_encoding = "encode"
)

parsed$clean_url
#> [1] "https://example.com/a"
parsed$parse_status
#> [1] "ok"

clean_url is a normalized canonical key built from scheme, host, and path. By default the query is dropped and port, fragment, and userinfo are always excluded — read them from the dedicated components (get_port(), get_query(), get_fragment(), get_userinfo()) instead. The query can be selectively kept on clean_url via query_handling on either safe_parse_url() or get_clean_url() (see Query handling). With path_encoding = "decode" the path is shown decoded, so clean_url is human-readable rather than guaranteed URL-safe.

Scheme-relative URL handling is configurable:

safe_parse_url("//example.com/path", scheme_relative_handling = "keep")$parse_status
#> [1] "ok-scheme-relative"
safe_parse_url("//example.com/path", scheme_relative_handling = "https")$clean_url
#> [1] "https://example.com/path"

For vectors, use safe_parse_urls():

safe_parse_urls(c("example.com", "https://www.example.com/path"))[, c("original_url", "clean_url", "parse_status")]

Normalization Controls

safe_parse_url() and get_clean_url() support these controls:

protocol_handling: keep, none, strip, http, https
www_handling: none, strip, keep, if_no_subdomain
case_handling: lower_host (default), keep, lower, upper
trailing_slash_handling: none, keep, strip
index_page_handling: keep, strip
path_normalization: none, collapse_slashes, dot_segments, both
scheme_relative_handling: keep, http, https, error
host_encoding: keep, idna, unicode
path_encoding: keep, encode, decode
subdomain_levels_to_keep: NULL, 0, or N > 0
query_handling: drop (default), filter, allow, keep (see Query handling)

Subdomain retention is applied after www_handling:

get_host("http://www.three.two.one.example.com", www_handling = "strip", subdomain_levels_to_keep = 1)
#> [1] "one.example.com"
get_clean_url("http://www.deep.sub.example.com/path", subdomain_levels_to_keep = 0)
#> [1] "http://www.example.com/path"

Host and path encoding controls:

get_clean_url("http://münich.com/a%20b",
              host_encoding = "idna",
              path_encoding = "encode",
              case_handling = "lower_host")
#> [1] "http://xn--mnich-kva.com/a%20b"

get_clean_url("http://xn--mnich-kva.com/a%20b",
              host_encoding = "unicode",
              path_encoding = "decode",
              case_handling = "keep")
#> [1] "http://münich.com/a b"

Accessors

u <- "https://user:pass@www.blog.example.co.uk/path/to/page?a=1&b=2#frag"

get_scheme(u)
get_host(u)
get_subdomain(u)
get_domain(u)
get_tld(u)
get_path(u)
get_query(u)
get_query(u, format = "list")
get_fragment(u)
get_port(u)
get_user(u)
get_password(u)
get_userinfo(u)
get_parse_status(c(u, "mailto:test@example.com"))

Query handling

By default clean_url is query-free (query_handling = "drop"), so a tracker like ?utm_source= never changes the canonical key. Opt in to keep contentful params while stripping known trackers (utm_*, fbclid, gclid, …) via a built-in denylist:

# "filter": keep contentful params, drop known trackers
get_clean_url("https://youtube.com/watch?v=dQw4w9&utm_source=nl",
              query_handling = "filter")
#> [1] "https://youtube.com/watch?v=dQw4w9"

# "allow": keep ONLY the names you list (params_keep is the allowlist)
get_clean_url("https://shop.com/p?id=42&ref=abc&fbclid=xyz",
              query_handling = "allow", params_keep = "id")
#> [1] "https://shop.com/p?id=42"

# "keep": keep every param, canonicalized (optionally sorted)
get_clean_url("https://ex.com/?b=2&a=1",
              query_handling = "keep", sort_params = TRUE)
#> [1] "https://ex.com/?a=1&b=2"

The same engine arguments flow through safe_parse_urls() (and therefore canonical_join()), so filtering also shapes the join key — ?v=1 and ?v=2 stay distinct while utm-only differences still collapse under "filter":

safe_parse_urls(c("https://ex.com/?v=1&utm_source=x", "https://ex.com/?v=2"),
                query_handling = "filter")[, c("original_url", "clean_url")]
#>                       original_url           clean_url
#> 1 https://ex.com/?v=1&utm_source=x https://ex.com/?v=1
#> 2              https://ex.com/?v=2 https://ex.com/?v=2

get_query() takes the same arguments (defaulting to query_handling = "keep") to pull a cleaned query directly:

get_query("https://youtube.com/watch?v=dQw4w9&utm_source=nl",
          query_handling = "filter")
#> [1] "v=dQw4w9"

Audit a URL set before choosing a policy with query_param_summary(), whose would_drop column previews what "filter" would remove:

urls <- c(
  "https://ex.com/?utm_source=nl&id=42",
  "https://ex.com/watch?v=abc&utm_source=x",
  "https://ex.com/?id=99"
)
query_param_summary(urls)
#>        param n n_urls example_value                             example_url would_drop
#> 1 utm_source 2      2            nl     https://ex.com/?utm_source=nl&id=42       TRUE
#> 2         id 2      2            42     https://ex.com/?utm_source=nl&id=42      FALSE
#> 3          v 1      1           abc https://ex.com/watch?v=abc&utm_source=x      FALSE

URL Joins

canonical_join() matches on one canonicalized key per URL and is the preferred option for large datasets:

A <- data.frame(URL = c("https://Example.com/page", "https://example.com/other"),
                ValA = 1:2, stringsAsFactors = FALSE)
B <- data.frame(URL = c("https://example.com/page?utm_source=nl", "https://example.com/missing"),
                ValB = c("x", "y"), stringsAsFactors = FALSE)

# Default canonicalization lower-cases the host and drops the query, so the
# first row of each side shares one canonical key.
canonical_join(A, B)

Caching

rurl memoizes URL parsing and punycode round-trips to speed repeated operations over large URL vectors; PSL query caching lives in pslr. Inspect, clear, and configure the caches:

rurl_cache_info()                          # entries / enabled / max per cache
rurl_clear_caches()                        # free memory in a long-running session
rurl_cache_config(max_full_parse = 1e5)    # bound the full-parse cache
rurl_cache_config(puny_encode = FALSE)     # disable a cache entirely

rurl_cache_config() covers three caches: full_parse, puny_encode, and puny_decode. The full_parse cache is bounded at 100000 entries by default: a hard reset-watermark that caps peak memory (once the bound is hit the cache empties completely and rebuilds, rather than evicting per entry). Set max_full_parse = Inf to opt into the historical unbounded behavior. The puny_encode and puny_decode caches are unbounded by design and can be disabled for workloads with very many unique hosts.

Public Suffix List

Domain and TLD extraction is delegated to the pslr package, which owns the Public Suffix List and its refresh cycle. rurl ships no embedded copy of the list. To update the PSL, call pslr::psl_refresh() (see the pslr documentation for details).

Acknowledgments

These packages build on data, libraries, and prior work from many others. See ACKNOWLEDGMENTS.md for the full list of thanks.

rurl is part of a small ecosystem of R packages by the same author:

pslr — the Public Suffix List engine that powers domain and TLD extraction in rurl. Use it directly when you need raw eTLD / registrable-domain queries without full URL parsing.
punycoder — the Punycode and IDNA codec that rurl uses for internationalized host handling. Useful on its own for host normalization and Unicode ↔︎ ACE round-trips.

Citation

If you use rurl in your work, please cite it. Run citation("rurl") for the current citation, or see CITATION.cff.

Each release is archived on Zenodo. Cite the concept DOI 10.5281/zenodo.20972584 to refer to the software in general (it always resolves to the latest version), or the version-specific DOI shown on the Zenodo record for a particular release.

Code of Conduct

Please note that this package is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.