Applies `rurl::get_clean_url` to specified columns of a data frame. URLs are cleaned under pagerankr's explicit canonicalization profile (see Details), with any arguments in `...` overriding individual knobs.
Usage
clean_url_columns(data_frame, columns = c("from", "to"), ...)Arguments
- data_frame
A data frame containing URL columns to be cleaned.
- columns
A character vector specifying the names of the columns containing URLs. Defaults to `c("from", "to")`.
- ...
`rurl::get_clean_url` arguments that override the canonicalization profile per key. Recognized knobs: `protocol_handling`, `case_handling`, `www_handling`, `trailing_slash_handling`, `index_page_handling`, `path_normalization`, `scheme_relative_handling`, `subdomain_levels_to_keep`, `host_encoding`, `path_encoding`.
Details
The canonicalization profile ([canonical_profile()]) pins every `rurl` knob explicitly so node identities do not depend on `rurl`'s own (version-dependent) defaults, and keeps the cleaning and domain-filtering paths symmetrical. Most knobs equal `rurl`'s current defaults; the two path knobs (`path_normalization = "dot_segments"`, `path_encoding = "decode"`) intentionally override `rurl` 2.1.0's redefined `"none"`/`"keep"` defaults to preserve the committed canonical key. See [canonical_profile()] for details.
NA values in the specified columns are preserved in the output. Downstream functions in the pagerankr workflow (such as get_unique_edges and pagerank) will automatically drop any edge where either from or to is NA.
Tokens that `rurl` cannot parse as a URL (e.g. a dotless bare label such as `"A"`, which newer `rurl` normalizes to NA) are left as their raw input value rather than becoming NA. This keeps unparseable but non-missing node identities as opaque nodes instead of silently dropping them, so an odd URL in a crawl is scored as its own node rather than vanishing. Only genuinely missing (NA) inputs stay NA. This raw-fallback is a deliberate, accepted divergence from the sibling `semantic` project (which drops such inputs); see [canonical_profile()] for why it does not desync the cross-repo node join.
Examples
df <- data.frame(
from = c(
"http://example.com/path",
"HTTPS://Example.com/PATH#frag", NA,
"http://example.com/path"
),
to = c(
"www.another.com?q=1", "another.com/?q=1&b=2",
"http://foo.bar", NA
),
other_col = 1:4
)
cleaned_df <- clean_url_columns(df, columns = c("from", "to"))
print(cleaned_df)
#> from to other_col
#> 1 http://example.com/path http://www.another.com/ 1
#> 2 https://example.com/PATH http://another.com/ 2
#> 3 <NA> http://foo.bar/ 3
#> 4 http://example.com/path <NA> 4
# Pass extra arguments to rurl::get_clean_url via ...
cleaned_df_custom <- clean_url_columns(
df,
columns = c("from", "to"),
protocol_handling = "http"
)
print(cleaned_df_custom)
#> from to other_col
#> 1 http://example.com/path http://www.another.com/ 1
#> 2 http://example.com/PATH http://another.com/ 2
#> 3 <NA> http://foo.bar/ 3
#> 4 http://example.com/path <NA> 4