Updates an edge list by replacing URLs with their final destinations based on a redirect data frame. Handles redirect chains, detects cycles, and resolves conflicting redirects using configurable policies.
Usage
resolve_redirects(
edge_list_df,
redirects_df,
edge_from_col = "from",
edge_to_col = "to",
redirect_from_col = "from",
redirect_to_col = "to",
duplicate_from_policy = c("strict", "first_wins", "last_wins", "most_frequent",
"prune_source", "resolve_if_consistent"),
loop_handling = c("error", "prune_loop", "break_arrow")
)Arguments
- edge_list_df
A data frame representing the edge list.
- redirects_df
A data frame containing redirect rules, with 'from' and 'to' columns specifying the source and target of a redirect.
- edge_from_col
Character, the name of the column in `edge_list_df` containing source URLs. Default "from".
- edge_to_col
Character, the name of the column in `edge_list_df` containing target URLs. Default "to".
- redirect_from_col
Character, the name of the column in `redirects_df` containing source URLs of redirects. Default "from".
- redirect_to_col
Character, the name of the column in `redirects_df` containing target URLs of redirects. Default "to".
- duplicate_from_policy
Character, how to handle conflicting redirects (same source URL mapping to multiple distinct targets). One of:
- "strict"
(Default) Error on any conflict.
- "first_wins"
Keep the first occurrence for each conflicting source.
- "last_wins"
Keep the last occurrence for each conflicting source.
- "most_frequent"
Keep the most common target. Ties broken by first occurrence.
- "prune_source"
Remove ALL redirects from any conflicting source.
- "resolve_if_consistent"
Allow exact duplicates; error only on true conflicts where targets differ.
- loop_handling
Character, how to handle redirect cycles (loops). One of:
- "error"
(Default) Error when a redirect cycle is detected.
- "prune_loop"
Remove all edges involved in cycles. URLs in the loop remain unresolved (map to themselves).
- "break_arrow"
For each cycle, keep the node with the highest in-degree as the sink and remove edges pointing away from it within the cycle. This preserves as much of the chain as possible.
Value
An updated `edge_list_df` with URLs in `edge_from_col` and `edge_to_col` replaced by their final resolved destinations.
Details
Self-referencing redirects (where from == to) and any redirects with NA in from or to are automatically filtered out before processing.
When crawl data contains conflicting redirects (the same URL redirecting
to different targets), use duplicate_from_policy to control the
behavior. The default "strict" preserves backward compatibility
by erroring on any conflict.
Redirect resolution uses a graph-based approach: an igraph is built from
the redirect rules, strongly connected components (SCCs) are used to
detect loops, and the loop_handling policy determines what happens
to cycles. After loop handling, each URL is mapped to its terminal
destination by traversing the acyclic graph.
Examples
edges <- data.frame(
from = c("A", "B", "C"),
to = c("B", "C", "D")
)
redirects <- data.frame(
from = c("B", "C", "E"),
to = c("B_final", "C_final", "E_final")
)
resolve_redirects(edges, redirects)
#> from to
#> 1 A B_final
#> 2 B_final C_final
#> 3 C_final D
# Example with a redirect chain
edges_chain <- data.frame(from = "X", to = "Y")
redirects_chain <- data.frame(
from = c("Y", "Z"),
to = c("Z", "Z_final")
)
resolve_redirects(edges_chain, redirects_chain)
#> from to
#> 1 X Z_final
# Example with conflicting redirects resolved via first_wins
edges_conflict <- data.frame(
from = "A", to = "B"
)
redirects_conflict <- data.frame(
from = c("B", "B"),
to = c("C", "D")
)
resolve_redirects(edges_conflict, redirects_conflict,
duplicate_from_policy = "first_wins"
)
#> from to
#> 1 A C
# Example with different column names
edges_custom <- data.frame(
source_url = "Page1", target_url = "Page2"
)
redirects_custom <- data.frame(
original = "Page2", final = "Page2_resolved"
)
resolve_redirects(edges_custom, redirects_custom,
edge_from_col = "source_url",
edge_to_col = "target_url",
redirect_from_col = "original",
redirect_to_col = "final"
)
#> source_url target_url
#> 1 Page1 Page2_resolved