Structural Smoothing of Empirical Page Transitions

Shrinks sparse empirical transition shares toward the crawl-graph link structure, so that no valid crawled link is assigned a probability of exactly zero. Observed page-transition data (e.g. from [ga4_page_transitions()]) is sparse: a low-traffic but real internal link may simply never have been traversed in the measured window, which leaves its raw empirical share at zero and destabilises the stationary PageRank vector. This helper combines the empirical distribution with a structural prior using a per-source shrinkage weight that grows with the source page's sample size.

Usage

smooth_transitions(
  empirical_df,
  structural_df,
  k = 5,
  min_support = 0,
  lambda_fn = NULL,
  count_col = "n",
  structural_weight_col = NULL,
  from_col = "from",
  to_col = "to",
  prob_col = "transition_probability"
)

Arguments

empirical_df: A data frame of observed transitions: a `from`/`to` edge list with a numeric count column. Typically the output of [ga4_page_transitions()] (or [aggregate_edges()] over behavioral counts). Duplicate `from`/`to` rows are summed.
structural_df: A data frame of crawl-graph links forming the structural prior: a `from`/`to` edge list, optionally with a numeric structural-weight column. Duplicate `from`/`to` rows are summed (so repeated link instances raise the prior weight). When `structural_weight_col` is `NULL`, each row contributes weight 1, i.e. a uniform prior over a source's crawled links.
k: Positive numeric. The pseudocount / Dirichlet concentration controlling shrinkage strength in $\lambda_i = n_i / (n_i + k)$. Larger `k` pulls under-sampled sources more strongly toward the structural prior. Must be `> 0`. Default `5`. Ignored for a source if `lambda_fn` is given.
min_support: Non-negative numeric. Sources with `0 < n_i < min_support` are treated as having insufficient empirical support and fall back fully to the structural prior ($\lambda_i = 0$). Default `0` (no minimum).
lambda_fn: Optional function mapping a source's sample size `n_i` (a single numeric) to a shrinkage weight in `[0, 1]`. Overrides the default `n_i / (n_i + k)` rule for sources that have both empirical data and a structural prior and meet `min_support`. Must return values `< 1` to preserve the non-zero-crawled-link guarantee. Default `NULL`.
count_col: Name of the numeric empirical-count column in `empirical_df`. Default `"n"` (the [ga4_page_transitions()] default).
structural_weight_col: Name of an optional numeric structural-weight column in `structural_df`, or `NULL` (default) for a uniform prior.
from_col, to_col: Names of the source / target columns, shared by both inputs and the output. Defaults `"from"` / `"to"`.
prob_col: Name of the output smoothed-probability column. Default `"transition_probability"`. Pass this to `pagerank(weight_col = ...)`.

Value

A data frame with one row per surviving source-target edge (the per-source union of empirical and structural edges, zero-probability edges dropped), ordered by `from` then `to`, carrying:

`from_col`, `to_col`: the edge endpoints (character).
`prob_col`: the smoothed transition probability; sums to 1 within each source.
`empirical_count`: the observed count for this edge (0 if the edge is `structural_only`).
`empirical_share`: `count / n_i`, the raw empirical share (0 if the source had no empirical data for this edge).
`structural_prior`: the structural prior share for this edge (0 if absent from the crawl graph).
`support`: `n_i`, the source page's total empirical out-count.
`lambda`: the per-source shrinkage weight applied.
`origin`: `"both"`, `"empirical_only"`, or `"structural_only"`.

Details

## The shrinkage model

For each source page `i`, the smoothed transition probability to target `j` is the convex combination

$$P(i \to j) = \lambda_i \cdot \mathrm{emp}(i \to j) + (1 - \lambda_i) \cdot \mathrm{prior}(i \to j)$$

where `emp(i -> j)` is the empirical share `count(i -> j) / n_i` (`n_i` = total observed out-transitions from `i`), `prior(i -> j)` is the structural prior share (the crawl-graph out-link distribution of `i`), and $\lambda_i$ is the per-source trust placed in the empirical data.

## Sample-size-dependent shrinkage (`lambda_i`)

By default $\lambda_i = n_i / (n_i + k)$, the standard Dirichlet / pseudocount shrinkage rule: it is monotonically increasing in the source-page sample size `n_i` and equals `1/2` at `n_i = k`. Equivalently, the model adds `k` pseudo-observations distributed according to the structural prior, then renormalizes — a Dirichlet prior with concentration `k`. A high-traffic source (large `n_i`) is trusted almost entirely to its own behavior; a barely-sampled source leans on the crawl structure. `k` must be strictly positive: this is precisely what guarantees $\lambda_i < 1$ for any sampled source, hence a non-zero $(1 - \lambda_i)$ weight on every crawled link (see *Guarantees*).

## Per-source special cases

Sources are matched between the two inputs and resolved as follows: - **No empirical data** (`n_i = 0`; crawled link absent from behavioral data): $\lambda_i = 0$, so `P(i -> .)` is the pure structural prior. The crawl link still receives mass. - **No structural prior** (observed transition whose source has no crawled out-links): $\lambda_i = 1$, so `P(i -> .)` is the pure empirical distribution — there is nothing to shrink toward. - **Insufficient support** (`0 < n_i < min_support`): $\lambda_i = 0$. The empirical sample is treated as too small to trust, and the source falls back to its structural prior (when one exists). - **Otherwise**: $\lambda_i = n_i / (n_i + k)$ (or `lambda_fn(n_i)`).

## Edge universe

The output covers the **union** of empirical and structural out-edges per source, with an `origin` column flagging each as `"both"`, `"empirical_only"` (an observed transition absent from the crawl graph), or `"structural_only"` (a crawled link never observed behaviorally). Edges whose smoothed probability is exactly zero — only possible for an `empirical_only` edge from a below-`min_support` source — are dropped, since they carry no transition mass.

## Time decay and segmentation

Time decay and device / template / channel segmentation are handled **upstream** by shaping the count input, keeping this function focused on the shrinkage itself. For time decay, supply decayed (fractional) counts in `count_col` — the `n_i` totals and $\lambda_i$ then reflect effective sample size, and `count_col` need not be integer. For segmentation, partition the empirical counts by segment and call `smooth_transitions()` once per segment (optionally against a segment-specific structural prior), then combine the results.

## Guarantees

For any `k > 0`, every crawled link present in `structural_df` receives a strictly positive smoothed probability: such an edge has `prior(i -> j) > 0`, and either $\lambda_i = 0$ (pure prior) or $\lambda_i < 1$ (since `n_i / (n_i + k) < 1`), so the $(1 - \lambda_i)$ weight on the prior is positive. Within each source, the returned probabilities sum to 1.

Examples

# Sparse behavioral data: only A->B was ever observed.
empirical <- data.frame(
  from = c("A", "A"),
  to = c("B", "C"),
  n = c(8, 0)
)[1, ]
# Crawl graph: A links to both B and C.
structural <- data.frame(
  from = c("A", "A"),
  to = c("B", "C")
)
smoothed <- smooth_transitions(empirical, structural, k = 5)
smoothed
#>   from to transition_probability empirical_count empirical_share
#> 1    A  B              0.8076923               8               1
#> 2    A  C              0.1923077               0               0
#>   structural_prior support    lambda          origin
#> 1              0.5       8 0.6153846            both
#> 2              0.5       8 0.6153846 structural_only
# A->C keeps a non-zero probability despite never being observed.

# Feed to pagerank() as a smoothed behavioral transition model:
# pagerank(smoothed, weight_col = "transition_probability",
#          clean_edge_urls = FALSE)

Usage

Arguments

Value

Details

See also

Examples