Skip to contents

Compose Screaming Frog node and link exports

Source: R/screaming_frog_bundle.R, R/screaming_frog_contract.R
screaming_frog_bundle.Rd

Builds the stable `screaming_frog_bundle` handoff object from an **Internal: All** export and one **All Inlinks** or **All Outlinks** export. The component adapters remain the source of truth: raw link observations stay separate from graph-eligible edges, and node, redirect, canonical, and indexability tables are exposed unchanged for downstream scoring.

`pagerankr` accepts a Screaming Frog **Internal: All** export as node metadata and one **All Inlinks** or **All Outlinks** bulk export as link observations. Internal: All is never treated as an edge list. Both link exports use `Source -> Destination`; the export kind records provenance and never changes orientation.

Usage

screaming_frog_bundle(
  internal,
  links,
  link_export_kind = c("all_inlinks", "all_outlinks"),
  origin_policy = c("all", "html", "rendered"),
  endpoint_action = c("drop", "error")
)

Arguments

internal

A path/data frame accepted by [screaming_frog_internal()], or an existing `screaming_frog_internal` object.

A path/data frame accepted by [screaming_frog_links()], or an existing `screaming_frog_links` object.

Declared link export kind when `links` is not already a `screaming_frog_links` object: `"all_inlinks"` or `"all_outlinks"`.

origin_policy, endpoint_action

Passed to [screaming_frog_links()] when `links` is not already imported.

Value

An S3 object of class `screaming_frog_bundle` with stable top-level fields `nodes`, `observations`, `edges`, `redirects`, `canonicals`, `indexability`, `diagnostics`, and `provenance`.

Input boundary

Future Screaming Frog adapters accept either a file path or a data frame. Files are CSV, may contain a UTF-8 byte-order mark, and are read by first inspecting the header and then selecting only contract columns. This keeps 500+ MB link exports bounded to the columns needed by the requested adapter. Data-frame inputs follow the same alias resolution and column ordering. Unknown extra columns are ignored and reported; missing required columns fail with their normalized contract names.

Raw URLs are preserved at this boundary. URL cleaning belongs to the scoring path and is performed once with `pagerankr`'s pinned `rurl` canonicalization profile.

Raw link observations and graph-eligible edges are separate tables. Duplicate observations are preserved. Only rows whose `type` is `"Hyperlink"` are graph-eligible by default. Resource, sitemap, hreflang, redirect, and canonical observations are not silently promoted to PageRank edges; redirects and canonicals remain dedicated signals.

`follow` is the primary nofollow field. `rel` is retained and parsed independently for diagnostics, so disagreements can be reported. Link position is normalized only for `Navigation`, `Content`, `Footer`, `Header`, and `Aside`; `Head`, blanks, and unknown values remain unmapped rather than being guessed. Link origin and link path are preserved as provenance. Link path is an XPath-like source locator, not a URL path.

Stable bundle shape

The public `screaming_frog_bundle` object introduced by the adapter tickets is an S3 list with these stable top-level fields, in order:

nodes

Normalized Internal: All node facts.

observations

Lossless normalized link observations.

edges

The graph-eligible `from` / `to` subset.

redirects

`from` / `to` redirect signals.

canonicals

`from` / `to` canonical signals.

indexability

URL-level indexability facts.

diagnostics

Counts, omissions, invalid values, and disagreements.

provenance

Input kind, source, schema aliases, and schema clues.

This topic freezes the contract consumed by the SF1-SF5 implementation tickets. The complete object is constructed by [screaming_frog_bundle()].

Examples

internal <- data.frame(
  Address = c("https://example.com/", "https://example.com/a"),
  `Status Code` = c("200", "200"),
  check.names = FALSE
)
links <- data.frame(
  Type = "Hyperlink",
  Source = "https://example.com/",
  Destination = "https://example.com/a",
  Follow = "TRUE",
  check.names = FALSE
)
bundle <- screaming_frog_bundle(internal, links, "all_outlinks")
bundle$edges
#>   input_row                 from                    to nofollow follow  rel
#> 1         1 https://example.com/ https://example.com/a    FALSE   TRUE <NA>
#>   rel_nofollow anchor alt_text target path_type link_path link_position
#> 1           NA   <NA>     <NA>   <NA>      <NA>      <NA>          <NA>
#>   placement link_origin destination_status_code destination_status
#> 1      <NA>        <NA>                      NA               <NA>
#>   destination_crawlability
#> 1                     <NA>