Import Screaming Frog Internal: All node facts — screaming_frog

Normalizes a Screaming Frog **Internal: All** CSV or data frame into node, redirect, canonical, and indexability tables. This is a node-side adapter: it does not reconstruct links from aggregate Inlinks/Outlinks counts.

Usage

screaming_frog_internal(x)

Arguments

x: A path to an Internal: All CSV file or an equivalent data frame.

Value

An object of class `screaming_frog_internal` with components:

nodes: Normalized node facts in input row order. Columns are `url`, `segments`, `content_type`, `http_status`, `status`, `indexability`, `indexability_status`, `canonical`, `redirect_to`, `redirect_type`, `crawl_allowed`, `indexing_allowed`, robots fields, language/timestamps, and selected crawl metrics. Optional absent fields are typed `NA` columns.
redirects: Raw `from` / `to` redirect pairs from valid 3xx rows.
canonicals: Raw `from` / `to` canonical pairs, including self-canonicals.
indexability: URL-level facts compatible with `pagerank()`'s indexability input.
diagnostics: Deterministic counts, missing optional and ignored columns, duplicate addresses, and row-level structural issues.
provenance: Input identity, retained input-row IDs, and the normalized-to-detected column manifest.

Details

URLs are preserved as exported. Redirects are emitted only for valid 3xx rows with a non-blank destination. Canonicals are derived independently, including self-canonicals for audit.

Examples

internal <- data.frame(
  Address = c("https://example.com/", "https://example.com/old"),
  `Status Code` = c("200", "301"),
  `Redirect URL` = c("", "https://example.com/new"),
  check.names = FALSE
)
imported <- screaming_frog_internal(internal)
imported$nodes
#>                       url segments content_type http_status status indexability
#> 1    https://example.com/       NA         <NA>         200   <NA>         <NA>
#> 2 https://example.com/old       NA         <NA>         301   <NA>         <NA>
#>   indexability_status canonical             redirect_to redirect_type
#> 1                <NA>      <NA>                    <NA>          <NA>
#> 2                <NA>      <NA> https://example.com/new          <NA>
#>   crawl_allowed indexing_allowed meta_robots x_robots_tag language
#> 1            NA               NA        <NA>         <NA>     <NA>
#> 2            NA               NA        <NA>         <NA>     <NA>
#>   crawl_timestamp last_modified size_bytes word_count inlinks unique_inlinks
#> 1            <NA>          <NA>         NA         NA      NA             NA
#> 2            <NA>          <NA>         NA         NA      NA             NA
#>   outlinks unique_outlinks response_time_seconds
#> 1       NA              NA                    NA
#> 2       NA              NA                    NA
imported$redirects
#>                      from                      to status_code redirect_type
#> 1 https://example.com/old https://example.com/new         301          <NA>