Architecture

This document is the map of how punycoder is built: the layers a call passes through, where each responsibility lives, and how the build and data pipelines fit together. It is written for a developer (human or agent) picking the package up cold.

Companion documents, each owning a different slice:

AGENTS.md — the terse, always-loaded working contract: conventions, commands, hard rules. Kept authoritative; this file expands on the shape of the system, it does not restate those rules.
DECISIONS.md — the why: an ADR log of the load-bearing choices (scope, profile, backend model, error policy, deprecations).
dev/normalization-contract.md — the deep normative spec for host_normalize() (profile parameters, algorithm, backend parity, versioning). The single source of truth for normalization behavior.

What the package is

punycoder is an RFC 3492-compliant Punycode/IDN codec plus a UTS #46 canonical host normalizer, implemented in C++ via Rcpp. It is the Punycode/IDNA engine for the sibling packages pslr (public-suffix) and rurl (URL parsing). Its scope is deliberately narrow — a resolvability- and safety-agnostic IDNA primitive; see DECISIONS.md ADR-001 and the README “Non-goals” section.

It exposes three concerns:

Surface	Functions	What it does
Punycode codec	`puny_encode()`, `puny_decode()`	Raw RFC 3492 transform with `xn--` framing + LDH checks. No Unicode normalization.
Host normalization	`host_normalize()`, `normalization_profile_info()`	UTS #46 non-transitional canonical host form (mapping + NFC + validation + Punycode).
Validators	`is_punycode()`, `is_idn()`, `validate_domain()`	Predicate/validation helpers.

Layering

Every call crosses the same four layers, top to bottom. The rule of thumb: the lower you go, the fewer types you may use. Only one file touches Rcpp; only R files touch R semantics.

┌──────────────────────────────────────────────────────────────────┐
│ R wrapper layer  (R/*.R)                                           │
│   Input validation, NA policy, strict/non-strict option,          │
│   S3 print/summary.                                                │
│   punycoder.R · normalize.R · validators.R ·                      │
│   results.R · helpers.R · zzz.R                                    │
└───────────────┬────────────────────────────────────────────────────┘
                │ .call_with_validation()  →  *_cpp shims
┌───────────────▼────────────────────────────────────────────────────┐
│ Rcpp glue  (R/RcppExports.R  ↔  src/RcppExports.cpp)  — GENERATED   │
│   Rcpp::compileAttributes(); never hand-edited.                    │
└───────────────┬────────────────────────────────────────────────────┘
                │
┌───────────────▼────────────────────────────────────────────────────┐
│ Rcpp boundary  (src/exports.cpp)  — the ONLY file that talks Rcpp   │
│   NA handling · strict→Rcpp::stop / non-strict→NA_character_ ·      │
│   R-facing error prefixes · compare_backends / backend_info.       │
└───────────────┬────────────────────────────────────────────────────┘
                │ std::string / std::vector only, below this line
┌───────────────▼────────────────────────────────────────────────────┐
│ Core  (namespace punycoder, src/*.cpp + punycoder_core.h)           │
│   service → domain/normalize → backend → algorithm/nfc/utf8 →       │
│   vendored Unicode tables.                                          │
└──────────────────────────────────────────────────────────────────┘

All declarations for the core live in a single header, src/punycoder_core.h; implementations are split by responsibility across src/*.cpp. Editing that header (especially the ErrorCode enum) requires a clean rebuild — R’s build does not track header dependencies, so stale .o files cause silent ABI skew. See ADR-009.

Module map

R wrapper layer (`R/`)

File	Responsibility
`punycoder.R`	`puny_*` surface + validators exports.
`normalize.R`	`host_normalize()` + `normalization_profile_info()`.
`validators.R`	`is_punycode()`, `is_idn()`, `validate_domain()`.
`results.R`	S3 `print`/summary for `punycoder_validation`.
`helpers.R`	Input assertions + `.call_with_validation()` dispatch.
`zzz.R`	`.onLoad` option defaults (`punycoder.strict`).
`RcppExports.R`	Generated shims — do not edit.

Native core (`src/`)

File	Responsibility
`punycoder_core.h`	All core declarations (single header).
`exports.cpp`	Rcpp boundary: NA/strict policy, error prefixes, introspection.
`punycoder_service.cpp`	`PunycodeService` facade wiring backend → domain layer, applies `strict`.
`punycoder_domain.cpp`	`validate_and_parse_domain`, label rules (length, hyphens, `xn--`).
`punycoder_normalize.cpp` / `.h`	`host_normalize_one`: the UTS #46 pipeline.
`punycoder_nfc.cpp` / `.h`	Unicode NFC (UAX #15) used by the normalizer.
`punycoder_backend.cpp`	`select_label_backend` + the `libidn2` adapter (all `#ifdef PUNYCODER_USE_LIBIDN2` live here).
`punycoder_algorithm.cpp`	RFC 3492 reference encoder/decoder (fallback).
`punycoder_utf8.cpp`	UTF-8 ↔︎ codepoint conversion + ASCII helpers.
`punycoder_errors.cpp`	`PunycoderError` + the `throw_error(ErrorCode, …)` map.
`unicode_tables_16_0_0.cpp` / `.h`, `unicode_tables_17_0_0.cpp` / `.h`	Generated vendored Unicode data, one self-contained unit per shipped version, each ending in a `struct Tables` facade.
`punycoder_unicode_version.h` / `.cpp`	Leaf. `PUNYCODER_UNICODE_VERSIONS(X)`, `UnicodeVersion`, `kDefaultUnicodeVersion`, string↔︎enum helpers.
`unicode_tables_registry.h`	Resolves the facade column; included only by the two units that instantiate the pipeline.
`init.c`, `RcppExports.cpp`	C entry points / generated Rcpp glue.

Request lifecycles

`puny_encode("café.com")` (codec surface)

puny_encode() (R) validates the input is character, non-URL-shaped (looks_like_url_input() rejects full URLs with an actionable error), and reads strict.
Dispatch through .call_with_validation() → puny_encode_cpp shim → exports.cpp.
exports.cpp calls PunycodeService, which splits labels and hands each to the selected LabelBackend for the RFC 3492 transform (xn-- framing added at the domain layer).
On failure: strict → Rcpp::stop with a contract prefix; non-strict → NA_character_ for that element.

The codec does no Unicode normalization — that is host_normalize()’s job.

`host_normalize("München.de")` (normalization surface)

Always in-tree, always backend-independent for accept/reject and output (ADR-003):

host_normalize() (R) validates the three logical flags and passes the vector down; it never reads punycoder.strict and never aborts on invalid data.
host_normalize_one (C++) runs the UTS #46 pipeline per element: terminal-dot capture → map (case fold / map / disallow) → NFC → label split → per-label validation + A-label canonical check → Punycode-encode non-ASCII labels → DNS length verification → reassemble.
Invalid data → NA for that element (never throws). Programming errors (wrong type) do abort.

The full normative algorithm and worked examples are in dev/normalization-contract.md.

Backend selection

select_label_backend(BackendPreference) returns a LabelBackend (encode/decode function pointers + a name):

automatic → "libidn2+fallback" when libidn2 is compiled in (libidn2 first, fallback on exception); otherwise "fallback".
"libidn2" forces native; "fallback" forces the in-tree algorithm.

libidn2 is a Punycode accelerator only, never the IDNA engine — normalization is always in-tree, so behavior is identical with or without it (ADR-003). The libidn2 path is Unix-only: configure defines -DPUNYCODER_USE_LIBIDN2 only on Linux/macOS; src/Makevars.win never sets it, so Windows always uses the fallback backend.

Parity between backends is asserted by tests/testthat/test-backends.R via punycoder:::.compare_backends() over the RFC 3492 vectors and representative multi-script domains; those tests skip_if libidn2 is unavailable.

Build system

./configure                         # detects libidn2 via pkg-config
   └─ generates src/Makevars from src/Makevars.in
        ├─ libidn2 found  → -DPUNYCODER_USE_LIBIDN2 + link flags (Unix)
        └─ not found      → in-tree fallback only
src/Makevars.win                    # Windows: fallback only, never sets the flag
./cleanup                           # removes generated src/Makevars

R CMD INSTALL . runs ./configure, which prints the selected backend. After touching // [[Rcpp::export]] attributes, regenerate glue with Rscript -e 'Rcpp::compileAttributes()' (rewrites src/RcppExports.cpp + R/RcppExports.R; commit both). After editing roxygen blocks, run Rscript -e 'roxygen2::roxygenise()'.

Unicode data pipeline

data-raw/generate_unicode_tables.R      # network access happens ONLY here
   ├─ downloads UCD files (cached under git-ignored data-raw/.ucd-cache/)
   └─ writes src/unicode_tables_<tag>.{h,cpp}    ← committed, generated
runtime / build                          # NEVER downloads anything

Normalization depends on this vendored data (combining class, decompositions, composition, UTS #46 mapping/status, combining-mark set, Bidi_Class, Joining_Type). Every output name — filenames, header guard, C++ namespace — derives from the single version string the generator takes as its required command-line argument, and the UCD cache is scoped per version.

Several table sets ship at once (ADR-015). 16.0.0 and 17.0.0 are both compiled in; PUNYCODER_UNICODE_VERSIONS(X) in src/punycoder_unicode_version.h lists them, and the enum, the version strings, the explicit nfc<T> / Normalizer<T> instantiations, and the dispatch switch are all expansions of that one list. Adding a version is two adjacent hand edits: one #include in src/unicode_tables_registry.h, one X(...) row in the list. The switch in host_normalize_one() has no default: label, so doing only half of that is a -Wswitch warning plus a link error.

The version is bound at compile time — one branch per host, never per code point — because the inline table bounds that ADR-013 and ADR-014 depend on cannot survive a function-pointer accessor. Trie shapes differ per version and are not shared.

The public surface defaults to one pinned version per release (currently 17.0.0, kDefaultUnicodeVersion) and selects another per call with host_normalize(unicode_version = ), listing what is available via unicode_versions() (ADR-016). NULL means the pin; an unshipped version is an error, never a fall back. A non-default selection appends +unicode-<version> to the profile token, so a call at the pin yields the bare token. Moving the pin itself is a deliberate, reviewed behavior change that also increments the token’s -vN revision, because the bare token is default-relative — see ADR-004, ADR-017, and dev/normalization-contract.md §3 and §8.

The four accessors that profile hot — combining class, UTS #46 mapping, decomposition, Bidi_Class — are two-stage tries: STAGE2[(STAGE1[cp >> SHIFT] << SHIFT) | (cp & MASK)], two loads and no branches, O(1) for every code point rather than only for ASCII. Identical blocks are stored once, which is what keeps them affordable; the unassigned gaps share one block. Where a table lists nothing below its first code point (U+0300 for combining class, U+00C0 for decomposition) a derived low bound still sits in front of the trie, because answering ASCII from a compare beats answering it from two loads. The two accessors consulted once per label rather than once per code point (is_combining_mark, joining_type) keep the range_lookup binary search.

canonical_compose is keyed on a pair, so it is a trie on the first element with a short scan behind it — the decomposition shape run backwards. The trie maps the starter to a run of (b, c) pairs sorted by b; keying on the first element rather than the second is measured off the data (391 distinct a against 72 distinct b, so runs are a median of 1 long instead of 3) and it is what puts the reject on the trie. Its derived b-bound is exposed inline from the generated header as composes_as_second() and applied by nfc() before the call, because 87% of the compositions nfc() attempts are answered by that bound alone — see ADR-013.

Above all of them sits a quick check: nfc() tests whether its input is already normalized (UAX #15 NFC_Quick_Check plus a combining-class ordering test) and returns it unchanged if so, skipping the decompose → reorder → compose pipeline entirely. Nearly all real host text is already in NFC, so this is the largest single win in the normalizer, and it is largest for ASCII — where nfc_inert(), inlined from the generated header, answers every character with one compare. Maybe is a real third value and falls through to the full pipeline; see ADR-014.

Every one of those arrays, constants, element types and block sizes is derived in the generator from the same UCD vectors the accessor reads, and the generator verifies each trie against its source ranges for all 1,114,112 code points before emitting it. Never hand-write a boundary into the emitted C++ — see ADR-011, ADR-012, ADR-013 and ADR-014.

Test taxonomy (`tests/testthat/`)

Grouped by concern; add tests to the matching file for any user-visible change:

File	Covers
`test-encoding`	`puny_*` codec behavior.
`test-rfc3492`	RFC 3492 golden vectors (`inst/testdata/rfc3492_vectors.csv`).
`test-backends`	libidn2 vs fallback parity.
`test-normalize`	`host_normalize` behavior + profile flags.
`test-idna-conformance`	UTS #46 conformance vectors, once per shipped Unicode version (`inst/testdata/IdnaTestV2-<version>.txt`).
`test-validators`	Predicate/validation helpers.
`test-contracts`	NA / error policy (strict vs non-strict).
`test-unicode`, `test-internals`, `test-lifecycle`, `test-performance`	Supporting coverage.

Where to add what

New host/IDNA behavior → host_normalize / puny_*. URL parsing belongs upstack in rurl; punycoder no longer carries a URL surface (ADR-006).
New error condition → add an ErrorCode + throw_error mapping in punycoder_errors.cpp; R-facing prefixes are contract (ADR-007). Clean rebuild after editing the enum (ADR-009).
Backend-specific code → punycoder_backend.cpp only; never sprinkle #ifdef PUNYCODER_USE_LIBIDN2 through domain code (ADR-008).
New Unicode table set → run Rscript data-raw/generate_unicode_tables.R <version> (the version is a required argument, not a line to edit), re-run it per already-shipped version to confirm git diff src/ stays empty for those, then add the #include to src/unicode_tables_registry.h and the X(...) row to PUNYCODER_UNICODE_VERSIONS — generation and registration in one commit (ADR-015); an unreferenced table object still links in.
Unicode version bump (which table set is the pinned default) → move kDefaultUnicodeVersion; normalization_profile_info() reads it. Follow dev/normalization-contract.md §8.