Skip to contents

Design decisions

Source: DECISIONS.md

An architecture-decision log for punycoder: the load-bearing choices, why they were made, and what they commit us to. Each entry cross-references the authoritative source; where a decision has a deep normative spec, this log is the index, not the spec.

Companions: ARCHITECTURE.md (how it’s built), AGENTS.md (working conventions), dev/normalization-contract.md (the full normative normalization spec).

Status legend: Accepted (in force) · Superseded · Proposed.

ADR-001 — Scope: a resolvability- and safety-agnostic IDNA primitive

Status: Accepted

Context. punycoder sits at the bottom of a three-package DAG: rurl → {punycoder, pslr}. It is consumed by opinionated upstack packages.

Decision. punycoder is a standards primitive for Punycode and host normalization and nothing more. Opinions about resolvability, safety, and policy never flow down into it. Explicitly out of scope:

  • spoof / homograph / mixed-script / display-safety detection (UTS #39 / UTR #36);
  • URL canonicalization (RFC 3986 / WHATWG);
  • DNS resolvability, registrability, or PSL classification;
  • email/address parsing;
  • per-TLD repertoire / registry-specific IDN tables.

Consequences. A successful host_normalize() result asserts only validity and normalization under the pinned profile — never visual safety. These concerns belong upstack (rurl, pslr, or a policy layer). See the README “Non-goals” section and dev/normalization-contract.md §1.

ADR-002 — Normalization profile: UTS #46 non-transitional STD3, not IDNA2008/2003

Status: Accepted (ratified 2026-06-14)

Context. A host normalizer must commit to one of several incompatible IDNA regimes. IDNA2003 (Nameprep) and IDNA2008 / UTS #46 disagree on real domains via the deviation characters ß, ς, ZWJ/ZWNJ.

Decision. Use UTS #46, non-transitional, UseSTD3ASCIIRules = true, with CheckHyphens, CheckBidi, CheckJoiners, NFC, and VerifyDnsLength all on. Non-transitional processing matches current browser behavior and preserves ß (so faß.de → xn--fa-hia.de, not fass.de).

Consequences. This is compatibility processing, deliberately not IDNA2008 / RFC 5891 conformance — it accepts labels IDNA2008 rejects (e.g. ☕.example → xn--53h.example). It must always be described as a UTS #46 profile. CheckBidi / CheckJoiners are always on and deliberately not knobs; the three relaxable flags (check_hyphens, use_std3, verify_dns_length) are monotone (relaxing only turns rejections into acceptances). Full normative detail: dev/normalization-contract.md §0.1, §3, §10.

ADR-003 — punycoder owns mapping/NFC/validation; libidn2 is a Punycode accelerator only

Status: Accepted (ratified 2026-06-14)

Context. libidn2 is a full IDNA engine. If it performed normalization, behavior would differ depending on whether it was present at build time — violating the requirement that the optional dependency “must not change behavior.”

Decision. The mapping, NFC, label validation, and A-label canonical check are always done in-tree over the vendored Unicode data. libidn2, when present, is used only for the deterministic RFC 3492 Punycode transform of an individual label. RFC 3492 output is fully determined by its input code points, so this substitution cannot change accept/reject or output.

Consequences. normalization_profile and unicode_version are properties of the vendored data, not the backend; they are identical with or without libidn2. Backend parity is a test invariant (test-backends.R). This narrows libidn2’s role from “IDNA engine” to “Punycode accelerator.” See dev/normalization-contract.md §0.2, §6, and ADR-008.

ADR-004 — One pinned Unicode version per release; tables generated offline, never downloaded

Status: Accepted — the “vendor exactly one table set” half amended by ADR-015 (several ship at once) and ADR-017 (which ones, and which is pinned). The offline-generation rule and the one-pin-per-release rule still stand.

Context. IDNA behavior depends on the Unicode version. Downloading UCD data at build or run time would be non-reproducible and would break offline/CRAN builds.

Decision. Vendor Unicode tables in-tree, generated by data-raw/generate_unicode_tables.R, with exactly one version pinned as the default per release (currently 17.0.0). Network access happens only at generation time; the generated C++ (src/unicode_tables_<tag>.{h,cpp}) is committed, and the package downloads nothing at build or run time.

Consequences. Bumping the pinned Unicode version is a deliberate behavior change: edit unicode_version in the generator, regenerate, commit the tables, move kDefaultUnicodeVersion, increment the -vN profile revision (ADR-017), and (per RFC 8753) trigger a pslr compatibility review. The generated tables are never hand-edited. See dev/normalization-contract.md §0.3, §7, §8.

ADR-005 — Two error policies: strict/non-strict for the codec; always-NA for normalization

Status: Accepted

Context. Different callers want different failure handling: some want a hard error, some want per-element NA, and downstream packages want to layer their own policy.

Decision. Two distinct contracts:

  • puny_* / url_* take strict = getOption("punycoder.strict", TRUE) (default set in zzz.R::.onLoad). Strict → the C++ layer throws and exports.cpp converts to Rcpp::stop; non-strict → NA_character_ per element.
  • host_normalize() deliberately does not follow this switch. It always reports invalid input as NA and never aborts on invalid data (only on programming errors), so a caller can layer its own invalid = c("na","error") policy.

Consequences. host_normalize’s contract is separate and must not be “unified” with the codec’s throw-on-strict behavior. See dev/normalization-contract.md §2 and tests/testthat/test-contracts.R.

ADR-006 — The URL surface is removed

Status: Accepted (removal completed; supersedes the earlier “deprecated and slated for removal” decision)

Context. url_encode() / url_decode() / parse_url() were always best-effort host extraction/rewriting — never an RFC 3986 / WHATWG URL parser (no percent-coding, scheme validation, robust port/path/query semantics, full IPv6, or serialization guarantees).

Decision. They were deprecated in 1.2.0 with a .Deprecated() warning, giving CRAN callers one warning cycle, and are removed in the following release: the R functions, the *_cpp shims, and the entire punycoder_url.cpp URL parser/host-classifier are gone. New host work goes through host_normalize() / puny_*; URL parsing/canonicalization belongs upstack in rurl. puny_encode/puny_decode still reject URL-shaped input (looks_like_url_input()) with an actionable error pointing at rurl.

Consequences. punycoder no longer exposes any URL surface — it is a pure IDNA/Punycode/host-normalization engine. The URL-only citations (RFC 3986/3987, WHATWG URL, RFC 5952/4291/6874) left dev/normalization-contract.md with the surface. See NEWS.md for the removal entry.

ADR-007 — R-facing error message prefixes are part of the public contract

Status: Accepted

Context. Downstream packages and tests match on error text.

Decision. The throw_error(ErrorCode, …) map in punycoder_errors.cpp and the prefixes exports.cpp adds (e.g. "Error encoding domain: …") are contract. Changing them requires bumping the tests that assert them.

Consequences. Treat error strings as an API surface, not incidental text. See CONTRIBUTING.md (“Native Code”) and AGENTS.md.

ADR-008 — Confine #ifdef PUNYCODER_USE_LIBIDN2 to the backend adapter

Status: Accepted

Context. Compile-time backend branching scattered across the codebase makes the two build configurations diverge and hard to reason about.

Decision. All #ifdef PUNYCODER_USE_LIBIDN2 guards live in punycoder_backend.cpp. Domain, URL, and normalization code are backend-agnostic and see only the LabelBackend abstraction.

Consequences. Because Windows never defines the flag (Makevars.win), Windows builds always use the fallback backend — a property that stays localized to one file. See CONTRIBUTING.md and ARCHITECTURE.md “Backend selection.”

ADR-009 — Single-header declarations; clean rebuild required after header edits

Status: Accepted

Context. The core declares everything in one header, punycoder_core.h, with implementations split by concern. R’s package build does not track header dependencies.

Decision. Keep all core declarations in punycoder_core.h and split implementations by responsibility into the matching src/*.cpp. After editing the header — especially the ErrorCode enum — do a clean rebuild (rm src/*.o or devtools::clean_dll()); an incremental build silently links stale .o files against the new header (ABI skew).

Consequences. Header edits are a rebuild hazard, not a routine change. Keep new logic in the file that owns its concern rather than spreading it.

ADR-010 — Trailing FQDN root dot is permitted (documented profile divergence)

Status: Accepted

Context. Strict VerifyDnsLength would reject the empty root label of a fully-qualified example.com..

Decision. Capture a single terminal root dot before processing and re-append it after; example.com. normalizes to example.com.. This is the one documented divergence from strict VerifyDnsLength, confirmed against the UTS #46 conformance corpora (inst/testdata/IdnaTestV2-<version>.txt, one per shipped Unicode version), where it accounts for every deviation and no false rejection (57 rows at 16.0.0, 59 at 17.0.0 — the count is a property of the fixture, so it is pinned per version).

Consequences. Leading dots, consecutive dots, and multi-terminal dots remain invalid (empty labels → NA). See NEWS.md (1.2.0 Internal) and dev/normalization-contract.md §4.

ADR-011 — Unicode accessor fast paths are derived in the generator, never hand-written

Status: Accepted

Context. The seven accessors in src/unicode_tables_16_0_0.cpp were seven near-identical binary searches. Profiling host_normalize over a 20k-host corpus put them at 56% of process time — the UTS #46 table alone is 9,185 ranges, ~14 data-dependent, branch-mispredicting probes per code point. Host input is overwhelmingly ASCII, and for ASCII almost every one of those probes is answerable without searching at all.

Decision. Collapse the searches to one range_lookup template (plus a key_lookup for the point-keyed decomposition index; composition is keyed on a pair and keeps its own search), and give each accessor a fast path:

  • a table that lists nothing below U+0080 — combining class, combining marks, decomposition, Joining_Type — takes a bounds test against its own first and last listed code point, answering ASCII with no memory access;
  • a table that covers ASCII — UTS #46 mapping, Bidi_Class — takes a 128-entry direct index instead, since no bounds test can skip it;
  • canonical_compose bounds its second element only: b is always a combining character, while a can be ASCII (the smallest is U+003C).

Every constant and array above is computed in data-raw/generate_unicode_tables.R from the same UCD vectors the search reads. A derived bound therefore stays correct whatever the data does; what a version bump could break is the shape choice. If a bounds-tested table grew down into ASCII, its guard would quietly stop firing and ASCII would fall back into the binary search — a silent performance regression, with no wrong answer to reveal it. The generator asserts the shape it assumed so that bump fails loudly at generation time instead.

Consequences. The fast and slow paths cannot disagree, and a Unicode version bump moves the boundaries automatically. Hand-writing a boundary into the emitted C++ — or “simplifying” a derived constant to a literal — reintroduces exactly the drift this prevents. Measured on a clean -O2 build over a 20k-host corpus, best-of-5 per sample, both build orders: 1.75x for all-ASCII input, 1.36x at 20% non-ASCII, 1.21x at 50%, and 1.12x even for all-non-ASCII input — the last because such hosts still carry ASCII TLDs, dots and digits, and because the bounds tests skip the search for most of the BMP, not just ASCII. No input class regressed. Verified byte-identical on 6,403 conformance inputs × 8 flag combinations.

ADR-012 — The hot Unicode accessors are two-stage tries, chosen and verified by the generator

Status: Accepted

Context. ADR-011 made ASCII free but left the non-ASCII path untouched: a code point outside the fast path still binary-searched, ~11–14 data-dependent, branch-mispredicting probes for the 9,185-range UTS #46 table. Re-profiling host_normalize over an all-non-ASCII corpus put table lookups at 33% of self time — combining_class 10.3%, idna_lookup 5.7%, canonical_decomposition 5.7%, bidi_class 5.6%, canonical_compose 4.1%, is_combining_mark 1.7%, joining_type below the sampling floor. The worst case is not an exotic character but a common one the table does not list: every CJK ideograph walks the whole combining-class table only to conclude 0.

Decision. Convert the four accessors that profile hot to two-stage tries — STAGE2[(STAGE1[cp >> SHIFT] << SHIFT) | (cp & MASK)], two loads and no branches, O(1) for every code point. Leave the rest alone, and say why:

  • is_combining_mark and joining_type are consulted once per label, not once per code point. A trie would add 15–20 KB of permanently cold table to shorten a search that barely registers, so they keep range_lookup and their bounds test.
  • canonical_compose is keyed on a pair of code points and does not fit the shape at all. It keeps its own search and its b-bound (PUNY-mbzhgbta).

Three design points carry the result:

  • Stage 1 stores a block number, not a byte offset. That is what keeps it in uint8_t for three of the four tables, and stage 1 is the array every lookup touches; its width matters more than the shift the CPU pays to undo the encoding.
  • The UTS #46 trie is keyed on distinct (status, mapping) values, not on ranges. Every range carrying no mapping differs only in status, so thousands of disallowed unassigned ranges collapse onto one value — and their blocks deduplicate with each other as a result. This is why the structure is smaller than the range table it replaced.
  • The derived low bound survives in front of the trie. Where a table lists nothing below its first code point (U+0300 for combining class, U+00C0 for decomposition) a compare answers all of ASCII with no memory access, which beats the trie’s two loads. Dropping it measured 3% slower on all-ASCII input, the dominant case in real host data. Tables that cover ASCII have no such bound and go straight to the trie, which is what lets it subsume their 128-entry ASCII arrays rather than sit beside them.

Every array, element type and block size is derived in data-raw/generate_unicode_tables.R, extending ADR-011 rather than qualifying it. The block size in particular is not a tuning constant: the generator builds each trie at every shift from 4 to 10 and keeps the smallest, so a Unicode version bump re-runs that choice on its own. The generator then verifies each trie against the ranges it was derived from for all 1,114,112 code points before emitting it — a stronger check than any conformance corpus can be, and the reason a structural rewrite of a generated file is reviewable at all.

Consequences. Measured on a clean -O2 build over a 20k-host corpus, best-of-7 per sample, A/B/A/B in both build orders: 1.25x all-non-ASCII, 1.16x at 50%, 1.08x at 20%, 1.00x all-ASCII (neutral, by design — that case was already optimal). Table lookups fell from 33% to 17% of self time. The installed shared object went from 510,768 to 445,040 bytes, 64 KB smaller — the size question this work opened with resolved in the opposite direction from what was feared. All surviving payload arrays (MARK_RANGES, COMP_TABLE, JOINING_RANGES, IDNA_MAP_DATA, DECOMP_DATA) are byte identical; CCC_RANGES, BIDI_RANGES, IDNA_RANGES, IDNA_ASCII and BIDI_ASCII are gone, and DECOMP_INDEX lost its now-redundant key field. src/unicode_tables_16_0_0.h is untouched, so the native API is unchanged. Verified byte-identical on 6,403 conformance inputs × 8 flag combinations, and on is_idn/is_punycode/puny_encode/puny_decode over the same corpus.

ADR-013 — Composition is a trie on the first element, and its bound belongs to the caller

Status: Accepted

Context. ADR-012 left canonical_compose on a binary search over all 961 pairs, because a key that is a pair does not fit a trie. Re-profiling after the tries landed made it the largest table cost remaining: 5.3% of self time on an all-non-ASCII corpus, against 12.8% for the four tries combined.

Decision. Two changes, and the second is the one the profile was really pointing at.

  • A trie on one element of the pair. Composition becomes the decomposition shape run backwards: the trie maps the starter a to a 1-based index into COMP_INDEX, which delimits a run of (b, c) pairs in COMP_DATA to scan. Which element is measured off the data, not picked: the 961 pairs hold 391 distinct a but only 72 distinct b, so keying on a leaves runs with a median of 1 and a maximum of 19, where keying on b would leave 3 and 117. Keying on a also puts the reject on the trie, which is where the time went — a starter that never composes (every CJK ideograph, every Hangul syllable) used to walk the whole search only to conclude 0.
  • The b-bound moves to the call site. The profile share was misleading about what was expensive. Tracing the calls nfc() actually makes over a 20k-host corpus found 87% of them answered by the bound alone — b is simply the next character after a starter, and ordinary text is not combining marks. For those the call itself was the entire cost, and no lookup structure can help. The generated header now exposes composes_as_second(b) inline, and compose_pair() applies it before the call. Both bounds are still derived from the pair table, so this exports a generated constant rather than hand-writing one (ADR-011). The Hangul test stays in front of it: Hangul composition is algorithmic and owes nothing to the pair table’s bounds.

Consequences. canonical_compose self time falls from 5.3% to 1.9%, and all table lookups from 16.5% to 12.8%. In isolation the accessor is 6.1x faster on the reject path and 2.5x on the hit path. End to end the change is smaller than either figure suggests, because most compose attempts never reached the table: measured on clean -O2 builds of both versions installed side by side and alternated (no rebuild between samples, min of 9 batched samples per point), host_normalize over 20k hosts goes 1.05x all-non-ASCII, 1.04x at 50%, 1.03x at 20%, and 1.01x — neutral — on all-ASCII input. A three-way run confirms both halves earn their place: the trie carries the non-ASCII end and does nothing at 0%, the inline bound carries the ASCII end. __const grows 5,700 bytes (the trie costs 7.8 KB; dropping the now-implied a field from every pair returns 3.8 KB), which page padding absorbs into a 224-byte increase in the installed shared object.

This is the first change since ADR-011 to touch src/unicode_tables_16_0_0.h. It only adds to it, and the addition is generated like everything else in the file, so the accessor signatures remain the API boundary they were.

Verified in three layers, as ADR-012 established: the generator proves the a → run mapping over all 1,114,112 code points and then checks that each run holds exactly its own pairs in ascending b; a standalone harness links the old and new generated files into one binary and finds no disagreement over 4.9 billion (a, b) pairs, plus the six untouched accessors over every code point; and host_normalize is byte-identical on 6,403 conformance inputs × 8 flag combinations, extended to is_idn/is_punycode/puny_encode/puny_decode. Every array outside the composition section is byte-identical to what ADR-012 emitted.

ADR-014 — NFC is quick-checked, not recomputed

Status: Accepted

Context. ADR-011 through ADR-013 made every Unicode table lookup as cheap as it can be. What none of them questioned is how many lookups the normalizer performs, and the answer was: all of them, always. host_normalize_one() called nfc() unconditionally, so an all-ASCII host ran the full decompose → canonical-order → compose pipeline — two vector allocations and three passes — to produce a byte-identical copy of its input. Instrumenting the composition attempts over a 20k-host corpus made the waste concrete: 295,792 attempts per pass on all-ASCII input, every one of them pointless, plus the decomposition that allocated the vector they walked.

Decision. Implement the UAX #15 “Detecting Normalization Forms” quick check and return the input unchanged when it passes. A sequence is already in NFC when every character is NFC_Quick_Check=Yes and combining classes never decrease within a run of non-starters.

  • NFC_QC is read from the UCD, not derived. DerivedNormalizationProps.txt is already parsed for Full_Composition_Exclusion, so the property costs one more pass over a cached file. It gets the same two-stage trie as its neighbours; Yes is the default and covers nearly the whole code space, which is exactly the case block dedup collapses (6.1 KB).
  • Maybe is a real third value — the character may compose with what precedes it — and falls through to the full pipeline alongside No. Folding it into Yes would be correct on almost every input and wrong on exactly the input the pipeline exists to fix.
  • The bound in front of the check is inlined, as nfc_inert() in the generated header, and is the lower of the two tables’ first listed code point (U+0300). Below it a character has combining class 0 and NFC_QC=Yes, so it can neither be out of order nor fail the check. That answers all of ASCII at one compare per character with no table read and, crucially, no call — the ADR-013 lesson applied at the start rather than discovered afterwards.
  • The generator cross-checks the parsed property against the composition pair table derived independently from UnicodeData.txt: every second element of a pair must not be Yes. A mis-parsed NFC_QC would otherwise be silent.

Consequences. Measured with both builds installed side by side and alternated (min of 9 batched samples, 8 rounds, 20k hosts): 1.22x all-ASCII, 1.21x at 20% non-ASCII, 1.22x at 50%, 1.19x all-non-ASCII — uniform across the range, with all 16 paired comparisons favouring the check. This is the largest single win since ADR-011, and unlike ADR-012 and ADR-013 it helps the all-ASCII case most, because that case was paying the most for nothing.

nfc() self time falls from 6.3% to 2.6% and combining_class from 3.5% to 0.45%; canonical_decomposition and canonical_compose drop below the sampling floor entirely, and all table lookups from 12.8% to 6.3%. __const grows 6,252 bytes and __text 320; the installed shared object steps 445,264 → 462,096 as it crosses a 16 KB page boundary.

The win depends on input already being in NFC, which nearly all real host text is. The constructed worst case — the same corpus transformed to NFD, so the check always fails and its pass is pure overhead — still does not regress: 1.13x at 20% non-ASCII, 1.07x at 50%, 1.01x at all-non-ASCII, because the check abandons at the first offending character rather than scanning to the end.

Verified against the pipeline it skips, not merely alongside it. A harness links nfc() and a copy with the early return deleted into one binary and tests two invariants — that the answers match, and that the skip decision matches whether the full pipeline was a no-op — over every single code point, all 99,825 sequences in NormalizationTest.txt, 280,887,296 pairs (every code point in planes 0–1 against all 2,143 code points that can affect normalization at all), and 438,976 triples plus 10,000 mark quadruples. Zero disagreements. The same run replays the official UAX #15 conformance corpus through nfc(): all 19,965 rows pass, which is a stronger statement about the NFC implementation than anything previously in the suite. host_normalize is byte-identical on the usual 6,403 conformance inputs × 8 flag combinations.

The two halves of the check are separately guarded by tests that were confirmed to fail when the half they guard is removed. The order test needs marks that are NFC_QC=Yes — with Maybe marks the property test fires first and the order test is never reached, so the obvious test case silently guards nothing.

ADR-015 — Several Unicode table sets ship at once, and the version is bound at compile time

Status: Accepted

Context. The package pins one Unicode version, and every table access named it: u16::combining_class, u16::Tables, u16::BidiClass. Supporting a second version meant giving the pipeline some way to reach a chosen table set. The obvious mechanism — a struct of function pointers, or a virtual accessor interface, selected once and passed down — would have quietly undone the two preceding ADRs. composes_as_second() and nfc_inert() are inline in the generated header and applied at the call site precisely because the call was the whole cost: a trace showed 87% of attempted compositions are answered by one of those bounds before any table is read. A function pointer cannot be inlined into a compare, so a runtime accessor would have paid ADR-013 and ADR-014 back in full, on every code point, forever.

Decision. Dispatch on the version once per host, at the entry to host_normalize_one(), and bind everything past that branch at compile time.

  • Each generated table unit emits a struct Tables facade — typedefs for its four enums, inline forwarders for its ten accessors. It is the only thing the pipeline names, so the pipeline never spells a version.
  • nfc() became nfc<T>() and the table-dependent normalize helpers became Normalizer<T>, explicitly instantiated once per shipped version. Each instantiation compiles against one table set, so the inline bounds survive intact — verified in the disassembly, not assumed.
  • PUNYCODER_UNICODE_VERSIONS(X) in src/punycoder_unicode_version.h is the single source of truth: the enum, the version strings, every explicit instantiation, and the dispatch switch are all expansions of that one list. The switch has no default: label, so adding a row without instantiating the pipeline for it is a -Wswitch warning and a link error rather than a silent fall-through to the wrong tables.
  • The facade column stays unexpanded tokens in that header, which is therefore a leaf that includes no table header. Only the two units that instantiate the pipeline expand it, via src/unicode_tables_registry.h. That is what keeps a version bump from recompiling exports.cpp.
  • Enums stay per version. u16::BidiClass and u17::BidiClass are unrelated types and no table enum may appear in punycoder_core.h, punycoder_normalize.h, or any struct crossing the dispatch boundary.
  • Adding a version is two adjacent hand edits — one #include, one X(...) row. #include cannot be macro-generated portably, so those stay manual; both halves fail loudly if you do only one.

Consequences. The mechanism is free. Clean -O2 builds, min of 15 × 3 runs at n=400,000, A/B/A in both build orders: two table sets give 0.194–0.199 / 0.195–0.202 / 0.198–0.199 s (ascii/unicode/mixed) against 0.190–0.191 / 0.198–0.199 / 0.198–0.200 for one. Unicode and mixed are flat; ascii sits within ~2% at the min, and mixed — 70% ASCII — shows nothing at all, so that residue is code layout rather than a per-call cost. The disassembly is the real check and it is unchanged: nfc_inert() is still cmp w22, #0x300 inline in is_nfc() and composes_as_second() still the inline sub w9, w1, #0x300 range test in compose_pair(), once per instantiation.

What it costs is size, and the cost is per shipped version, not one-off: the second table set takes the installed shared object 463,216 → 715,008 bytes and __text 82,132 → 89,876 (the duplicated pipeline). Generation and registration therefore land in one commit — an unreferenced table object still links in, because R builds pass no --gc-sections, so a split would pay the ~250 KB for nothing. How many versions to ship, and which is default, is a policy question deliberately left out of this ADR.

Trie shapes legitimately differ between versions and must not be assumed shared: the UTS-46 mapping trie is block 64 with a uint16_t stage 1 at 16.0.0 (77.9 KB) and block 128 with a uint8_t stage 1 at 17.0.0 (70.8 KB). Every one of those constants is derived by the generator (ADR-011, ADR-012), so this needed no edit to the emitted C++.

Two failure modes are worth naming because neither announces itself. A facade forwarder body must stay fully qualified: inside the struct the member name hides the namespace-scope one, so an unqualified body is infinite recursion, and -Winfinite-recursion is GCC 12+ and misses the indirect case. And nfc<T> must have external linkage — an explicit instantiation of an internal-linkage template produces a symbol no other translation unit can name. The test that each table set reports its own registry version through the facade covers the first of those; it stack-overflows immediately if the forwarders ever lose their qualification.

Correctness gate, run against a one-version build of the same tree: element-wise host_normalize over 7,891 conformance inputs × 8 flag combinations, 0 differences in 63,128 comparisons.

ADR-016 — Version selection is an argument, not an option, and tags the profile token

Status: Accepted. Its “byte-for-byte historical token” claim describes the release that added a selectable version, where the pin did not move; ADR-017 moved the pin and incremented -v1 → -v2. The rule stated here — selecting a non-default set tags rather than bumps — is unchanged.

Context. ADR-015 made two Unicode table sets coexist but left them unreachable: the choice existed in C++ and nowhere in the API. Two questions had to be answered together, because they are the same question asked twice — how a caller selects a version, and how a caller records which one ran.

Selection had three candidate shapes: a per-call argument, a getOption("punycoder.unicode_version") global with a .onLoad default, or both. The option form has real precedent here — punycoder.strict works exactly that way, and every puny_* function reads it.

Decision. A per-call argument, unicode_version, on host_normalize() and normalization_profile_info(), plus an exported unicode_versions() listing what the build ships. No option.

The punycoder.strict precedent does not transfer, and the reason is the distinction the contract already draws. strict is an error-policy preference: it changes what happens to input the package has already judged invalid, never what the answer is. The Unicode version is profile identity — UTS #46 phrases all three of its conformance clauses as “Given a version of Unicode…”, and section 7 of the contract makes unicode_version a reported column that pslr keys reproducibility on. Making identity ambient would let the same source mint different keys in two sessions depending on options set elsewhere, which is precisely what a reproducibility key exists to rule out. host_normalize() is also deliberately off the strict/non-strict switch already, so there was no consistency argument pulling the other way. The option form is therefore absent by decision, not merely unimplemented.

NULL means the pinned default and is the only implicit answer. It does not mean “newest”: a caller who has not thought about versions must keep getting the same answers when a later release compiles in another table set, and “newest” would silently change behavior underneath them.

An unshipped version is an error naming what is available, never a fall back to the pin. A silent fallback is worse than a wrong answer here — it would let a caller record a profile identity describing a normalization that never ran, the failure PUNY-nblrvplp was raised to prevent.

Consequences. The profile token appends +unicode-<version> for a non-default table set, in fixed order after the flag tags, rather than incrementing the -vN revision. Three constraints meet at that choice and only this one satisfies all of them: existing cache keys must not move (a call at the pin yields the historical token byte-for-byte), the token’s one promise is that two genuinely different normalizations can never mint identical() tokens (so the version cannot be omitted), and -vN denotes a change to the profile — non-transitional, STD3, the section 3 parameters — which selecting a table set does not touch. The tag is the same mechanism a relaxed flag already uses, so downstream parsing logic needs no new case.

Adding a table set to a build stays additive: the pin does not move, no default-path result changes, and no existing token is affected. Moving the pin remains a reviewed behavior change under contract section 8 — a separate decision from making the pin selectable, and one that still requires a pslr compatibility review.

The C++ host_normalize_version_cpp introduced in ADR-015 is gone; the version folded into host_normalize_cpp instead. A parallel entry point reachable only from tests can drift from the one users actually reach, and the whole point of the argument is that there is now a single path.

ADR-017 — Ship current + previous, pin the newest, and bump -vN when the pin moves

Status: Accepted

Context. ADR-015 made several table sets coexist and ADR-016 made them selectable, and both deliberately deferred the policy question: how many versions ship, which one is the pin, and what happens to a version when a newer one arrives. With 16.0.0 and 17.0.0 both compiled in, that question could no longer be deferred — 17.0.0 was shipped but unreachable by default, so every default-path caller was still normalizing against data one version behind the newest the package contains.

Decision. Four rulings, taken together because the fourth is a consequence of the second.

  1. Ship current + previous, not an archive. Two table sets is a real migration story — a caller can pin the old version for one cycle while it revalidates — at a bounded cost. N versions turns punycoder into a Unicode data distribution, which ADR-001 puts out of scope.
  2. The pin moves to the newest shipped set: kDefaultUnicodeVersion is now 17.0.0. Unicode version bumps in this pipeline are accept-only in practice — newly assigned code points turn NA into a value and nothing else — so the newest set is the one that answers the most inputs correctly. Leaving the pin behind would mean the package’s own default disagrees with the newest data it ships, which is the worse default to explain.
  3. Retirement is deprecation for one cycle, not a same-release drop. When 18.0.0 is pinned, 17.0.0 stays selectable and 16.0.0 is announced as deprecated in NEWS.md before being dropped in the release after. A caller who named a version explicitly — the only caller affected — gets a cycle’s warning, and unicode_versions() is the programmatic check.
  4. Moving the pin increments the profile revision: -v1 → -v2. This is the consequence of (2) and the one non-obvious ruling.

Why (4). ADR-016 established that selecting a non-default table set appends +unicode-<version> rather than incrementing -vN, on the grounds that the profile — non-transitional, STD3, the section 3 parameters — is unchanged. That reasoning is still correct and is not disturbed here. Moving the pin is a different event, because the token is default-relative by construction: the bare uts46-nontransitional-std3-vN means “whatever the pin is”. Move the pin without bumping and one string denotes 16.0.0 before the release and 17.0.0 after, while 16.0.0 simultaneously starts carrying a +unicode-16.0.0 tag it never had. Two genuinely different normalizations would mint identical() tokens — the one thing the token promises cannot happen.

The alternative considered was to always tag the version and never leave the token bare. It is cleaner in principle and it removes the default-relativity at the root, but it discards the byte-stable historical token ADR-016 preserved on purpose, invalidating every existing key rather than only the ambiguous ones. The -vN mechanism already exists for exactly this and costs nothing, since moving the pin is a NEWS-worthy behavior change either way. Section 8 of the contract already required a -vN bump for “the pinned Unicode data”; this ADR records that the rule was applied rather than waived, and section 3 now states the reason inline so the next pin move does not have to re-derive it.

Downstream evidence made the bump necessary rather than merely tidy. pslr (R/matcher.R:200-209) compares both normalization_profile and unicode_version and rebuilds its index on either mismatch, so it detects the move regardless — its residual exposure is a rebuild cost on load until it reships, not a wrong answer. rurl is the opposite shape: it calls host_normalize() at three sites (R/parse-phases.R:827, :1882, :2163), passes only the three flags, never names a version, and references neither normalization_profile_info() nor unicode_version anywhere. It has nothing cached to go stale, but it also has no detection. A consumer keying on the token alone — which the contract sanctions as a coarse cache key — is exactly the case the bump protects.

Consequences. Default-path results change: over both vendored corpora × 8 flag combinations, the default now matches explicit 17.0.0 in all 102,448 comparisons, and differs from 16.0.0 on 3 rows (16.0.0 corpus) and 5 rows (17.0.0 corpus), every one of them NA → value, zero value changes. That is the accept-only widening (2) assumes, measured rather than asserted.

The cost of the second table set is ~1 MB, not the ~250 KB of .so ADR-015 records: the shared object grows 463,216 → 715,008 bytes and the conformance corpus Unicode publishes with each version adds a further +776 KB installed (+108 KB gzipped). Two versions install at ~2.4 MB and R CMD check raises no installed-size NOTE, which is what makes (1) affordable and also what makes an archive not.

Callers pinned to unicode_version = "16.0.0" are unaffected in result and see their token change from uts46-nontransitional-std3-v1 to uts46-nontransitional-std3-v2+unicode-16.0.0 — both halves move, which is the intended loud miss. Anyone treating the bare token as a durable constant must re-key once; that is the deprecation the -vN digit exists to signal.