Skip to contents

Introduction to punycoder

Bart Turczynski

2026-07-26

Source: vignettes/punycoder-intro.Rmd
punycoder-intro.Rmd

Introduction

The punycoder package provides high-performance Unicode and Punycode encoding/decoding for internationalized domain names (IDNs). It addresses critical gaps in R’s URL processing capabilities by offering reliable, fast conversion between Unicode and ASCII representations of domain names.

Why punycoder?

The Problem

International domain names containing Unicode characters (like café.com or москва.рф) need to be converted to ASCII format for use in many network protocols and systems. Existing R packages have limitations:

  • Inconsistent legacy helpers: Existing workflows may produce incorrect punycode output
  • Limited functionality: No comprehensive IDN handling
  • Performance: No efficient bulk processing

The Solution

punycoder provides:

  • Reliable encoding/decoding following RFC 3492 standards
  • High performance for large datasets
  • Comprehensive validation with informative error messages

Basic Usage

Domain Encoding and Decoding 

library(punycoder)

# Encode Unicode domains to ASCII
puny_encode("café.com")
# Returns: "xn--caf-dma.com"

puny_encode("москва.рф")
# Returns: "xn--80adxhks.xn--p1ai"

# Decode ASCII domains back to Unicode
puny_decode("xn--caf-dma.com")
# Returns: "café.com"

# Vectorized operations
domains <- c("café.com", "москва.рф", "北京.中国")
encoded <- puny_encode(domains)
print(encoded)

Validation and Utilities 

# Check if domain is already punycode
is_punycode("xn--caf-dma.com") # TRUE
is_punycode("café.com") # FALSE

# Check if domain contains Unicode characters
is_idn("café.com") # TRUE
is_idn("example.com") # FALSE

# Comprehensive domain validation
result <- validate_domain(c("café.com", "invalid..domain", "valid.org"))
print(result)

Host Normalization

puny_encode() is the raw RFC 3492 transform: it Punycode-encodes the labels you hand it, and nothing more. host_normalize() is the higher-level operation. It applies the UTS #46 canonical-host pipeline — character mapping, case folding, NFC normalization, label validation — and then encodes, so it answers “what is the canonical ASCII form of this host?” rather than “what is the Punycode of this string?”.

library(punycoder)

# Mapped, case-folded, validated, then encoded
host_normalize("Café.Example.COM")
#> [1] "xn--caf-dma.example.com"

# The codec transforms what it is given, and leaves case alone
puny_encode("Café.Example.COM")
#> [1] "xn--Caf-dma.Example.COM"

Invalid Input Returns NA

host_normalize() does not follow the strict/non-strict switch described above. It is a separate contract: a host that fails UTS #46 validation is reported as NA and never raises an error, so callers can layer their own policy on top.

host_normalize(c("valid.example", "example..com", "-bad-.example"))
#> [1] "valid.example" NA              NA

Profile Identity

Normalization is pinned to a named profile. normalization_profile_info() returns that identity as a one-row data frame, which is what you record alongside results you intend to reproduce later:

normalization_profile_info()
#>                         profile unicode_version  idna transitional use_std3
#> 1 uts46-nontransitional-std3-v2          17.0.0 uts46        FALSE     TRUE
#>   check_hyphens check_bidi check_joiners verify_dns_length  backend
#> 1          TRUE       TRUE          TRUE              TRUE fallback

Unicode Version Selection

A build of punycoder ships a set of vendored Unicode table versions, one of which is the pinned default:

unicode_versions()
#> [1] "16.0.0" "17.0.0"

Any shipped version can be selected per call. The difference between versions is confined to code points assigned in the newer one — a character that does not exist yet in the older tables is disallowed there, so the host is invalid:

host <- paste0(intToUtf8(0xA7CF), ".example")

host_normalize(host, unicode_version = "16.0.0")
#> [1] NA
host_normalize(host, unicode_version = "17.0.0")
#> [1] "xn--078a.example"

Selecting a non-default version is reflected in the profile token, so the identity you record always describes the normalization that actually ran:

normalization_profile_info(unicode_version = "16.0.0")$profile
#> [1] "uts46-nontransitional-std3-v2+unicode-16.0.0"

The argument defaults to NULL, which means the pinned version — deliberately not “the newest available”, so adding a table set in a later release cannot silently change results under existing code. Naming a version the build does not ship is an error listing what is available, never a quiet fall back to the pin.

See ?host_normalize for the relaxable validation flags and the full profile parameters.

Data Analysis Workflows

Bulk Domain Processing 

# Example: Processing large datasets
set.seed(123)
sample_domains <- c(
  rep("example.com", 1000),
  rep("café.com", 1000),
  rep("test.org", 1000)
)

# Efficient vectorized encoding
system.time({
  encoded_domains <- puny_encode(sample_domains)
})

# Check results
table(is_punycode(encoded_domains))

Error Handling

The package provides robust error handling with informative messages:

# Strict validation (default)
try({
  puny_encode(c("valid.com", "")) # Empty string causes error
})

# Non-strict mode returns NA for invalid input
result <- puny_encode(c("valid.com", ""), strict = FALSE)
print(result)

# Validation provides detailed error information
validation <- validate_domain(c("valid.com", "invalid..domain", ""))
print(validation)

Performance Considerations

The package is designed for high-performance processing:

  • Vectorized operations: Process thousands of domains efficiently
  • C++ backend: Native implementation for speed
  • Memory efficient: Handles large datasets without excessive memory use
# Benchmark with large dataset
large_domains <- rep(c("example.com", "café.com"), 5000)

system.time({
  encoded <- puny_encode(large_domains)
})

# Should process 10,000+ domains per second

Package Options

You can configure package behavior using R options:

# Set global strict validation
options(punycoder.strict = FALSE)

# Check current setting
getOption("punycoder.strict")

# Set encoding preference
options(punycoder.encoding = "UTF-8")

Integration with Other Packages

punycoder is designed to integrate well with other R packages:

# With data.table
library(data.table)
dt <- data.table(
  original = c("café.com", "москва.рф"),
  encoded = puny_encode(c("café.com", "москва.рф"))
)

# With dplyr
library(dplyr)
domains_df <- data.frame(
  unicode_domain = c("café.com", "москва.рф")
) |>
  mutate(
    ascii_domain = puny_encode(unicode_domain),
    is_international = is_idn(unicode_domain)
  )

Next Steps

Technical Details

The package uses a C++ backend with Rcpp for performance, and follows RFC 3492 standards for punycode implementation. When libidn2 is available at build time, punycoder uses it behind the same R-level API and falls back to the built-in implementation otherwise.

See also

punycoder is used as the Punycode and IDNA engine by two sibling packages:

  • pslr — Public Suffix List engine. Uses punycoder for host canonicalization before PSL matching. Reach for it when you need eTLD or registrable-domain queries.
  • rurl — Full URL parsing, normalization, cleaning, and joining toolkit. Builds on both punycoder and pslr to handle the complete URL processing pipeline.

Acknowledgments

punycoder descends from the earlier R hrbrmstr/punycode package and implements published standards directly: RFC 3492 (Punycode), the IDNA2008 RFCs, Unicode UTS #46, and UAX #15 normalization, validated against the Unicode Consortium’s IdnaTestV2 conformance data. It is built on Rcpp with an optional GNU libidn2 backend.

The full list of credits — prior art, dependencies, the standards this code implements, and the data sources it serves — is in ACKNOWLEDGMENTS.md.