Validation

Current baseline

Latest recorded gates on the Windows x64 development host (clang-cl and LLVM SDK 23.1.2, /MT, _ITERATOR_DEBUG_LEVEL=0); later steps record theirs in .agents/00-finished.md:

Running the gate

From an x64 Visual Studio developer shell with C:\Program Files\LLVM\bin on PATH (automatic SDK selection reuses thirdparty/):

cmake --preset debug --fresh -DBUILD_TESTING=ON -G Ninja -DCMAKE_C_COMPILER=clang-cl -DCMAKE_CXX_COMPILER=clang-cl
cmake --build build/debug
ctest --preset debug-fast                  # during development
ctest --preset debug -j 16                 # full mode, at feature completion
cmake --build build/debug --target check-quality      # changed files, one unit per header
cmake --build build/debug --target check-quality-all  # whole tree

• CLAUSE_TEST_MODE=fast runs golden corpora at O0 positional plus O2 specialization-off project, runs mutations once and skips full_only tests (separate consumer projects such as runtime_generated_link, measurements, the host-OTP library_freshness audit, and the slow workflows linking_lto, linking_exports, linking_compile_cache, programs_compile, transforms_cache, printing_source and the qlc_queries executable case). Full mode (default) runs all driver/policy combinations.
• Long tests driven by a Python or CMake script carry a fixed CTest COST: CTest learns a wrapper's cost from its own small CPU time and would start them last.
• Load-sensitive tests (CLAUSE_LOAD_SENSITIVE_TESTS in the top CMakeLists.txt: frontend_cli, codegen_examples, linking_debugger, and the executable cases applications, ets_portions, gen_statems, interpreted_modules, many_ports and supervisors) only timed out or lost timing races under a full parallel run. They DEPENDS on every other test, so they start after the rest of the selection finishes and have the processors to themselves; a failing dependency does not skip them. They reserve four CTest slots instead of two, so half as many of them run at once.
• The executable golden runner gives each clau it starts --jobs 2; a case links two policy combinations at once, so it runs within its two CTest slots instead of taking every hardware thread twice.
• transforms_host_setup (fixture transform_host) builds Clause's transform host before the tests that run library parse transforms (transforms_* and the executable cases whose sources name a parse transform or include ms_transform.hrl, qlc.hrl or merl.hrl), so they do not wait for its lock holding their slots. It starts first and reserves eight slots; with a built host it takes a fraction of a second.
• CLAUSE_TEST_TIMEOUT_SCALE=N multiplies the per-run limits of the golden runner (60 s) and the semantic case runner (15 s) for instrumented builds.
• check-quality, make format and make-format.bat cover files changed since HEAD plus untracked files; cmake/quality_scope.py adds, for each changed header, one translation unit that includes it (its own source, else its first user, from Ninja's dependency data of any configuration; the header's other dependents are left to check-quality-all in the full gate) and, for clang-tidy, units whose compile command differs from the one the last passing run recorded (.state/quality/<build directory>/tidy-commands.json, in the ignored .state/ directory that make clean keeps), so a build rule edit rechecks only the units it changes. Each passing batch merges its units into that record, so after a failing run the next one rechecks only what did not pass. Changes to .clang-tidy, CheckClangTidy.cmake, tidy_filter.py or the quality toolchain template select everything, as do build rule edits before the first recorded run. The scope scripts (quality_scope.py, QualityScope.cmake) and CMakeLists.txt edits that only add, remove or comment source files do not.
• CLAUSE_OPTIMIZED_COMPILER (default ON) compiles clau and the compiler libraries optimized in Debug builds too, keeping their debug information and the Debug inference budgets; tests spend most of their time in clau. When a compiler crash needs stepping through, configure with -DCLAUSE_OPTIMIZED_COMPILER=OFF, rebuild and rerun the failing test.
• Test programs carry no debug information (no PDBs); configure with -DCLAUSE_TEST_DEBUG_INFO=ON to rebuild them for a debugger. Nested native consumers link the parent build's runtime through ClauseRuntimeTargets.cmake instead of compiling it again; only runtime_link builds the runtime standalone.
• Thresholds and suppressions are never raised to pass the gate. Two third-party exceptions are configured (user decision): cmake/tidy_filter.py drops static analyzer reports located in Boost headers (Boost.Multiprecision limb-storage false positives on Linux that rewriting the calling code only moves) and counts them in the output, and bugprone-exception-escape ignores bad_array_new_length (MSVC's map/set move constructors allocate; allocation failure is fatal anyway).
• clang-tidy checks project headers on every host: the header filter accepts / and \ separators.

Linux

The same gate runs on Linux with the pinned LLVM 23.1.2 Linux SDK as host compiler (CC/CXX from its bin/), the system libstdc++, Ninja and CMake 3.28 or newer. Boost 1.90 and toml++ 3.4.0 come from installed packages or are extracted into thirdparty/ (boost_1_90_0/, tomlplusplus-3.4.0/); only Windows downloads them. The release archive's ld.lld links ICU 70 (Ubuntu 22.04) and its lldb links Python 3.14; on hosts without them linking_lto needs ICU 70 on LD_LIBRARY_PATH, and linking_debugger uses GDB.

cmake --preset debug --fresh -G Ninja -DBUILD_TESTING=ON
cmake --build build/debug
ctest --preset debug -j 16

check-quality (Lizard, clang-tidy) runs on the Windows development host only; other hosts build and run the tests (it was clean on Linux once, see history).

Other targets

32-bit and ARM targets are checked without a compiler built for them (no LLVM SDK exists for those hosts):

• A runtime-only build (-DCLAUSE_BUILD_COMPILER=OFF -DBUILD_TESTING=ON) for the target runs the runtime and ABI tests there: natively, in a target userland (chroot), or through the host's emulator (qemu binfmt).
• tests/compiler/executables/cross.py runs every executable golden and program fixture with this host's clau, --target-triple, that build's runtime archive and a --linker that reaches the target's libraries (run.py --target-option passes them through).

Fixtures and provenance

• Normal builds and tests need neither OTP nor its source checkout. Goldens were generated once from OTP and are committed with hashes; hash checks run before any fixture is used.
• Nineteen pattern/guard corpora hold 67,748 native expected values/errors and 106 semantic acceptance rows. Each native corpus runs positional and project drivers at O0/O2 with specialization on/off, local and remote calls.
• Committed Erlang inputs are locally authored (tests/fixtures/patternmatch/fragments/, preprocessor semantic/headers/); fixture_sources enforces isolation from OTP sources.
• Six end-goal program fixtures (tests/fixtures/programs/) hold OTP stdout/exit-status goldens and their compile diagnostics; programs_compile checks them OTP-free and tests/compiler/programs/regenerate.py --check reproduces them under OTP (fixture map).
• Term printing goldens (tests/fixtures/printing/) hold OTP ~w and erlang:display/1 text for 9,542 values plus OTP stdout of compiled display calls; tests/compiler/printing/regenerate.py --check reproduces them (fixture notes).
• Executable golden cases (tests/fixtures/executables/) are source directories plus one golden.json with OTP stdout/exit status and an authored stderr pattern; tests/compiler/executables/run.py links and runs each under the policy/driver matrix (CTest executables_<case>), and regenerate.py --check reproduces them under OTP (case notes).
• Regeneration and live audits are explicit: -DCLAUSE_OTP_AUDITS=ON and tests/compiler/patternmatch/regenerate.py (instructions). Nothing refreshes goldens or the pin silently.

Test design

Behavior is tested through the real CLI, emitted objects, linked native consumers and OTP goldens. Focused unit tests remain only where source cannot reach the state, each with its purpose stated in the test:

• Injected budgets and allocation/IO faults (project_limits, project_creation_failure, codegen_limits, codegen_write_failure, runtime_lifecycle_failure, generated-call fault seams).
• Private ownership, invalid/stale handle and rollback invariants in the parser, semantic type graph and backend.
• Raw word validation and 32/64-bit term boundaries (runtime_immediate, runtime_term_tag, abi_integers).
• Diagnostic sink failures and the stable feature ID snapshot.

Filesystem capability cases (links, case aliases) report per-case skips; they never stand in for a whole-test pass.

Platform and sanitizer status

• Native generated-code execution: Windows x64; Linux x86-64. 32-bit and ARM results (runtime tests and cross-linked goldens) are in the history.
• Apple Silicon: not validated since the backend exists.
• Compiler and runtime ASan, UBSan and LeakSanitizer: Linux x86-64, one combined Debug configuration (-fsanitize=address,undefined -fno-sanitize=vptr -fno-sanitize-recover=undefined), full CTest with CLAUSE_TEST_TIMEOUT_SCALE=10 (instrumented code runs several times slower): no sanitizer report. The compiler, the runtime and the test programs are instrumented; programs clau links take the sanitizer runtime from a clang++ wrapper on PATH, their generated code is not instrumented. The four injected host-refusal tests (codegen_failure_*, runtime_lifecycle_failure, runtime_memory) are outside ASan's scope: its operator new aborts on an impossible size instead of throwing.
• ThreadSanitizer: Linux x86-64, the same instrumentation scope with -fsanitize=thread; full CTest and the concurrency tests five times each report no race. Out of scope as well: the two tests that replace operator new.
• On Windows the prebuilt LLVM SDK still conflicts with instrumented code (annotate_string 0 vs 1).
• Runtime-only ASan passes on Windows with Release probes, /EHsc /fsanitize=address, /MT, Clang's ASan import library and static runtime thunk, and the ASan DLL on PATH.
• Historical macOS arm64 runs (2026-09-19/20) passed full Debug, C++26, ASan+UBSan, compiler-only and runtime-only builds for the parser and project stages; they predate the backend.

Benchmarks

Manual performance comparison with Erlang/OTP. It is never part of CTest, the commit gate or check-quality, and only it needs a host Erlang/OTP; results are host-specific evidence, not gate results.

make-benchmark.bat                         # Windows; set JOBS=N to cap parallel builds
make-benchmark.bat --repeat 9 --only bench_gc,bench_msg
make-benchmark.bat --cpus all                 # unpinned set; default --cpus performance
make-benchmark.bat --baseline HEAD           # performance work: HEAD vs the working tree
make benchmark BENCH_ARGS="--repeat 9"         # other hosts

• Both configure and build a Release clau and runtime in build/bench, then run benchmarks/run.py (--help lists its options).
• Programs: benchmarks/bench_*.erl with bench_util.erl, built unchanged by erlc and clau -O2. Each times its kernels with erlang:monotonic_time/1 after one warm-up call and prints BENCH <name> <nanoseconds>: data generation, data processing, garbage collection, message passing, file I/O (in an empty temporary working directory under build/bench/runs, inside the repository so that a repository antivirus exclusion covers it), loopback sockets and libraries (libraries.qlc: qlc_pt transforms of queries and modules shaped like qlc_SUITE cases, timed together). Iteration counts make every kernel last 1-2 s on OTP, so timer resolution and host noise stay small against it; socket kernels grow the work per connection, not the number of sockets. Untimed, the warm-up and the timed calls each wait until the processes they spawned have exited (processes_left error after 10 s). Each kernel runs in its own fresh process, which exits before the next kernel starts, so no kernel pays for heaps an earlier one grew.
• Every program run is its own Windows job object or POSIX session; the next run starts once all its processes are gone, and leftovers still alive 30 s after exit are killed and reported as a failure.
• CPUs: every process of a run, OTP's and Clause's alike, runs on the --cpus set: performance (default: the cores of the highest Windows EfficiencyClass, Linux /sys/devices/cpu_core/cpus or largest cpu_capacity, so the P-cores of a hybrid CPU and every CPU otherwise), all, a list such as 0-15 or 2,4,6, or a hex mask such as 0x5554. On Windows the run's job object carries the affinity limit before the program resumes, on Linux the program sets it before exec; macOS cannot pin and runs unpinned. Both OTP and Clause start one online scheduler per CPU of the set, unbound to single CPUs. The report header and each history record (cpus) name the set; ratios of runs on different sets are not comparable.
• Systems, on the same host in one run, interleaved: OTP with its default flavor (JIT), OTP -emu_flavor emu when installed (the Windows installer ships only the JIT, so those columns read n/a), Clause HEAD at O2 and with --lto, and only with --baseline REV another Clause commit, built in a cached worktree under build/bench-base/<hash> that shares thirdparty/. A baseline is for performance work: --baseline HEAD measures the committed code against the uncommitted changes in build/bench. The day-to-day trend comes from the history, not from a baseline.
• Report: per kernel the median of --repeat runs (default 3) in milliseconds with six decimals and its spread, the change of Clause O2 against each system (+N%: Clause faster, the other takes N% longer; -N%: Clause takes N% longer; (Fx) the time ratio) with a geometric mean, --lto against OTP and O2, program wall times including startup, executable sizes, failures and timeouts (--timeout, 600 s per run). A default run measures for about 15 minutes.
• Executable sizes, in bytes: every program at -O2, with --lto and at -O0 (an extra build only measured, never run; both skipped by --no-lto) and any baseline at -O2, with a total row and the changes O2 vs O0, --lto vs O2, --lto vs O0 and O2 vs baseline (-N% smaller); and clau at HEAD and the baseline. The history keeps them as sizes_bytes (programs: program → column → bytes, clau: head/baseline → bytes; null where a build failed).
• History: benchmarks/history/<host>.json keeps one record per run day (run date, HEAD hash and commit date, Clause, OTP and any baseline medians, sizes, and Clause's time ratios otp_ratio and baseline.ratio per kernel), benchmarks/history/reports/<host>/<day>-<hash>.txt the text reports; a rerun on the same day replaces both. Records and reports run more than 183 days before the newest are pruned; a run with uncommitted source changes is reported but not recorded. The report's history tables show the ratios of the last 8 run days, which stay comparable when kernels change their work; records from before the 1-2 s rescaling keep only their ratios. The history directory is gitignored: it stays on the host that ran it.
• Listings: with --listings, before measuring, a run writes each program's --print-asm and --print-ir listings (with bench_util, --entry <program> so unreachable library code is left out as in the executable) beside the reports as <program>.debug.asm / <program>.debug.ll (-O0) and <program>.release.asm / <program>.release.ll (-O2, the code the clause-o2 column runs), replacing the previous run's listings, --no-history and uncommitted changes included. --print-ir is the IR before the LLVM pipeline; the assembly comments quote the IR after it. A program that fails to compile loses its stale listings.

History

Condensed from the former per-step records (originals in Git history up to commit 2777c98). Unless noted: Windows x64, LLVM 23.1.2, pin 21776803ecd1, oracle OTP 29.1.1 / ERTS 17.1. Test counts are full CTest passes with zero skips.

PG = pattern/guard plan step; plan 11 step records are archived in .agents/00-finished.md.