Testing the Compiler

In plain words

A change to a compiler can break programs far from the place you touched. So the Turf repository keeps a large collection of small test programs, and runs them all after every change. This page explains that collection for someone changing the compiler. To test your own Turf project instead, read Testing a Project.

How the TurfLang compiler is tested, for someone changing it: what each suite covers, how to run it, and how to add a test that will catch the bug you just fixed. AGENTS.md holds the same conventions in short form; this page explains them.

A project’s own tests use the same runner and the same markers. That side is in Testing a TurfLang project and the reference, Projects and packages: turf test.

1. What runs, and when

CI runs all of them, the suite at every optimisation level. Build first: cmake --build build -j<n> rebuilds build/turfc, build/turf (when Go is installed) and build/turf-api-tests.

Two habits save time. Use about half your cores (cmake --build build -j8, TURF_TEST_JOBS=8), so the machine stays usable. And do not export TURF_HOME for a checkout: it points every build at one checkout’s lib/std, so another worktree or build directory silently tests the wrong standard library. Unset it (env -u TURF_HOME ...) if your shell sets it.

2. The regression suite

./scripts/test_runner.sh            # the compiler's default level
./scripts/test_runner.sh -O0        # every compile at -O0
./scripts/test_runner.sh -O2 -g     # a level and debug info

It is a thin wrapper over turfc --test tests/, the same runner turf test uses for a project. The wrapper adds what is particular to this repository: it skips tests/custom/ and tests/99_known_bugs/ (not live regressions), and it enforces tests/DIAGNOSTIC_GAPS.txt (section 4).

TURF_TEST_JOBS caps how many tests run at once, TURF_TEST_TIMEOUT_SEC bounds one test’s run (raise it for TURF_GC_STRESS=1), and TURF_TEST_EXTRA_EXCLUDE (space-separated paths under tests/) skips tests a special build cannot pass by design. TURF_COMPILER runs the suite against another turfc, such as a sanitizer build.

To run one folder, give the runner that directory:

TURF_TEST_JOBS=8 build/turfc --test tests/36_numeric_semantics

--test takes a directory, not a file. For one file, compile it (and run it, for a plain test) yourself:

build/turfc tests/36_numeric_semantics/10_int_power_exact.tr -o /tmp/t && /tmp/t

The layout

Tests live in numbered topic folders, tests/01_syntax/ to tests/52_arrays_of_references/ and on. Put a test in the folder for the feature it exercises and follow that folder’s naming; add a folder only for a new area. tests/api/ holds the facade tests’ C++ source, not .tr files.

3. Kinds of test

The file name decides what passing means:

A plain test checks behaviour with the test module:

import test;fn main(): int {    test::assert_eq(7 / 2, 3, "integer division truncates");    test::assert((-7) % 2 == -1, "the remainder takes the dividend's sign");    return 0;}

A bare assert(...) is E041: assert lives in the test module, not the prelude.

Pin the reason, not just the outcome

A fail_* test proves the file was rejected, not that it was rejected for the reason it was written for. A typo in the test, or an unrelated error the compiler hits first, passes it just as well. So the runner requires every compile-time fail_* test to say what the compiler must report:

// EXPECT-DIAG: E148// EXPECT-DIAG: 'Point' is not generic -- it takes no type arguments

The markers, all matched literally against the output with colour stripped:

Every line must hold, and they work on plain, fail_* and abort tests alike: a plain test can pin the warnings it compiles with. Match prose, never column numbers. Where one code covers several mistakes (E066 is the type-mismatch catch-all; E056 covers several “no such overload” shapes), pin the message title or a phrase of the message too, since the code alone cannot tell them apart.

A RUNTIME_ABORT_EXPECTED test must carry EXPECT-RUNTIME (or EXPECT-EXIT when the abort is silent by design). Because stdout is flushed before a panic, an abort test can also check what a program printed: print, then panic, and pin both.

After writing a fail_* test, compile it and read the error. Wrong-reason passes have been found this way more than once, for example a test failing on a reserved word used as a name instead of on the rule it was written for.

Helper modules: the _module.tr trap

The runner skips any file whose name contains _module.tr (a substring match, not a suffix of the meaningful name), treating it as a helper with no main. That cuts both ways:

• a real test named something_module.tr is silently never run;
• a helper not named *_module.tr is run as a test and fails with “No entry point found”.

Sweep both directions after adding or renaming a test:

# tests wrongly skipped: named *_module.tr but have a main()
find tests -name "*_module.tr" | xargs grep -lE '\bfn\s+(int\s+)?main\s*\('
# helpers wrongly run: no main(), not *_module.tr, not fail_*
find tests -name "*.tr" -not -name "*_module.tr" -not -name "fail_*" \
  | xargs grep -LE '\bfn\s+(int\s+)?main\s*\('

Everything a module declares is private unless it says pub, so a helper needs pub on exactly what its test uses (E088 names what to add). Do not widen lib/std to make a test pass; declare what the test needs in the test.

4. The diagnostic register

tests/DIAGNOSTIC_GAPS.txt lists the fail_* tests whose diagnostic is not pinned, and the diagnostics a beginner-friendly compiler should not produce:

The runner checks every entry against the real output, so a fixed entry left in the file fails the suite: delete the line when you fix it. When a test cannot be pinned to the diagnostic it was written for, register it rather than weaken the test. New work does not add UNPINNED lines. The file’s header has the exact format and the fix for each kind.

5. Every fix gets a test

A bug fix comes with the test that would have caught it, in the folder for its area: a fail_* test showing the bad case is now rejected, and a plain test showing the good case still works, where both make sense. A crash found by the fuzzer (section 7) gets its minimised input as a fail_* test.

When a fix changes what users see, the same commit updates the reference pages under docs/reference/ and the error pages docs/errors/codes/Exxx.txt and docs/slm/codes/Exxx.txt. They describe the language as it is, never its history.

A new error code

Every error has a code, and every code has two pages and a raiser:

1. Add a constant and a table row at the end of src/diagnostics/ErrorCodes.h (never renumber or reuse a number).
2. Raise it: reportError(Loc, ErrCode::X, ...) in the type checker, LogErrorAt(Loc, ErrCode::X, ...) in the parser, lexer or codegen. The code is a required parameter in each, so forgetting it does not compile.
3. Write docs/errors/codes/Exxx.txt (for people) and docs/slm/codes/Exxx.txt (for the --smart model). Write their examples by running them.
4. Add a fail_* test pinning it, and an assertion of the code and a phrase of the message in tests/api/analysis_test.cpp.
5. Update the code count and category table in docs/reference/cli_and_diagnostics.md.

scripts/check/error_codes_check.sh fails on a code missing a page or a raiser.

6. Checks over the repository

Adding a pub item to lib/std means adding a use of it to tests/46_stdlib_surface/ in the same change; renaming or removing one breaks the 2.x surface.

7. Smoke scripts and other suites

The runner only covers tests/. These cover the rest:

• scripts/smoke/tree_programs_smoke.sh: every lib/std module checked as a module, the prelude modules compiled directly, and every showcase/ program at -O0, -O2 and -g with llvm-dwarfdump --verify. Run it after any language-rule change, since a stdlib or showcase file using a removed form is otherwise never compiled.
• scripts/smoke/opt_levels_smoke.sh: the showcase programs give the same output at every level.
• scripts/smoke/runtime_smoke.sh, gc_smoke.sh, mem_smoke.sh: the C runtime and collector from C.
• scripts/smoke/fmt_smoke.sh: the formatter’s modes and that it is idempotent over the repository. Run it after touching src/tooling/Formatter.cpp.
• scripts/smoke/test_cmd_smoke.sh: breaks each test convention on purpose and checks the run fails for that reason. Run it after changing src/tooling/TestRunner.cpp or TestExpectations.cpp; a suite of passing tests cannot show a convention still bites.
• scripts/smoke/cli_smoke.sh, examples_smoke.sh, release_smoke.sh: the turf front end, the examples/ projects, and an installed release archive.
• scripts/slm/slm_replay_smoke.sh and slm_deterministic_fix_smoke.sh: the --smart path without a model (TURF_SLM_REPLAY). Run them after changing src/slm/SLMIntegration.cpp or Sandbox.cpp.

Each POSIX smoke script with a Windows meaning has a PowerShell twin of the same name; change both together.

The garbage collector

Conservative stack scanning can hide a missed root at -O2, when a value happens to survive in a register. Run tests/41_gc/ (or the whole suite) at -O0 after any change to runtime/turfrt_gc.c or the code that allocates. Every compiled program also reads:

• TURF_GC_STRESS=1: collect on every allocation, the surest way to expose a missed root;
• TURF_GC_VALIDATE=1 (Linux, glibc, x86-64): an independent full-stack mark that panics GC root missed if the real scan missed a block;
• TURF_GC_NO_PROCFS=1: behave as if /proc/self/maps were unreadable.

For an ASan-instrumented runtime, configure a separate build_asan/ with -DTURF_RT_SANITIZE=ON and run the suite with TURF_COMPILER=$(pwd)/build_asan/turfc ASAN_OPTIONS=detect_leaks=0.

Crashes and the fuzzer

Invalid input must produce a diagnostic, never a crash. scripts/dev/fuzz/crash_fuzz.py mutates the repository’s own programs and compiles each; a signal, a sanitizer report, an assertion, LLVM ERROR, an internal compiler error, a hang or exit status 2 is a finding, minimised into fuzz-findings/. Run it against a compiler built with -DTURF_SANITIZE_COMPILER=ON (ulimit -s 65536 first: ASan’s larger frames overflow the default stack on the deep-nesting tests).