Sprite unit tests
=================

1. RUNNING TESTS
----------------

To run all tests, say `./run_tests`.

To run a specific test, say, e.g., `./run_tests func_kiel.py`.

To run several tests, use a shell-style pattern, e.g.,
`./run_tests 'unit_p*.py'`, or name several files.  run_tests runs each
file in a child process under a memory watchdog and takes options: the
width (-j), the backend, the files that a change touches (--changed), the
fast tier (--fast), and a listing (--list).  See section 10.

The drivers cache the output of the Curry front end in `.cache/icurry.db`
under this directory (environment variable SPRITE_CACHE_FILE).  A repeated run
compiles only the Curry texts that changed.  The key of an entry names the
route from Curry to ICurry (see section 8), so the two routes never share an
entry.  Set SPRITE_CACHE_FILE to the empty string to run without the cache,
or delete the file to start over.

An entry holds the ICurry of a module and the two interface files the front
end wrote for it, M.fint and M.icurry.  A hit writes all three beside each
other under .curry/sprite-<pakcs>/, and runs the binding optimization over
the FlatCurry file of the front end beside the source as well, when the file
is current, so the pair agrees as after a translation (section 8).  The key
format is 3 since the interface
files joined the entry; the rows of format 2 stay in their own table, unread.
The first run after the change compiles each test module once more and fills
the new table; delete .cache/icurry.db to drop the old rows.

The key also names the version of the route, ROUTE_VERSION of
curry.toolchain._frontend: the steps between the front end and the
translation.  The digest of the route (cache.frontend_digest) covers the
programs, the flags and the sources of the translation and of the binding
optimization, not the step that applies the pass to the FlatCurry file.
The binding rewrite is version 1 (#85, #99), in the key since #101, so an
entry of a route without the step is never served; a bump makes every
entry miss once, and the first run after it compiles each test module once
more.  The rows of the old key stay in the table, unread (KEY_FORMAT does
not change); delete .cache/icurry.db to drop them.  The rows made before
the pass existed were out of reach already,
because the sources of the pass joined the digest with it; the pre-rewrite
ICurry of a tree came from its files on disk (section 8).

The C++ backend keeps a second cache, the product cache, under
`.cache/products` (environment variable SPRITE_PRODUCT_CACHE; the runner and
cytest set it unless the environment names a directory, and the empty string
turns it off).  A compile stores the shared object of a module, its ABI
stamp and the generated C++ there, keyed by the stamp and by the texts of
the module (its source, its ICurry and its JSON) and of its imports, and a
later compile of the same module places them from there instead of running
g++: a pool whose compiled products were removed comes back in under a
second (section 10).  No part of the key names a tree, and an object names
no path (it names the objects of its imports by SONAME, and its record
names the source relative to the object), so a cache that two trees share
serves the second tree the products of the first.  Under interpret:new
nothing is placed (the
mode never compiles).  After its prepare pass the runner prunes this cache
and no other: the digest directories of other runtimes go, so the directory
does not grow with every change to a runtime header that the generated code
includes (the CI cache entry carries it).  Both directories under .cache may
be deleted at any time.


The Python API tests build terms with module calls (M.f(x)), curry.term
for Python values, and curry.parse for Curry text. curry.eval yields Python
values; curry.solve also reports the bindings of named variables. Use
curry.show for Curry text and curry.typeof for types. A list is a list value.

unit_stub_*.py and unit_flow_f*.py are the frozen API acceptance tests:
432 of 435 pass on C++. The three expected failures await the choice
interpreter: unit_flow_f13.TestFlowF13.test_choice_protocol and
test_fork_precedence, and unit_stub_terms.TestStubTerms.test_choice_type_conversion.
unit_flow_f11.TestFlowF11.test_free_argument_answers
passes; partial lists print cons without spaces in values
and bindings (for example {x="":_a}), with nested heads parenthesized.
Run the acceptance with ./run_tests 'unit_stub_*.py' 'unit_flow_f*.py' --backend cxx.

unit_issue103.py, unit_issue104.py, unit_issue108.py and unit_issue137.py
pin the tracker reproducers resolved by the API. unit_issue133.py and
unit_issue134.py pin the LiftGuard reproducers, plain and with checker:true:
#133 expects no value but C++ gives an empty set (two expected failures);
#134 expects [True] once but the C++ runtime give it twice (two expected
failures per backend). Their repair needs runtime work. The free-argument
control in unit_issue134.py checks outside narrowing, including the empty
set for a binding that the function refuses, under the owner's decisions.

Tests of runtime nodes use cytest.raw from lib/cytest/raw.py. raw.expr
builds nodes, raw.values yields evaluation nodes, and raw.show prints them.
The helper also supplies the internal markers and raw.interp for tests of
the runtime, collector, indexing and forwarding. Keep that facility in the
test library; tests of values use the public term and evaluation surface.


2. TEST CATEGORIES
------------------

Three kinds of tests are used:

  * Unit tests:

      These tests, beginning with 'unit_', pinpoint specific code areas.
      Running the unit tests should cover every line of code.

  * Functional tests:

      These tests, beginning with 'func_', compare the output of Sprite to
      another Curry system called the oracle (see section 5).  Their purpose is
      to provide evidence that Sprite behaves as intended.

  * Example tests:

      unit_examples.py runs the examples under examples/ in child processes
      and compares their output with the expected.out file of each example.
      Set SPRITE_UPDATE_EXPECTED=1 to regenerate those files, once per
      backend of EXAMPLES, and review the diff.  The file takes about 40 s
      warm and about two minutes cold on one backend (see section 6).


3. FILE LAYOUT
--------------

The files under this directory are arranged as follows:

        data/  Test data (e.g., source code, sample inputs, expected output).
               data/cxx holds C++ programs that a test compiles against the
               installed runtime and runs (see unit_cxx_variable.py).
        lib/   Test harness code.
        *.py   Test cases.


4. ADVANCED USAGE
-----------------

The following drivers are provided.  All are links to run_tests.

        run_tests             Run unit tests.
        debug_tests           Run unit tests under gdb.
                              (For debugging C++ code.)
        valgrind_tests        Run unit tests under valgrind.
                              (For debugging memory problems.)
        valgrind_debug_tests  Run unit tests under valgrind with vgdb.
        run_benchmarks        Run the benchmarks (see section 9).
        shell                 Start a shell with the test environment.

The test drivers use the same command line interface:

        <driver> [pattern]

The pattern should be quoted.  To run just the unit tests, for example, say:

        ./run_tests 'unit*.py'

run_tests takes options as well (section 10).

The drivers other than run_tests cap the address space of the test process
at 6 GiB, so a diverging program fails with an allocation error.  run_tests
watches the resident set of each file instead and keeps the address-space
limit as a backstop (section 10).  The valgrind drivers run without the
cap.  To change the cap, set SPRITE_TEST_MAX_VMEM_KB (in KiB, or
"unlimited").

The collector of the C++ backend has a stress mode.  With SPRITE_GC_STRESS=1
it runs at every safepoint of the scheduler, after every rewrite step, so a
missing root fails a test at once instead of once per million nodes.  Every
collection of the mode also runs the heap verifier (src/curryrt/graph/gc/
blockheap.cpp), which checks every live node and every node Python holds
against the block heap and aborts the process on an inconsistency;
curryrtbindings.gc_verify() runs one such check between evaluations and
returns the problem found, or None.  To run the unit tests that way, say:

        SPRITE_GC_STRESS=1 SPRITE_INTERPRETER_FLAGS=backend:cxx ./run_tests 'unit_*.py'

A test whose child must grow a large live heap, recurse deeply, or run out
of memory skips in that mode (cytest.skipIfGcStress).  The tests that skip
or shrink there, each with its reason beside it: the examples in
STRESS_EXCLUDED of unit_examples.py (11, 12, 20, 21, 22 and 23, each a
search over a large live state); unit_long_lists.py and unit_cxx_literals.py
(two thousand elements); unit_cxx_rotation.py (the countQueens 8 half of
test_counters_of_a_search skipped, test_counters_repeat on a board of six);
unit_split.py (the solutions of the split programs at a size of five
instead of six);
the walks of unit_cxx_gc.py and unit_cxx_heap.py (shorter); and single
tests of unit_cxx_stack.py, unit_cxx_runtime.py, unit_cxx_gc.py,
unit_cxx_heap.py, unit_cxx_rewrite.py, unit_cxx_scheduler.py and
unit_cxx_gc_counters.py (deep recursions, tests of the threshold and growth
policies, and tests that the mode makes moot).  The nightly CI job
runs the unit suite on the C++ backend in stress mode (job "stress" in
.github/workflows/ci.yml).

The C++ backend can run a module without the C++ compiler: the runtime
interprets the ICurry of the module from a bytecode (src/curryrt/icurry.hpp,
curry.backends.cxx.bytecode).  The interpreter flag ``interpret`` selects
how a module without a compiled object runs.  Its default, 'tiered',
interprets the module at once, compiles it in the background in a child
process (sprite-make --so), and swaps the functions of the module to the
compiled code when the object is ready, also in the middle of a running
evaluation (curry.backends.cxx.tiered, src/curryrt/tiered.hpp).  'new'
interprets such a module and never compiles it, 'all' interprets every
module, the Prelude included, and 'off' compiles every module with the C++
compiler before it runs.  To run the unit tests under one of the other
modes, say, for example:

        SPRITE_INTERPRETER_FLAGS=backend:cxx,interpret:all ./run_tests 'unit_*.py'

The tests that drive the compile steps, or read a file the compiler wrote
for them, set the flag to 'off' for themselves and pass under every mode
(unit_cxx_toolchain.py, unit_make.py, and single tests of unit_api.py,
unit_compile_modules.py, unit_findcurry.py, unit_optimize.py, and
unit_prebuild.py).  unit_optimize_applies.py and unit_optimize_inline.py
pin 'new' for their steps tests, and unit_optimize_applies.py skips one
test under 'all'.  Their tests of values evaluate the unoptimized form of
a module, imported from its ICurry object: under 'off' such a module
compiles on its first use (the trap step of backends/cxx/materialize.py;
issue #102), so those tests run under every mode.  A transformed object
compiles into a directory of the process; the translation on disk
compiles beside the source.  The tests
that read the compiled object of a corpus module (the files named below,
with TestInlineAllocation of unit_cxx_heap.py and TestStepFunctions of
unit_cxx_passthrough.py, which read the objects of Tak1 and Fib under
data/curry/benchmarks) and the two tests of unit_inspect.py that expect a
module loaded from its object (testGetICurry, testGetImpl) skip under
'new' and 'all' (cytest.skipIfInterpreted), as the generated-code tests
of unit_cxx_emit.py skip whenever the flag is not 'off'.  The two tests
of unit_cxx_bom.py that read the object of the Prelude skip when
the Prelude is interpreted: under 'all', and under the other modes
while its object is missing or stale.  The tests that load a compiled
object (curry.load) or run a saved program, which loads its object, skip
under 'all', where the loader refuses to open an object
(cytest.skipUnlessObjectsLoad); curry.save works under every mode, since
the compile step links an interpreted import by its object on disk
(backends/cxx/toolchain.py, Cpp2So._sofilename).  So the unit suite
passes whole under 'all' (issue #82).  unit_cxx_interp.py tests the
interpreter.  The nightly jobs "interpreter" of .github/workflows/ci.yml
run the unit and functional suites under interpret:all (each suite in two
shards of its own, since the interpreter needs more than the limit of one
job for both) and under interpret:off (a job per suite); a workflow_dispatch with the input "interpreter" runs it on
request.  The job passes the flag to run-tests.sh through
SPRITE_TEST_FLAGS.

Under 'tiered', a module compiled from a string and an expression stay
interpreted; a compiled object is loaded only when every import of its
module was loaded from an object, else the module is interpreted too; a
reset, a reload, and the exit of the process cancel the compiles in
flight, so a test that writes a fresh module must wait for its compile
(tiered.wait) before it removes the directory, and a short program under
sprite-exec ends interpreted with no object written.  So a test process
never compiles a shared corpus module whose object is missing or stale,
and a test that inspects the compiled object of a corpus module
(unit_cxx_bom.py, unit_cxx_heap.py, unit_cxx_literals.py,
unit_cxx_partial.py, unit_cxx_passthrough.py, unit_cxx_rewrite.py)
finds it interpreted until the object exists.  The prepare pass of
the runner (--prepare, section 10) runs sprite-make, which always
compiles; run it once after a make stage that changed a runtime
header, which makes every object stale, and the files find the
corpus compiled.  --prepare-only runs the pass alone and exits with
its status; the CI script runs it before the files of a C++
shard.  unit_cxx_tiered.py tests the mode: the swap and its statistics
(tiered.status and the fields swapped and failed_compiles of curry.stats),
the values and steps of both tiers, the swap between the values of one
evaluation and at a safepoint of a long one, a module that imports a fresh
module, the tables and the shim across a reload, an edited module and an
edited import after a reload, a module compiled from a string made again
after a reset, the failure path in the process and under sprite-exec, a
fresh program under sprite-exec, a curry.load of the object of a module
while its background compile runs (the loader waits for the compile, after
a reset as well, and loads the object it wrote; the object of a module
that runs, or ran, interpreted without a library of its name loaded is
refused, before and after a reset; issue #109), and the save-then-load
sequence of the Quickstart twice with an edit between, each in a child
process.  unit_loadsave.py
tests the refusal of a second object of a loaded module name, decided
before the object is opened (issue #109).  unit_reload.py tests that an
edited source is not read again in one process and that the process says
so (issue #110): the warning of import_ once per edit, of :load once per
:load, the warning of a :load or an import of another file of the same
module name, the swap refused for an object compiled from an edited
source, which keeps the module interpreted, and the swap kept for an edit
of a comment (the check is of the ICurry file, not of the JSON).  An interpreted module that applies
readFile of the compiled Prelude to a nondeterministic name raises the
nondeterminism error, as compiled code and a program interpreted whole do;
it raised ValueError while the primitive prim_readFile carried the name of
its wrapper and took its place in the symbol table (the dated TODO entry
on the saturation of apply chains).

The C++ backend rotates its queue of alternatives in one of two modes (the
interpreter flag ``rotation``; src/curryrt/ticker.hpp).  In time mode, the
default of the tools (time:10ms), a ticker thread sets one byte every
quantum and the scheduler polls it after every completed step.  In step
mode (steps:65536) the scheduler rotates every N completed steps, so the
exact steps and forks of a search and the order of its values reproduce
between runs.  The test files run in step mode: cytest sets
SPRITE_ROTATION=steps:65536 before the first import of curry unless the
environment names a mode (an empty value counts as unset), and the runner
(section 10) and the CI script set the same default in the environment of
every file.  To run a file in time mode, say:

        SPRITE_ROTATION=time:10ms ./run_tests unit_cxx_stack.py

The rotation tests of unit_cxx_rewrite.py and unit_cxx_stack.py hold in
both modes: they need a rotation, not a cadence.  Two tests pin the
cadence of 65536 steps and force step mode themselves whatever the
environment says: test_rotation_inside_the_constraint of
unit_cxx_scheduler.py (a scan of preamble lengths over one period) and
test_writeFile_keeps_prefix_across_rotation of unit_io_bind.py (a
rotation inside a write); test_split_suite of unit_benchmarks.py runs its
harness in step mode, because its rows compare the counters of a search
(section 9).  unit_cxx_rotation.py tests the setting, both
modes, the ticker thread (it parks while no evaluation runs, and a child
of os.fork starts one of its own), and runs the rotation tests of the two
files under each mode.

A runtime built with make GC=mps (the Memory Pool System, an experiment
behind a gate; see src/curryrt/graph/gc/mps.cpp and the dated TODO entry)
names its collector in cytest.gc_backend().  The tests of the block heap,
of its threshold policy, of the exact counts of queues and sets, of the
weak free-variable table, and of the out-of-memory child hold for the
default collector alone and skip there (cytest.skipUnlessGcBackend);
unit_cxx_mps.py tests the MPS back end.

A runtime built with make GC_WRITE_COUNTERS=1 counts the writes into old
nodes (the fields gc_old_redexes, gc_old_slot_writes, gc_old_nodes_written
and gc_old_blocks of curry.stats; see src/curryrt/graph/gc/wdgc.cpp and the
sprite-exec page of the documentation).  The default build has none of the
code, and the push and nightly jobs of CI build the default, so the
branches of unit_cxx_gc_counters.py that check the counts
(test_writes_into_old_nodes, test_stress_mode) and the branch of
test_tak_indexes_inline in unit_cxx_heap.py that expects the calls run on
such a build alone: by hand, or through the job "write-counters" of
.github/workflows/ci.yml on request (the input "write_counters").  On the
default build those tests check that the counts are zero and that a
compiled module has no call to the counters.  The macro is part of the ABI
stamp, so the first run after a switch compiles the shared modules again
(--prepare).


5. ORACLE CONFIGURATION
-----------------------

Some tests rely on another Curry system, called the "oracle," to generate
expected results.  To run these tests, you will need access to another Curry
system such as PAKCS or KiCS2.

The oracles are configured through the 'oracle*' scripts in this directory.  To
disable oracle-based tests, remove these or make them non-executable.

The oracle evaluates a goal in the REPL of PAKCS, in safe mode, so a goal of
an I/O type is refused.  The script `oracle_type` prints the type the REPL
infers for an expression (its :type command) without the expression:

        ./oracle_type Prelude '1+2'
        Num a => a

func_goal_defaulting.py compares it with the type Sprite prints.  A golden
result (*.au-gen) that begins with the line "!error" records a goal the
oracle rejected; lib/cytest/oracle.py describes when one is written.  A
golden of a text goal of func_goal_defaulting.py carries the text on its
first line, "!goal <text>", before the result.  The oracle scripts quote the
goal, so a text with a shell glob character, such as 1 ? 2, reaches the REPL
as written.


6. TESTING GUIDELINES
---------------------

Unit tests match the pattern unit_*.py.

  - To run: run_tests 'unit_*.py'
  - Unit tests check isolated code.  The key metric is line coverage.
	- Every new feature requires a unit test.
  - Unit tests are expected to run quickly, as they are part of the
    edit-build-test cycle.  Tests that take a long time to run do not belong in
    the unit test area.
  - unit_sigtable.py compiles data/curry/SigTable.curry on its first run and
    compiles one module from text with the ICurry cache off, so the front
    end runs once per test run; the interface files it reads are the .fint
    files beside the .icy.
  - unit_goals.py runs the REPL (python -m curry.tools.icy) and sprite-exec
    in child processes on the backend of the test process, and compiles
    text goals, so the front end runs for texts the cache does not hold.
    Its module is data/curry/UnsignedGoals.curry.  It also pins the
    command prefixes of the REPL (:l, :e, :t,
    :q): the handlers check the full command name, and commands.eval writes
    the resolved name back.
  - unit_typecheck.py tests the engine of the typed boundary
    (curry.typecheck.engine) on the C++ runtime: unification, the Apply
    rule, context reduction, the defaulting of the front end and of the
    REPL, the exprtype parser, the error catalogue, and the reference
    materializer.  Its module is data/curry/TypeCheck.curry, compiled on
    the first run; it imports Data.Functor.Identity.  The eight cases of
    the PAKCS REPL compare the engine's types and values with the goldens
    of func_goal_defaulting.py under data/curry/goal_defaulting/, so the
    file runs no oracle.
  - unit_typed_expr.py tests module calls, curry.term and curry.parse on
    the C++ runtime: method dictionaries, annotations, construction errors,
    the REPL defaulting table, containers, iterators, free variables,
    recursive bindings and values fed into later calls. The node builder
    checks use cytest.raw for dictionary arguments, forwarding, targets,
    weak type records and the node-walk cap. It imports UnsignedGoals,
    TypeCheck and Data.Functor.Identity. Two tests build 100001-node lists
    to reach the walk cap; they take a few seconds.
  - Tests that call a Prelude primitive directly, without the $## of its
    wrapper, must pass a string in normal form: a list of characters.  The
    typed builder makes the list of characters natively.
  - unit_typecheck_serialize.py tests the serializer of the typed boundary
    (curry.typecheck.serialize) on the C++ runtime: the
    Curry literal of every Python value, the text of every kind of spec,
    the parameters of the operands without text, the let of the free
    markers and of the anchors, the annotations, and the entry points.
  - The exception is unit_examples.py.  It runs every example in a child
    process: about 40 s warm and about two minutes cold on one backend,
    because the first run compiles each Curry module and two examples compile
    Curry text on every run.  It keeps the unit_ prefix so that CI runs the
    examples on every push.  To leave it out of a quick cycle, name the
    files you need, for example run_tests 'unit_prelude.py'.
  - unit_boundary.py runs the drivers of the boundary corpus under
    data/boundary in child processes, as unit_examples.py runs the
    examples: one driver per directory NN-name with a run script, go.py,
    expected.out and a README, on the backends its DRIVERS entry declares;
    SPRITE_UPDATE_EXPECTED=1 rewrites expected.out.  The corpus exercises
    the Python-Curry boundary seam by seam (values in, symbols and
    sharing, free variables and bindings, results out, modules and the
    loader, set functions and the tools, classic Curry programs of PAKCS
    driven from Python), and data/boundary/README.md holds the coverage
    table, the programs Sprite cannot compile, and the attribution of the
    copied programs.  A driver that pins an open issue records today's
    output, so the fix changes the file visibly.  The file runs no oracle:
    an oracle.out beside a driver is a committed transcript of the PAKCS
    REPL on the same goals.  The test also checks the rules of the corpus
    (the files of a directory, the length of go.py, no machine path).
  - unit_configure.py runs CMake in scratch directories and make dry runs;
    it builds nothing.  It checks the configuration options, the installed
    runtime facts, and the thin make wrappers.
  - unit_setup_script.py runs scripts/setup-dev-machine.sh with --dry-run
    against a temporary prefix and checks the printed plan; it installs
    nothing and needs no network.
  - unit_runner.py tests the test runner (section 10) with small Python
    children; it compiles no Curry.

Functional tests match the pattern func_*.py:

  - To run: run_tests 'func_*.py'
  - Functional testing ensures Curry programs produce the correct output.
	- These tests check the behavior of sample programs against other Curry
		implementations or known results.
	- These tests are allowed to consume more resources than unit tests, but
		should still be made as small as possible.  For instance, to check a
		list-reversal program, it is reasonable to reverse a handful of elements,
		not millions.  These tests are not stress tests and do not need to
		explore the limits of the implementation.
  - A heavy corpus runs in several files.  FILE_PATTERN of the driver takes
    one glob or a list of globs, and a source that two globs match is
    listed once.  func_eqconstr.py (the 128 hand-written programs) shares
    data/curry/eqconstr with five files of generated programs, by data
    type (func_eqconstr_a0.py, three parts of a0b0c0, func_eqconstr_a2b1c0.py);
    func_math.py (compare and sort) shares data/curry/math with
    func_math_int.py and func_math_float.py.  unit_func_parts.py checks
    that the parts of a corpus are disjoint and cover it once, one test per
    source.  The runner knows the parts: CORPUS in lib/testrunner/prepare.py
    lists them, and --changed selects func_<dir>*.py for a corpus directory.
  - A goal without a type signature keeps its class constraints as leading
    dictionary parameters.  The driver (lib/cytest/testcase/functional.py)
    reads the number of value parameters from the type scheme of the goal
    and runs every goal without value parameters; curry.eval supplies the
    dictionaries, and the PAKCS REPL defaults the constraints the same way.
    A goal with value parameters is skipped, as before.
  - func_goal_defaulting.py pins the parity of goals and types with the
    PAKCS REPL: the modules under data/curry/goal_defaulting/ through the
    driver, a list of text goals through curry.parse, and
    the :type of the REPL through oracle_type.  The goldens are committed;
    a change to defaulting.curry regenerates them all, about 80 runs of
    PAKCS.  The golden of a text goal records its text on its first line
    and is made again when the text changes.  IO goals stay out, because
    the oracle runs the REPL in safe mode.
  - func_typed_expr.py pins the twins of typed module calls and curry.term against the
    PAKCS REPL: per case a Python call and the equivalent Curry text with
    the Python values written as Curry literals.  The corpus lives in the
    file; the modules under data/curry/typed_expr/ are generated from it
    (python func_typed_expr.py --write, from this directory, in the
    environment of a test run, outside the sandbox) and committed with
    their goldens.  The driver evaluates the Python call in place of the
    goal and compares the values with the oracle modulo variable names and
    float format; the type oracle compares the engine's undefaulted type of
    the call with the type of the goal in the FlatCurry interface, a twin
    with free markers compiled once more with the markers lifted to
    parameters; the conversion rows of the design round-trip through
    curry.parse.  TypeCorpus.curry holds goals with a type
    but no comparable value (function values, types the table rejects,
    IO); the driver skips it by its name.  The REPL of PAKCS refuses a
    goal whose type names a type of a module the goal module imports,
    such as Identity, so the newtype twins reach it through show,
    runIdentity and ==, and a module-local newtype shows the erasure.
    The raw text of the Float rows differs on the C++ backend until #34
    is fixed; test_float_text is an expected failure there.  No goal
    yields an empty string: Sprite prints a raw empty [Char] as [] and
    PAKCS as "".  A regeneration changes the time stamp of a module, so
    the driver remakes its goldens, about a second per goal.
  - func_docs.py runs the two API pages of the documentation as doctests
    (docs/source/Quickstart.rst and docs/source/PythonAPI/UsingTheAPI.rst).
    Each page runs in a child process, from a directory that holds
    examples/Peano.curry, with curry imported, on the backend of the
    session, in step mode, with the doctest option ELLIPSIS.  The
    directives on single examples (SKIP on help, dir and the load of
    Fib.so) are in the pages.  Every example of a page must pass.  Using
    the API names an object file and runs on the C++ backend alone.  The
    Quickstart runs on the C++ runtime up to its section "Saving Compiled
    Curry".  The load of Peano.so in that section, during the background
    compile, is a known failure (issue #109, unittest.expectedFailure).  It
    is a functional test because every run compiles Curry with the front
    end.  It uses no oracle, and no corpus under data/curry, so --changed
    selects it by its own name only.


7. EDITOR INTEGRATION
---------------------

A makefile is used to (possibly) simplify integration with text editors.  For
instance, in vim, pressing <Enter> runs "make" by default.  That shortcut can
be used to easily run the tests directly from a text editor.  Other editors
have similar features.

8. THE ICURRY ORACLE
--------------------

The Python port of the FlatCurry-to-ICurry translation (the package
curry.toolchain.flat2icurry) is checked against the ICurry files that icurry
3.1.0 wrote.  Those files, with the FlatCurry files they were made from, are
in the overlay archive at the root of the repository.  The test
unit_flat2icurry.py checks a fixed sample of 60 files and the probe modules
under data/curry/flat2icurry; func_flat2icurry.py checks the whole archive.
Sprite compiles with the port by default: the Curry front end writes the
FlatCurry, and the port writes the .icy file.  unit_curry2icurry.py checks
that route end to end.  SPRITE_CURRY2ICURRY=icurry selects icurry instead,
when CMake was given -DWITH_ICURRY=PATH.  To check files by hand, say, e.g.,

        ./shell
        python lib/flat2icurry_oracle.py --overlay
        python lib/flat2icurry_oracle.py data/curry/.curry/pakcs-3.4.1/Peano.fcy

The harness prints the first differing byte and the first differing subterm
of each file that differs.

Both routes apply the rewrite step to the FlatCurry file before the
translation (curry.toolchain.flat2icurry.rewrite_file), two passes in
order.  The binding optimization (flat2icurry.bindingopt): a Boolean
equality required to be True, as in the condition of a rule, becomes
constrEq, as the preprocessing of PAKCS makes it.  The set0 lift
(flat2icurry.setlift; decision D28 of 2026-10-12): set0 e over the
captured variables x1 .. xk of e becomes setk L x1 .. xk, with a new
private function L x1 .. xk = e, so the captured variables are arguments
of the capsule, boxed at the entry; a closed set0 is unchanged.  The step
rewrites M.fcy in place when a pass changes the program, in the format of
the front end, and leaves the file alone otherwise; a second run over a
rewritten file changes nothing.  The writer (flat2icurry.flatcurry.write)
reproduces the text of the front end: a file of the front end read and
written again is byte-identical, which func_flat2icurry.py checks over the
whole archive and the products on disk, with the idempotence of both
passes.  The icurry program reads the rewritten file as well, so the two
routes write the same .icy file.  The PAKCS oracle shares the .fcy under
.curry/pakcs-3.4.1/ with the routes, so for a module with a set0 over
captured variables it reads a lifted file: the lifted function is private,
has a type the front end could have written, and the interface M.fint is
never rewritten.  That PAKCS accepts such a file is not yet verified: the
oracle does not run where the lift was made, and no module of the
functional corpus holds a set0 over captured variables (SetLift, FailSib
and SetFunctionsFailures are modules of unit tests).  To verify it, where
the oracle runs: import SetLift under Sprite, then load SetLift in pakcs
and evaluate heap2; record the result here.

A run of the front end on M compiles more than M: an import whose files
are missing, older than its source, or older than the interface of one of
its own imports is compiled again, and its .fcy, .fint and .icurry are
written again in the text of the front end (a make stage that writes the
interfaces of the library anew makes every module of a tree such an import
at its next contact).  The ICurry of such an import is current by the rule
of the toolchain, so nothing translates it again.  So the rewrite covers
every FlatCurry file a run wrote, the module's and those of such imports
(_frontend.curry2flat; issue #99), and the pair beside each other agree.
A test that needs the text of the front end passes rewrite=False.

The oracle compares the translation alone, without the pass.  The .fcy
files of the archive are the front end's and pair with the .icy files
icurry wrote from them.  A product on disk pairs with the rewritten .fcy
beside it, so the plain translation of the file on disk is the .icy on disk
for either route (test_products_on_disk of func_flat2icurry.py).  One
writer outside the toolchain leaves a file in the text of the front end:
the PAKCS oracle (the scripts oracle and oracle_type).  Its :load runs the
front end, which compiles the stale modules of the chain, and its :eval
runs the front end once more for the goal with the targets --acy --flat,
which compiles every module without an .acy file again; the binding
optimization of PAKCS runs over the modules it compiles to Prolog, not
over those files.  The oracle leaves its own products beside such a file,
the Prolog of :load (M.pl) and the AbstractCurry of :eval (M.acy), and the
on-disk check accepts an unrewritten file beside one of them alone, when
the pass over it in memory translates to the .icy beside it (the keyword
accept_unrewritten of flat2icurry_oracle.check_file, which
func_flat2icurry.check_on_disk passes for those pairs; the result carries
the mark unrewritten, and the test reports how many pairs needed it); the
archive is checked without it.  The routes leave no such file: they
rewrite every .fcy a run of the front end wrote, and a hit of the ICurry
cache rewrites the .fcy of the front end as well (below), so a pair that
differs elsewhere is a defect, or a file a front end run by hand wrote.
The oracle writes M.fcy before it compiles the module to Prolog and writes
M.pl, so a functional file that runs beside func_flat2icurry.py in a
parallel run can leave a pair without a product for the time of that
compile; a pair seen then is reported DIFFERENT, and a rerun of the file
passes it (CI runs the files of a shard one at a time).
TestUnrewrittenFile of the same file pins the acceptance, and its
narrowing to the oracle, on a pair of its own.  A pair made before the
routes rewrote the file, an unoptimized .fcy beside an .icy of the former
in-memory pass, differs either way.

A pair made before a pass of the rewrite existed, the .fcy of the front
end beside an .icy translated from it, is stale by the rule of the
toolchain (issue #101; decision D28;
curry.toolchain._curry2icurry.translated_before_rewrite): the .fcy holds a
Boolean equality the binding optimization replaces, and neither file
holds constrEq; or the .fcy holds a set0 call over captured variables the
lift lifts, and the .icy lacks the lifted function.  Such a pair is
current by the file times, and its program binds no variable through ==
in a guard, or keeps the captured variables of a set0 inside the capsule.
The first import of the module, sprite-make and the prepare pass run the
step from Curry to ICurry again: the front end leaves the current file,
the step rewrites it, and the translation follows; sprite-make prints
"sprite-make: pre-rewrite pairs: N translated again", and the prepare
pass repeats the count on the line of the directory and in its summary.
The check on an import reads the .fcy under .curry/pakcs-3.4.1/ beside
the source: a file without an equality name and without set0 has nothing
to rewrite, and a file with constrEq and without set0 was rewritten (a
fraction of a millisecond); else the step runs over the text in memory,
and the .icy is read when a count is not zero.  The parse
costs about 0.3 ms per KB: 1 ms for the median module of the corpus, 9 ms
for the largest that reaches it (assembler of kiel, 28 KB); 679 of the
880 .fcy files of the corpus reach it (an == in an if is common).  The
count is kept for the process by the size and time of the file, so a
module imported through several plans is parsed once per process.  A
.fcy larger than PARSE_LIMIT (256 KB; the Prelude's 840 KB would parse
in 1.2 s) is not judged, with one warning per process.  A .fcy older
than its source is not judged (a hit in the ICurry cache writes no
FlatCurry), nor is a module of the Curry library of the installation,
whose ICurry is committed, nor a .fcy the pass cannot write again (a
read-only file or directory; the import uses the pair as before the
rule, and one warning names sprite-make --rewrite-flat); a pair the
PAKCS oracle leaves, an unrewritten .fcy newer than an .icy of the
rewritten program, is current, since the product is right.  When the
ICurry cache of the drivers holds the entry of a flagged module (make
overlay extracted the archive over a tree that had made the module
already), the cache writes the .icy, and the route runs the pass over
the .fcy of the front end as well (rewrite_on_hit of
curry.toolchain._curry2icurry: a read, and a parse when the text has an
equality name; the file is written when the pass replaces an equality; a
.fcy older than the source, of another version of it, and one the pass
cannot write again are left, the latter with a warning), so the pair
agrees as after a translation; the line of sprite-make counts it apart
("0 translated again, 1 from the ICurry cache").  A tree after an upgrade across the
rewrite holds such pairs for the modules whose source has a required
equality: the products of the archive (atableNoflex and
poker_four_of_a_kind under tests/data, 2 of its 880 .fcy files), a tree
whose products predate 2026-10-06, and copies of either
(scripts/new-worktree.sh copies the products of the front end).  Nothing
is to be done by hand: the first import makes each again, ./run_tests
--prepare does it in one go for the pool and the corpora, and sprite-make
--rewrite-flat M does it for one module, current or not.
unit_curry2icurry_stale.py tests the rule, the key and the line of
sprite-make.  Rewrite the file by hand:

        python -m curry.toolchain.flat2icurry.rewrite M.fcy...

rewrites FlatCurry files in place, and sprite-make --rewrite-flat M runs the
step of a module again, current or not, and makes the ICurry from the
rewritten file (the option implies --icy and stands alone).  The option
--rewrite of python -m curry.toolchain.flat2icurry applies the step to
the program in memory and leaves the file alone.  unit_setlift.py tests
the set0 lift, its plumbing and the programs of data/curry/SetLift.curry.
unit_flat2icurry_bindingopt.py tests the pass, the
rewrite, and the programs that need it, such as sendMoreMoney;
unit_curry2icurry.py checks that both routes write the same .icy for it.
A rebuilt archive packs rewritten .fcy files with the .icy files icurry
wrote from them: the pass is checked against the answers of PAKCS, not
against icurry, and the translation stays checked against icurry.

`make overlay` extracts the test products and copies the interfaces of the
front end, M.fint and M.icurry, beside each extracted .icy file (the target
overlay-interfaces of the root Makefile): an .icy file without them is stale
and would be made again at its first import.  The committed archive predates
the copies; a rebuilt archive packs them.

Between the two steps, the target overlay-prune removes the extracted
products of every test source that git shows changed since the commit that
last changed the archive, in a later commit or in the working tree.  The
extraction gives every product the time of the extraction (tar -m; the
archive records one fixed time, 2000-01-01, for every member), so by the
modification times the toolchain compares an extracted product is newer
than its source whatever the source holds, and a stale product would be
read: CxxGc.curry changed after the archive was packed, and a fresh
checkout followed by `make overlay` failed unit_cxx_gc.py on the old
module.  The rule prints each file it removes; the front end makes the
products again at the first import.  Without a git history (an exported
tree) nothing is pruned, and the rule says so.  OVERLAY_ROOT names the
directory that holds an extracted tests/ tree, for a scratch extraction.

The archive is rebuilt with `make overlay-archive` at the root of the
repository.  The rule packs the .fcy, .fint, .icurry, .icy, and .json.z files
under tests/ and the library interfaces that the front end writes under
curry/lib/.curry, with fixed owner and time stamps.  The .icy files must come
from icurry: run the tests with SPRITE_CURRY2ICURRY=icurry (CMake
-DWITH_ICURRY=PATH) before the rebuild, or the port becomes its own oracle.  Two
probe modules, TypedRoot and Externals, differ on purpose between the two
routes; see data/curry/flat2icurry/README.md.


9. BENCHMARKING
---------------

The benchmark harness (lib/benchmarks) measures Sprite and PAKCS on the
programs under data/curry/benchmarks, which data/curry/benchmarks/README.md
describes.  It writes one JSON record per measurement and compares two record
files.  To run a suite, say:

        ./run_benchmarks [options] [PROGRAM ...]

Say `./run_benchmarks -h` for the options and `./run_benchmarks list -s SUITE`
for the programs of a suite.  The suites are selected with --suite (-s):

        throughput  Evaluates main of each program in a new process:
                    sprite-exec -t --stats -m PROGRAM, or PAKCS with
                    :set +time (--backend pakcs).  The default programs are
                    the 30 programs of the dissertation.
        compile     Compiles each program from source in an empty directory,
                    with the ICurry cache cold and warm, and compiles the
                    expression 1+2 from a Python child (item "expression").
        import      Times the start of the interpreter (item "python"),
                    import curry ("import"), the import of the Prelude
                    ("prelude"), and Hello end to end ("hello").
        memory      Peak memory of each program with the node collector on
                    and, on the C++ backend, off (SPRITE_GC_THRESHOLD).
        split       Evaluates a search program whole and in parts: its
                    search space split by hand into 2, 4, and 8 independent
                    subproblems, each part in a process of its own
                    (sprite-exec -t --stats -m PROGRAMSplit -g partK_I).
                    The programs are those with a module under
                    data/curry/benchmarks/split.
        applications
                    Two programs of the examples against the tools of their
                    trade, in one process each, the same question and the
                    same data on every side (below).

Every measurement runs in a child process under a cap on its address space
(--cap, default 6G) and a timeout (--timeout, default 600 seconds).  The
peak RSS of the child and its descendants comes from GNU time
(/usr/bin/time, package "time"), which the harness runs inside the wrapper
chain; without it the peak comes from wait4, which never reports a child
below the size of the harness itself, and the record says so
(meta.rss_source).  An item gets N measured repetitions (--repeat; the
defaults are 5, 3, 5, 1, 3, and 3 for the six suites) after warm-up runs
that are not recorded (--warmup, default 1).  The warm-up compiles a module
on its first use, so the measured runs find the cached files.  A run that
times out ends its item: a warm-up that timed out is the one sample of the
record, and no repetition follows a measured one that timed out, so a
program past the limit costs one timeout.  --backend selects cxx (the
default), py, or pakcs (throughput only); repeat it for several.  In the
applications suite the opponents are backends too (below).  The backends
and variants of one program run one after the other, so a drift of the
machine hits them alike.

The applications suite measures two programs of the examples against the
tools of their trade.  The programs run in one process each, warm, with the
question and the data the same on every side; the child programs are under
lib/benchmarks/apps.  The item "dependency" asks the solver of example 15
(examples/15-dependency-solver, imported as the driver of the example
imports it) for the first preferred plan of the root of a generated package
index, newest first, or for the answer that no plan exists, and asks
resolvelib, the resolver of pip, the same question with a provider over the
same index whose candidates are listed newest first.  The provider orders
the open requirements as the example provider of resolvelib does, the
package with the fewest candidates first, and resolvelib backjumps to the
cause of a conflict, where the solver of the example takes the demands in
the order it meets them.  So the two sides agree on whether a plan exists,
each plan holds on the index (apps/depindex.py, holds), and the plans are
the same only when the orders of the two searches agree: in the committed
record at 10 and 20 packages, not at 50.  The index is generated from a
fixed seed (apps/depindex.py): the sizes are the package counts of
DEPENDENCY_SIZES in suites.py, 10 to 800, in a solvable and an unsolvable
case, so the variant is SIZE/CASE. The measured time, eval_wall in the
record, runs from the index in Python data to the answer: the conversion
across the boundary (or the tables of the provider) and the search.  The
item "overloads" resolves OVERLOAD_CALLS (1000) distinct calls against one
overload set of eight candidates with example 21
(examples/21-cxx-types-overloads), the overload set built once before the
clock starts, as the driver of the example builds a set once for several
calls, and the terms of the arguments of every call built from Python data
inside the measured time, and against a plain-Python implementation of the
same ranking (apps/cxxoverload.py); the Sprite side runs the Python
implementation on the same calls after the clock stopped and records whether
every answer agrees (extra.probe.agree).  The opponents are backends of the
suite: resolvelib, python, and clang, the overload resolution of clang
itself under -ftime-trace with the process start and the parse of the
declarations subtracted (apps/clangprobe.py).  Without -b the suite runs
cxx, py, resolvelib, and python; clang runs on request (-b clang).  An
opponent that is not installed is skipped: the row of the item says why, and
no record is written.  resolvelib is not part of Sprite: pip install
resolvelib into the Python of the installation, or put a virtual environment
that has it on PYTHONPATH.  The Sprite backends run the modules of the
examples compiled (the interpreter flag interpret:off, set unless the
environment names a mode), so the warm-up run compiles them.  The Sprite
backends run the sizes of SPRITE_DEPENDENCY_SIZES, 10 to 100, because the
solver of example 15 does not end within the timeout above them; the
opponent runs every size.  The nightly job runs the sizes of
NIGHTLY_DEPENDENCY_SIZES, 10 to 50.  The committed record is described in
data/curry/benchmarks/results/README.md: Curry loses at every size, and the
record says by how much.

The records go to standard output, or to the file of -o (appended); the
progress table goes to standard error.  A record is one line of JSON with the
medians of the repetitions: wall and CPU seconds of the child process and its
descendants, the peak RSS in bytes, the time of the evaluation alone
(eval_wall: the -t time of sprite-exec, or the elapsed time of PAKCS), the
compile seconds, the counters steps, forks, and collections of sprite-exec
--stats, and the instructions (below).  The counters are exact and must
agree between the repetitions; a record whose repetitions disagree carries a
warning.  Every record also holds the commit (with -dirty for a changed
tree), the date, a label (--label, for example the name of a machine or of
a step), the samples of the repetitions, and facts about the machine without
a host name (meta: cpu_model, cores, mem_gb in GiB, Python, compiler, front
end).  The fields are documented in lib/benchmarks/records.py.  Records of
the first format (schema 1) load and compare; they have no instructions.

The instructions are the user-space instructions that the child process and
its descendants retire, counted by perf (perf stat -e instructions:u).  The
count does not depend on the load of the machine: on a loaded machine two
runs of Hello differed by 14 percent in CPU seconds and by 0.02 percent in
instructions.  perf costs a few milliseconds of CPU per run, so the harness
counts the instructions in one more repetition after the measured ones and
keeps the measured seconds free of perf.  That repetition is the last sample
of the record ("perf": true); its wall and CPU seconds include the overhead
of perf and never compare with seconds measured without it.  The harness
detects perf at start and says once, in the header of the progress table,
how the instructions are counted or why they are not (the line
"instructions:"; meta.instructions_source in the records).  Without perf, or
with --no-perf, the column is None.  The progress table shows the
instructions in millions (instr_m).  perf needs kernel.perf_event_paranoid
at 2 or below, or the capability CAP_PERFMON; a virtual machine needs a
virtual PMU.

Every run of the harness on a Sprite backend has the rotation of the C++
backend in step mode (SPRITE_ROTATION=steps:65536 in the environment of the
child, unless the environment of the harness or -e names a mode; an empty
value counts as unset), so that the counters steps and forks of a record
reproduce exactly: in time mode, the default of the tools, a rotation
lands on wall time and moves the counters of a search program.  To
measure time mode, say ``-e SPRITE_ROTATION=time:10ms``.  The nightly job
sets step mode in its workflow as well.

To compare two record files, say:

        ./run_benchmarks compare OLD NEW [-m METRIC] [-t FRACTION] [--strict]
        ./run_benchmarks compare --deterministic OLD NEW [-t FRACTION] [--strict]

Items are matched on suite, program, backend, and variant.  The command
first prints the label, the commit, and the machine of each file and warns
when the machines differ.  The ratio is NEW over OLD of the metric (CPU
seconds by default); a ratio within the noise threshold of 1 (10 percent by
default) counts as the same.  The column instr is the ratio of the
instructions when both records have them.  The counters steps and forks
must be equal: a change in them is a change in the program or the compiler,
not in the machine.  With --strict the command exits with status 1 when an
item is slower, a counter changed, a run failed, or an item lacks the
metric: a gate that could not measure has not passed.

With --deterministic the columns that do not depend on the load of the
machine decide.  The instructions are the metric, with a tolerance of 1
percent by default (-t); steps, forks, and collections are listed under
counters when they move beyond the tolerance ("within" marks a move inside
it); wall and CPU seconds are advisory and appear as the ratios ~cpu and
~wall.  The rule of the performance program: the deterministic columns
decide the intermediate steps of a change, on any machine, loaded or quiet;
CPU seconds on a quiet machine decide the gates; CPU seconds never compare
across machines.

On a shared machine, compare with --deterministic; wall and CPU seconds vary
with the load.  Authoritative seconds come from a quiet machine.  A smoke
test and the two comparisons:

        ./run_benchmarks -r 1 -b cxx -o before.jsonl Hello Last2 Peano
        ./run_benchmarks -r 1 -b cxx -o after.jsonl Hello Last2 Peano
        ./run_benchmarks compare --deterministic before.jsonl after.jsonl
        ./run_benchmarks compare before.jsonl after.jsonl

A C++ runtime built with the scheduler counters (make COUNTERS=1; see the
sprite-exec page of the documentation) appends fields to the --stats line:
the steps taken while the queue held one configuration, the steps inside set
functions, the steps on a redex that another configuration created, the
largest queue, and the lifetimes of the configurations.  The harness keeps
the whole line in extra.stats of every sample.  To print them as one row per
program, with the serial, nested, shared, and failed fractions of the steps,
say:

        ./run_benchmarks counters FILE [-b BACKEND] [--csv]

An item measured on a plain runtime shows dashes.  unit_cxx_counters.py
tests the counters; its tests of the values run on an instrumented runtime
only.

The split suite is the process-split experiment of the parallel-evaluation
gate (see the dated TODO entry): what a design that runs independent parts
of a search on separate workers could gain.  To tabulate its records, say:

        ./run_benchmarks split FILE [-m METRIC] [--parts] [--csv]

For every program and part count the row gives the speedup bound (the whole
over the longest part, in CPU seconds by default; -m picks wall or
eval_wall, the time of the evaluation alone), the duplicated work (the sum
of the parts over the whole), and both again by rewrite steps, which are
exact; --parts lists the parts below their row.  A row needs the whole and
every part with status ok.  The split modules take the size of the problem;
unit_split.py checks at a small size that the parts of a split partition the
search space, and unit_benchmarks.py tests the suite and the command.

The nightly performance job (.github/workflows/perf.yml) runs the harness
on the C++ backend every night, and on request (workflow_dispatch): the
throughput suite on a fixed set of ten programs with three repetitions,
the compile suite on the same programs and the expression item, the
import suite, and the applications suite at its nightly sizes against
resolvelib and the plain-Python ranking (the job installs resolvelib with
pip; clang is left out).  The set is NIGHTLY in lib/benchmarks/__init__.py,
and --nightly selects it, so the same measurement runs by hand:

        ./run_benchmarks -s throughput --nightly -r 3 --timeout 300 -o t.jsonl
        ./run_benchmarks -s applications --nightly -b cxx -b resolvelib -b python -o a.jsonl

The job appends the records to the branch perf-history of the repository
with the token of the workflow (no secret), through the history command:

        ./run_benchmarks history DIR FILE...

DIR is a checkout of the branch.  records/SUITE.jsonl holds every record of
a suite in the order of arrival (a record file like any other, so compare
reads it), and points.json holds the points of the chart page, derived
from the record files.  The command prints each new record against the
previous record of its item, as compare does, before it appends; the job
puts that table in the summary of the run.  .github/scripts/run-perf.sh
runs the suites, and .github/scripts/perf-history.sh fetches the
branch into a worktree (or starts it), runs the command, commits, and
pushes.  Timings from shared runners are advisory: the job never fails on
a timing, only on a run that failed.  Read the CPU seconds with the
counters, and take authoritative numbers from a quiet machine.

The page docs/source/extra/perf/index.html (perf/index.html of the built
documentation; plain HTML and JavaScript, no library) draws one figure per
suite from points.json: a panel per item with the metric over time, the
last value, a hollow marker where the steps changed, a cross where a run
failed, a readout on hover, and a table view.  Served from GitHub Pages it
reads the file from the perf-history branch (raw.githubusercontent.com);
elsewhere it reads points.json beside itself, or the URL of ?history=.
unit_perf_history.py tests the page under node, the scripts against a
local remote, and the workflow; unit_benchmarks.py tests the history
command and the nightly set.

10. THE TEST RUNNER
-------------------

run_tests hands its arguments to the runner under lib/testrunner.  The
runner starts one process per test file and backend, in a session of its
own.  It prints one line per file as the file finishes, then a summary
table: file, backend, tests, failures, duration, peak RSS, status.  The
output of each file goes to .cache/runner/<backend>/<file>.log.  The exit
status is 1 when a file failed, crashed, or was killed, 2 when a pattern
matches no test file, and 130 after Ctrl-C.

        ./run_tests [-j N|auto] [--mem GB] [--backend cxx] [--fast [S]]
                    [--changed [REV]] [--prepare] [--prepare-only]
                    [--update-manifest] [--timeout S] [--list] [-v]
                    [PATTERN ...]

Without options every unit_*.py and func_*.py file runs on the C++ runtime,
at the width of -j auto: the budget described
below decides how many files run at once, up to the core count.  The
default is the constant DEFAULT_JOBS in lib/testrunner/__init__.py; -j 1
runs the files one at a time.  The environment of a file names the
installation (SPRITE_HOME), the test library and the Curry pool
(PYTHONPATH, CURRYPATH), the ICurry cache (SPRITE_CACHE_FILE), the backend
(SPRITE_INTERPRETER_FLAGS), the rotation mode of the C++ backend
(SPRITE_ROTATION=steps:65536 unless the environment of the run names a
mode; an empty value counts as unset; section 4), and the warnings of the
Curry front end off (SPRITE_FRONTEND_WARNINGS=0 unless the environment
sets the variable; cytest sets the same default in a file run by hand, so
a run compiles a module with or without a word on its overlapping rules
alike; curry.toolchain._frontend).  The variables of a compiler toolchain
that the shell exports (CC, CXX, CFLAGS, CXXFLAGS, CPPFLAGS, LDFLAGS, AR
and the like: TOOLCHAIN_VARIABLES in lib/testrunner/cli.py) do not reach
a file or the prepare pass: the corpus compiles under the compiler and the
flags of the installation whatever the shell exports, because the ABI
stamp of an object does not record the flags, and the run prints one line
that names the variables it dropped (issue #111).  An inline budget in
SPRITE_INTERPRETER_FLAGS of the run does reach the files and the prepare
pass, which then compile the shared products under it; the stamp does not
record the budget either, so the run prints one line that names it
(issue #116).  A PATTERN is a file
name or a shell-style pattern of file names.  Several patterns select
their union.  -v prints the output of every file.  With -v at width 1 the
output streams as it comes, and the file gets the standard input, so a
breakpoint in a test works (the editor targets of the Makefile pass -v).
Without -v the output goes to the log alone, and the file reads an empty
standard input.

The manifest.  tests/manifest.json maps a file to a backend to its
duration, its peak resident set, the counts of tests and failures, and
the size of the machine of the measurement: cores and memory in GiB.  The
runner reads two things from it.  The duration orders the files, longest
first, and selects the fast tier; a file without a duration is assumed to
take the median of the measured ones, so a measured heavy file starts
before it.  The peak sets the cap of the file: twice the peak, at least
1 GB.  The floor covers the part of the toolchain that runs inside the
test process on a cold tree (the translation of FlatCurry, the JSON
writer, the code generators), which a warm measurement never saw.  The
floor costs width where the budget binds, because the budget sums the
caps: a budget of 3.6 GB (60 percent of 6 GB available) admits three files
at the floor, where caps of twice a 200 MB peak would admit nine.  On the
calibration machine (12 cores, 23 GB) the core count binds first.  A file
without a peak gets 2 GB.  --update-manifest writes the values of the run:
the latest duration and counts of a file that ended by itself with a
unittest verdict, and the largest peak seen in any run.  The durations
and the peaks of the committed manifest are warm-tree numbers: the
products of every module were current when they were measured, except
the peaks that a cold calibration raised.

The budget.  A watchdog thread polls every half second.  It walks /proc
once, matches the session id of each running file in /proc/<pid>/stat,
walks the parent links down from the file's process, and sums the
resident sets of the processes it found: the test process, the Curry
front end, the C++ compiler, and the children of the tests, also one that
started a session of its own (as the benchmark harness and func_complete.py
do).  The processes of one poll are the roots of the next, so a process
whose parent ended stays with its file.  The sum counts a page that two
processes share once per process, so a peak is an upper bound; the caps
come from peaks measured the same way.  A session over the cap of its file
is killed with SIGKILL, and the file is reported as "killed: memory".
While a process of the Curry toolchain runs in the session (the front end,
icurry or PAKCS on swipl, the C++ compiler and the linker: TOOLS in
lib/testrunner/procs.py), the cap is at least 2 GB (TOOL_CAP).  icurry on
swipl alone takes about 1 GB: unit_curry2icurry.py peaks at 131 MB when
the ICurry cache holds its modules and at 1042 MB when it does not, and
whether a tool runs depends on the state of the tree, not on the test.
The line of the file notes the allowance ("over cap 1024 MB while swipl
ran"), and the manifest records the peak, so the cap of the next run
covers it.  The budget counts the caps of the manifest, not the
allowance: on a cold tree many files compile at once, and the memory the
budget leaves free (40 percent of MemAvailable, by default) absorbs their
compilers.  A file past --timeout (1800 s by default) is reported as
"killed: timeout".
A kill reaches every process of the file, also the children that the
timeout command moved to another process group.  The next file starts
when the caps of the running files plus its own stay within --mem (60
percent of MemAvailable at the start, by default) and fewer than -j files
run.  -j auto lets the budget decide, up to the core count.  When nothing
runs, a file starts whatever its cap.  Two rules protect shared files.
The py and the cxx run of one file never overlap, because they share the
ICurry and JSON products of its modules.  A file in EXCLUSIVE
(lib/testrunner/selection.py) runs alone; unit_currylib.py is there
because it removes products of the installed library.  ulimit -v stays as
a backstop: 6 GB or three times the cap, whichever is larger, or
SPRITE_TEST_MAX_VMEM_KB when that is set.  Ctrl-C, SIGTERM, and SIGHUP
end the run the same way: every file is killed, the summary lists the
files that did not run, and the status is 130.

Selection.  --fast keeps the files whose manifest duration is below S
seconds (5 by default) on every backend of the run; a backend without an
entry counts as fast.  The functional tests and the other files that
compile a corpus of their own never enter the fast tier: the files that
CORPUS in lib/testrunner/prepare.py names, today unit_benchmarks.py and
unit_cxx_variable.py (selection.compiles_corpus).  The manifest measures a
warm tree.  On a cold tree, after a fresh checkout or a rebuild of the
runtime, such a file first compiles its corpus: the math corpus (218
programs, in func_math.py, func_math_int.py and func_math_float.py since
the split) takes about 90 s on the C++ backend where the entries of its
three files total 3 s.  A unit file compiles at most a
few modules of the shared pool; --prepare warms the pool first.  To run
the functional tests, name them: run_tests 'func_*.py'.  --changed REV
maps the paths that differ from REV (HEAD by default, with the untracked
files) to test files through RULES in lib/testrunner/selection.py:
src/curryrt and the C++ backend select the unit_cxx files;
the toolchain, curry/, and the ICurry reader
select the compile, cache, toolchain, flat2icurry, curry2icurry, make, and
prelude files, with unit_plan.py, unit_currylib.py and unit_prebuild.py,
and unit_icurry.py respectively; the ICurry analyses select
unit_icurry.py and the optimizer files; expressions.py, the interpreter,
and inspect select the expr, api, conversions, and evaluation files, with
unit_loadsave.py and unit_inspect.py, and the optimizer selects the
optimizer files besides; a test file selects itself; a corpus
directory selects its functional test, data/curry/benchmarks selects
unit_benchmarks.py and unit_cxx_variable.py, and data/curry/flat2icurry
selects the files of the ICurry oracle; a module of the shared pool,
data/curry/M.curry, selects the test files whose text names M or a module
of the pool that imports M (by the import lines, through other modules of
the pool), with func_flat2icurry.py, which checks the products of every
module of the pool, and everything when no file names M or an importer
(the mapping says so); the benchmark harness, the runner,
and the setup script select their own tests; tests/lib selects everything;
documentation selects nothing; an unknown path, and a rule whose test
files do not exist, select everything.  The runner prints the mapping.
--list shows the selection with the manifest numbers, the caps, and the
reasons, and exits.

The prepare pass.  A product under a .curry directory is written in
place by the front end (.fcy, .fint), the ICurry writer (.icy), the JSON
writer (.json.z), the code generator (.py, .cpp), and the C++ compiler
(.so).  Two files that write the same product at once can leave a torn
file for a third to read.  The ICurry cache (SQLite, busy timeout 30 s),
the bytecode cache, the ABI stamp, and the precompiled header are written
atomically.  The ABI stamp of a shared object names the runtime headers
the generated code includes (the include closure of curryrt/curryrt.hpp; a header
of the runtime outside it rebuilds the runtime library alone), the flavor
and the installation it was compiled under, so a product compiled under
another SPRITE_HOME is compiled again here (the object
itself names no path: its imports enter it by SONAME).  --prepare runs
sprite-make over the shared pool (data/curry),
the frozen API fixtures (data/curry/flows),
and the corpus of each selected functional test, one directory and one
process at a time, for each backend of the run, before any file starts.
After it the parallel files only read the products.  Use it on a fresh
checkout, after a change to a source under data/curry, after a change to
a code generator, and after make stage installed new runtime headers of
that closure.
The pass is advisory.  sprite-make runs with -k, so a module that fails
to compile does not stop the others.  The job of its directory ends
"incomplete", not "FAILED": its line names the modules that have no
product after the pass (the .py or the .so under the product directory,
which sprite-make -S names; on the C++ backend under the interpreter flag
interpret set to new or all, the JSON, where sprite-make --so ends because
the runtime interprets the module) and its log
(.cache/runner/<backend>/prepare-<directory>.log); one line after the
pass sums it up ("prepare: 37 modules in 1 directory on cxx, every
product present"); and the last line of the summary table adds
"prepare: N of M incomplete".  sprite-make stores the products it compiles
in the product cache (section 1) and places the cached products of a
module instead of a compile; its line about the counts reaches the log of
the directory (the pass runs it without -q), and the line of the directory
and the summary say how many products came from the cache ("43 of 43 from
the product cache"), and how many pairs made before the binding rewrite
the pass translated again ("2 pre-rewrite pairs translated again"; section
8), and how many the ICurry cache served instead ("1 pre-rewrite pair from
the ICurry cache": the cache wrote the .icy of such a module, and the route
rewrote its .fcy as well).  After
the pass the runner prunes the cache when it
is the runner's own (.cache/products, not a directory the environment
named): the digest directories of other runtimes and the leftovers of an
interrupted store go, and a line says so when something went ("prepare:
product cache pruned: 1 digest directory of other runtimes, 0 temporary
directories"); an installation without a digest prunes nothing.  The exit
status of the run does not
depend on the pass: the test that needs the module reports the failure
in its own terms.  A failed compile that leaves an old product in place
is not seen by the pass; that test reports it too.  A module whose
source names a preprocessor the machine lacks cannot be made from its
source: poker_four_of_a_kind under data/curry/smap names currypp in its
pragma OPTIONS_CYMAKE -F --pgmF=currypp.  make overlay, run after make
stage, supplies its FlatCurry and ICurry (section 8); the pass then makes
the module from them without the front end, and no line names it.
The overlay records a source digest only when the source matches the
commit that packed the archive. A shallow history ending at that commit
cannot establish this provenance and leaves the products without a stamp.
Without them the front end runs currypp and fails, and the module is
without a product as any other: the pass fails, and the line of the
directory adds the cause when the log shows it ("poker_four_of_a_kind.curry
needs currypp, which the PATH lacks; make overlay after make stage
supplies its products").  The examples under examples/ are not covered;
unit_examples.py alone compiles them.

--prepare-only runs the pass alone, for the pool and the corpus of each
selected functional test, one process at a time, and exits with its
status: 0 when every directory compiled whole, 1 when one did not, 130
after an interrupt.  No test file runs; --list shows the directories.  The
CI script (.github/scripts/run-tests.sh) runs it on the C++ backend before
its loop over the files of a shard, each in a process of its own: a fresh
checkout has no compiled objects, and under the tiered default a test
process never compiles a shared corpus module (section 4).  A failed pass
fails the shard.  The workflow extracts the overlay archive (make overlay)
after make stage in every job that runs tests: the products are then newer
than the interfaces of the installed library, so the front end leaves them
alone, and the pass makes poker_four_of_a_kind of smap from its archived
FlatCurry without currypp, which the runner lacks (above).  A plain make
at the root of the repository (the default goal) runs the stage and then
the overlay, in the same order.  The checkout fetches the whole history
for it: overlay-prune compares the sources with the commit that packed the
archive, and a shallow clone whose history ends before that commit prunes
nothing; the rule then warns, with a sentence that names fetch-depth 0
(test_overlay_in_a_shallow_clone of unit_conda.py clones the tree at depth
1 and checks the message).  The rule cannot tell a history that ends
before the commit from one that ends at it, so a clone whose boundary is
the packing commit prunes as a full clone does and warns all the same; the
sentence is conditional for that case.  The extraction lists its members under
make overlay V=1 alone: the list of about six thousand lines filled the
log of every job before.
The script skips the pass under interpret:all and interpret:new, the
modes under which a test process never compiles.

Calibration of a new machine.  On a quiet machine, run every file once
per backend with --update-manifest at width 1:

        ./run_tests -j 1 --backend cxx --update-manifest

Review the diff of tests/manifest.json and commit it.  The durations and
the peaks belong to the machine size recorded in the entries.  A smaller
machine inherits the caps and, with them, a smaller width under its own
budget.  A file that the run killed for memory keeps the peak it reached,
so its cap is larger on the next run: run those files again with
--update-manifest.  A test whose child grows to its address-space cap by
design (unit_cxx_gc.py, unit_cxx_runtime.py) needs a cap above that sum.
Repeat the run after a change that moves the peaks, such as a new runtime,
and after the heavy files are split.

Sprite.Stream and the I/O oracle
-------------------------------
The shim in ../curry/oracle/Sprite/Stream.curry is shared by PAKCS and KiCS2.
Put ../curry/oracle first on their CURRYPATH when running an I/O stream goal;
its yield writes show x on one line. The normal oracle script uses safe mode
and refuses I/O, so unit_stream.py invokes PAKCS directly for this check.
The Sprite library module lives in ../curry/lib/Sprite/Stream.curry.
