Library modules

Clause provides Erlang implementations of OTP library modules, derived from OTP's documented behavior without copying OTP source. These modules use runtime builtins and are compiled with the programs that reference them.

Contents

Each provided module exports the complete OTP 29 interface for that module name. Its <module>.terms inventory records the implementation status of each function; see files.

Results and error reasons match OTP 29, including those produced by Erlang implementations and BIFs. For example, lists:map(F, x) raises {case_clause, x}, lists:nth(0, L) raises function_clause, and lists:seq(1, 10, 0) raises badarg. BIFs such as lists:member/2, lists:keyfind/3, lists:reverse/2 and the maps functions use the applicable badarg, {badmap, M} or {badkey, K} error. lists:member/2 uses exact equality (=:=), whereas lists:keyfind/3 uses non-exact equality (==). lists:sort/1 is a stable runtime builtin; see comparison and order. Stack traces identify Clause's library functions rather than OTP's implementation functions.

Interactive shell

shell:start_interactive/0,1 starts the Erlang shell on the standard streams. clau --shell links the shell into an executable and selects it as the entry point unless --entry specifies another function; see compile options.

The clause_shell process performs the role of OTP's user_drv: it prints the system version, starts the shell server and restarts it after a shell exits through exit().. The server reads commands through io's standard input server, repeats the prompt for each continuation line, maintains history and starts the evaluator. The evaluator checks expressions with erl_lint, expands records and evaluates them with erl_eval. Results use the shell's record definitions and the runtime pretty printer, clause_io_lib:pretty/2; see io_lib. End of input prints *** Terminating erlang (nonode@nohost) and terminates the program with exit status 0.

Shell commands resolve modules by name. --shell therefore links every library module and preserves every export; see export liveness. Without this option, the shell can access only modules referenced by the program. Input uses the terminal's line editing and is read as delivered by the terminal. edlin line editing, history keys and tab completion are not integrated; see differences.

Module interpreter

clause_interp runs a module from its abstract forms, so a program can call modules that were not compiled into it. Clause's transform host uses it for project parse transforms.

• clause_interp:load(Forms) checks the forms with erl_lint, expands records, adds module_info/0,1 and behaviour_info/1 and returns {ok, Module, Warnings}, or {error, Errors, Warnings} in the shape of compile:forms/2. Loading a module of the same name again replaces it; a module the program links cannot be interpreted. clause_interp:unload(Module) removes it.
• Calls of the program's code into the module (M:F(...), apply/3, fun M:F/A, spawn with a module, function and arguments, behaviour callbacks) enter error_handler:undefined_function/3, as OTP's calls of modules that are not loaded do; the handler passes them to the interpreter (dynamic calls). A function the module does not export raises undef.
• erlang:function_exported/3, erlang:module_loaded/1, erlang:loaded/0 and the code module's ensure_loaded/1, is_loaded/1, which/1, all_loaded/0 and module_status/0,1 see the module as loaded from the file of its first file attribute, as code:load_binary/3 with that file name would show it (code module).
• clause_interp:set_path(Dirs) also interprets modules from .beam files in Dirs: a call of a module the program neither links nor interprets loads Dir/Module.beam's debug_info (beam_lib:chunks(File, [abstract_code])) first, as OTP's code server loads a module on its first call; ensure_loaded(Module) does it ahead, returning {module, Module} or {error, {Reason, File}} (no_debug_info among the reasons). The module then shows as loaded from its .beam.
• compile's binaries are interpreter images: clause_interp:beam(Forms, Chunks) wraps linted forms in a BEAM container (an AtU8 chunk naming the module, Attr, compile's chunks and a ClFm chunk with the forms), and clause_interp:load_binary/3 interprets one. code:load_binary/3, code:load_abs/1,2 and code:load_file/1 hand binaries to it when the program has clause_interp (any program that calls compile; code reaches it through a non-literal module, so its users do not link the interpreter); other binaries, images of another module and images of a linked module give {error, badfile}. module_info(compile) then shows the image's compile information, and set_path/1 loads images from .beam files as well.
• Each function runs as an erl_eval fun. Local calls, imported functions and fun F/A reach the interpreter's non-local function handler, and calls in tail position stay tail calls.
• Exceptions leaving interpreted code show OTP's frames for the interpreted functions, {Module, Function, Arity, [{file, File}, {line, Line}]}, with the arguments for function_clause and undef. The interpreter keeps those frames on a stack of its own in the process dictionary; differences lists what it does not reproduce.

<a id="how-programs-get-them"></a>

Module discovery

A compilation batch consists of positional inputs or one project target. If a batch references a module by a literal atom and no input defines that module, the compiler searches for M.erl in the library application directories, then in the project's source_search_paths; see project discovery. The first matching file is parsed and compiled into the batch. Discovery repeats for references in the added modules according to the module resolution order.

References that trigger discovery are M:F(...) with literal M, fun M:F/A, apply(M, F, Args) with literal M, -behaviour(M) and -import(M, ...). A call to a catalog builtin, such as io:format/2 or os:type/0, does not itself trigger module discovery. --print-inputs lists the additional inputs.

• Library roots: library/ in the build tree is located relative to clau, using the same mechanism as the runtime archive. It contains one directory per OTP application, including erts, kernel and stdlib. The second root, library-asn1/, contains modules generated from library ASN.1 specifications; see public_key.
• Headers: -include_lib("kernel/include/logger.hrl") and the corresponding references to file.hrl, eep48.hrl and inet.hrl (#hostent{}) resolve to Clause's headers unless the project defines a kernel application. The same rule applies to stdlib/include/zip.hrl, qlc.hrl, ms_transform.hrl and assert.hrl when no project stdlib application is defined. ms_transform.hrl selects a transform executed by the host OTP; see transforms. assert.hrl provides OTP's macros, error terms and NOASSERT, NODEBUG and ASSERT switches. syntax_tools/include/merl.hrl resolves to Clause's header unless the project defines a syntax_tools application; it selects merl_transform, which runs in the transform host. OTP internal headers erl_bits.hrl and erl_compile.hrl, distribution headers dist.hrl, dist_util.hrl and net_address.hrl, and the SCTP header inet_sctp.hrl are not provided.
• Overrides: a batch module with the same name replaces the library module and produces a warning. Catalog builtins retain precedence; see registration.
• Compilation: library modules are ordinary batch modules. They link into executables and produce separate artifacts for object or IR output. Their Erlang code yields according to the same rules as other Erlang code.
• Dynamic references: a reference using a runtime module name, such as M:F(...) with variable M, does not add a module. The call raises undef unless another reference includes the module in the batch.

Runtime module replacements

Clause provides replacements for OTP modules implemented by ERTS in C, including NIFs, BIFs, port drivers and preloaded modules under erts/preloaded/src. It also replaces modules generated during OTP builds, such as unicode_util. OTP's Erlang kernel and stdlib modules depend on these interfaces.

Provided or missing

Runtime-dependent functionality is classified per module and, where necessary, per function group. A module is either provided or missing:

• Provided: the module exports the exact OTP 29 interface. New modules initially contain stubs for every function so that discovery and caller compilation can proceed. Function groups are then assigned the statuses defined below.
• Missing: no module is supplied. The library catalog records the reason and the higher-level Clause interfaces that replace its callers. Examples include the low-level file and socket modules prim_file, prim_buffer, prim_socket, prim_net, prim_inet and socket_registry. Programs use Clause's file, gen_tcp, gen_udp, inet, net and socket interfaces. Clause modules take precedence over OTP modules, so OTP implementations that depend on these missing modules, such as gen_tcp_socket and inet_tcp, are not discovered. Missing modules have no stubs; unresolved references produce compile-time errors.

Each function in a provided module has one of four statuses:

Stub behavior follows the erlang stubs. Declined features are tracked in the project plan. Functions directly supported by existing runtime services are implemented without a separate classification decision.

A function implemented by a catalog builtin retains its export and the placeholder body F(_, ...) -> clause_builtin_placeholder.. Registration binds the builtin, and qualified calls M:F(...) compile to the bridge. An unqualified call within the module would execute the placeholder, so internal calls to these functions must be qualified. library_inventory rejects placeholders without matching catalog builtins, catalog builtins represented by not_implemented stubs, and unqualified calls to placeholder functions.

<a id="which-module-a-name-resolves-to"></a>

Module name resolution

A host OTP installation is optional and must not affect module selection. Resolution order is:

1. Explicit inputs: sources listed through sources, source_dirs or positional arguments. An input with the name of a Clause module replaces it and produces a warning naming both files.
2. Clause library modules: including provided runtime replacements.
3. Project source_search_paths: see project discovery.

Missing module names are reserved. A matching search-path file, such as OTP's erts/preloaded/src/prim_file.erl, is excluded, and the diagnostic states that Clause does not provide the module. An explicit input can provide the module, with a replacement warning.

ignore_modules also applies to Clause modules. An ignored module is excluded from the batch; an explicit input can then provide that name without a replacement warning.

Files

• Library modules are stored under library/<application>/, using the owning OTP application's name, such as erts for preloaded counterparts and erlang, kernel, or stdlib. Private helpers such as clause_socket, clause_re, clause_unicode, clause_zlib, clause_receive, clause_maps, clause_hashed, clause_array, clause_features, clause_io_lib, clause_disk_log and clause_dets are stored beside the modules they support. Discovery searches every application directory; module names must be unique across those directories.
• The library catalog, compiler/src/driver/library_modules.inc, records module classifications as PROVIDED(Module, Application) or MISSING(Module, Application, "reason"), one entry per line. It is compiled into clau, so it changes with the library sources and clau together and is not read from disk; a changed catalog changes clau's identity and thereby the library and transform host caches. The compiler uses it to identify Clause module names and reserved missing names, and library_modules.py check compares it with the library's files.
• Each provided module has an adjacent <module>.terms inventory containing one {Name, Arity, Group, Status}. entry per OTP 29 export, sorted by name and arity. Group identifies the function group, such as stdlib, distribution or beam. erlang.terms uses {Name, Arity, Flags, Status} instead; see erlang module.
• Export lists are observed from a host OTP 29 through M:module_info(exports) without copying OTP source. Implemented results and error terms match OTP 29; deliberate differences are recorded in differences.

Documented functions

library/documented.terms lists public functions documented by OTP 29 for erts, kernel, stdlib, crypto, public_key, ssl, syntax_tools and compile. Each {Module, Application, Name, Arity}. entry identifies a non-deprecated function whose module and function documentation are not hidden. The file is generated explicitly from host OTP documentation chunks by tests/compiler/library/observe_docs.escript. Undocumented exports remain in the inventories and may remain stubs.

• library_inventory requires every documented function of a provided module to appear in its inventory with a non-stub status. Exceptions are modules listed as partial in library_modules.py, with the completion step identified in the project plan.
• python tests/compiler/library/library_modules.py coverage . reports implemented and documented function counts for each documented module.

Types

Clause modules take precedence over OTP modules, so references such as file:filename() and erlang:time_unit() must resolve to types in the Clause modules. Each provided module whose OTP counterpart exports types includes an adjacent <module>.types.hrl. This header contains the -export_type list, exported type definitions, and required local types and records matching OTP. Definitions are extracted from BEAM debug information or the supplied source of a preloaded module such as erlang; installed documentation chunks do not contain type definitions.

escript tests/compiler/library/observe_types.escript LIBRARY APP MODULE generates the header from the host OTP. --check compares its forms against a new observation as part of CTest library_freshness. library_modules.py check requires each module to include its type header. References to types in modules outside the program, such as unicode:chardata() and calendar:datetime(), resolve to term() without warnings. The same rule applies to unions exceeding the type graph's capacity, such as file:posix(). These library type limitations do not produce diagnostics for the user's program.

Kernel services at startup

OTP's kernel boot sequence starts servers used by OTP's file, io and code, including file_server_2, code_server and a group leader's I/O server. Clause executables do not start these OTP services. Clause's file and io implementations do not require these boot servers; logger starts its own server on first use. An OTP module requiring an absent server fails with noproc, as it does in OTP when that server is absent. OTP code requiring code_server therefore cannot use that service at run time. Starting the complete OTP kernel application is not planned.

Applications

• Resource compilation: application specifications are compiled from program resources rather than discovered on a runtime code path. clau includes every *.app and *.app.src beside batch sources, every *.app in their adjacent ebin directory, and ebin/Name.app or src/Name.app.src for each [targets.options.applications] entry. Resources are compiled into clause_applications, a generated module added whenever the batch includes application_controller. The first file for each application name takes precedence. A resource that lists no modules is assigned the batch modules beside it. Diagnostics identify locations in the resource file.
• Loading: application:load/1 also accepts {application, Name, Keys} terms. An application name without a resource returns {error, {"no such file or directory", "Name.app"}}, matching OTP's missing-file result.
• Initial state: kernel (11.0.4) and stdlib (8.1) are loaded and running at startup, matching OTP 29.1.1's application listing, without application masters. kernel uses the default environment from kernel.app.
• Lifecycle: an application with a start module receives an application master as its process group leader. This supports get_application/0, get_env/1 and start_type/0 within the application. The master relays I/O requests to standard I/O. On stop, it calls prep_stop/1, shuts down the top process, calls stop/1 and terminates the remaining group processes. Termination of a permanent application, or abnormal termination of a transient application, halts the program with OTP's termination message and exit status 1.
• Logger dependency: logger and clause_logger reference application through a runtime module name. These references do not cause application support to be linked unless the program otherwise references application.

Ledger

The project plan schedules implementation of every documented erts, kernel and stdlib module. Real program requirements, initially luerl, determine priorities. Compile and execute each target program, classify diagnostics as unknown module, undefined function M:F/A, unsupported syntax or feature, or other errors, and assign each issue to a plan step, backlog item or recorded decline.

python tests/compiler/library/library_modules.py report FILE classifies saved stderr into tables. The report lists unknown modules with error counts, calling modules and called functions with arities; undefined functions grouped by module; and other messages with occurrence counts.

Tools

The following tools are stored in tests/compiler/library/:

• escript observe_module.escript LIBRARY APP MODULE writes LIBRARY/APP/MODULE.terms from the host OTP's MODULE:module_info(exports). Existing groups and statuses are preserved; new exports receive {Name, Arity, default, stub}.
• python library_modules.py generate LIBRARY APP MODULE creates a module with a stub for every export. Subsequent edits are maintained manually.
• python library_modules.py check ROOT, run by CTest library_inventory without OTP, validates consistency among module classifications, files, inventories, exports, function bodies and catalog builtins. It rejects provided modules with missing files, missing modules with files, inventories for modules not classified as provided, exports absent from inventories and the reverse, stub or declined entries without not_implemented stubs, and implemented or refused entries without an Erlang body or catalog builtin. library_rejects verifies rejection using an intentionally inconsistent fixture library.
• CTest library_freshness, labeled host_otp and full_only, repeats inventory observations and requires unchanged results. It then observes, generates, checks and compiles a new module end to end. library_report runs clau on a fixture program and checks the generated report rows.