Clause
← All documentation

Parse transforms

Using parse transforms

clau --transform-path ebin -o app app.erl

Clause applies {parse_transform, Module} the way erlc does. It exports the parsed module as OTP abstract format forms, runs the transform module in Clause's transform host (or on an Erlang/OTP 29 installed on the host), imports the returned forms as the module's new syntax and compiles that (compiler/src/transforms/). Generated programs never need OTP.

OTP behavior

From pinned lib/compiler/src/compile.erl (transform_module, foldl_transform, maybe_strip_columns) and observed under OTP 29.1.1:

Clause pipeline

  1. Preprocess and parse the module. Its preprocessor and parse errors and warnings go to the transforms as {error, {Location, clause_source, Text}} (warning) forms before eof, as epp's do; the forms that failed are left out. They are reported after the transforms' own messages, unless a transform failed, as erl_lint reports epp's; errors then fail the module.
  2. Collect the transforms in OTP's order: --parse-transform options and the project's parse_transforms, then -compile attributes. A module without any skips the rest and never starts OTP.
  3. Remove those options from the forms and export them: {Line, Column} annotations as erl_parse assigns them, epp's file attributes, and eof. The predefined functions are not part of the forms yet.
  4. Run the loader once for the module in the transform host, or on the host OTP for the chains it takes.
  5. Import the returned forms by turning them into tokens located at their annotations and parsing them with Clause's parser; a file attribute switches the file of the following forms as in erl_lint.
  6. Add the predefined functions, then lint, analyze and compile as usual.

The actions that run transforms are listed under using parse transforms.

Interchange

Importing forms

Returned forms, and .abstr inputs (positional or project sources), become tokens located at their annotations and go through Clause's parser, so they get the same syntax checks as source; operator precedence decides where parentheses are added. {Line, Column}, a bare Line (shown as file:line:) and property lists with location and file are accepted; generated is ignored. Line 0, which OTP's qlc transform gives generated code, takes the place of the node imported before it. A file attribute switches the file of the following forms; when that file exists, imported code keeps its debug lines and source comments. Malformed forms are errors at their annotation (unknown expression bogus, expected {Name, Arity}, invalid annotation) and the module fails. OTP's {error, _}/{warning, _} forms that transforms return are formatted by the loader with Module:format_error/1 (as {Location, clause_message, Text}) and reported with that text; in .abstr inputs they are reported with their descriptor, which needs OTP to format. clause_source forms are reported by clau itself. Importing an exported module and exporting it again gives the same forms.

Host OTP

Transform host

Clause's transform host runs the transforms Clause's library provides (ms_transform and qlc_pt among them) without a host OTP. It is the loader module linked with the whole library and every export kept, as --shell links it, with clause_transform_loader:main/1 as its entry (compiler/src/transforms/host.cpp, compiler/src/driver/transform_host.cpp).

Loader

compiler/src/transforms/loader/clause_transform_loader.erl is project-owned and embedded into clau; the transform host is built from it too. Each run writes it and the request into a fresh temporary directory, clause-transform-* below <build_dir>/tmp (build directory) or, for positional inputs without --build-dir, the system's temporary directory, removed after the run; erl compiles and loads the loader in memory. The request holds the forms, the options, the transform list, the module's file name, the code path and the project transform sources as {File, Forms}. With a build directory, the reply and the loader's output are kept in the build cache and reused while the request and everything it depends on are unchanged; the host's sticky module list is kept the same way. The loader:

  1. checks the OTP release;
  2. loads the project transform sources: the host OTP compiles their forms in memory (compile:forms/2) and loads them, the transform host interprets them;
  3. for each transform: loads it, strips columns as parse_transform_info/0 asks, calls parse_transform/2 and formats warnings and errors to {File, Location, Text} with Module:format_error/1;
  4. writes {ok, Forms, Warnings} or {error, Errors, Warnings} and halts.

Transform modules

A transform module is looked up in this order:

  1. Precompiled .beam files in the directories of --transform-path DIR (repeatable) and the project option transform_paths, which go on the loader's code path before OTP's. This is the only way to use a transform that comes from a dependency library: build that library with the host OTP first (for example erlc +debug_info -o deps/lib/ebin deps/lib/src/*.erl); Clause never builds dependencies. The .beam must come from the host's OTP major release.
  2. Clause's library (ms_transform, qlc_pt, ...), run in the transform host when every transform of the module is there.
  3. The current project's sources: a Module.erl in the batch or its source_search_paths, together with the project modules it calls by name (found the same way). clau preprocesses and parses each with the target's include directories and defines, runs its own transforms and sends its forms to the loader on every run. The search passes over sources that ignore_dirs or ignore_modules leave out, as the module search does. Clause looks for these sources once per batch and keeps them in memory; the loader still compiles them for every module the transform runs on, so precompile a large transform to save that time (with debug_info for the transform host). Modules the host loads from its sticky directories (kernel, stdlib, compiler, such as ms_transform or qlc_pt when OTP's sources are on a search path) cannot be reloaded, so the host's own modules are used: their sources, and the modules only they call, are not looked for or compiled. Clause asks the host erl for these modules once per invocation.
  4. OTP's own modules from the host's code path, for those Clause's library does not provide. OTP ships erl_id_trans only as an example source, not as a module.

Reading a project transform can fail where its real build succeeds: headers of dependencies, a transform of the transform that cannot run, or macros that only another build tool defines. Clause then reports its own errors against the transform's source with the hint to precompile it into a .beam with the host OTP and pass its directory with --transform-path; the host OTP's compiler errors for its forms get the same hint. A project transform is also compiled by Clause as an ordinary module when it is part of the batch; its own calls (erl_syntax, erl_parse, ...) may then need library modules Clause does not ship, so keep transforms out of a target's sources when only the transform needs them.

Diagnostics

SituationMessage
No host OTPFile: parse transform 'M' needs Erlang/OTP 29 on the host; pass --erl
--erl names no executableFile: erl not found: PATH
Wrong OTP releaseparse transforms need Erlang/OTP 29 on the host, found N
Transform not foundFile: undefined parse transform 'M' (OTP's text)
Transform raisedFile: error in parse transform 'M': ... (OTP's text)
Transform warning or errorFile:Line:Column: Text as returned
Transform source does not compileits messages, then the precompile hint
No host OTP, .beam without debug_infoFile: parse transform 'M' in PATH has no debug_info; pass --erl to run it on the host Erlang/OTP
A transform path does not existFile: parse transform path is not a directory: DIR
A .beam on a transform path does not loadFile: cannot load parse transform 'M' from PATH: Reason
Loader failed, timed out or wrote no replyparse transform loader failed: ...
The transform host cannot be builtFile: cannot build the parse transform host (...) and the build's output
Invalid returned formsthe parser's diagnostic at the form's location

Options

CLIProject ([targets.options])Meaning
--parse-transform Mparse_transforms = ["m"]Apply M to every module, before its own -compile transforms (CLI ones first)
--transform-path DIRtransform_paths = ["deps/*/ebin"]Code path for precompiled transform .beam files; patterns as in source_search_paths
--erl PATH or none—Host erl executable, which then runs every chain that is not all library transforms; none never runs host code (host OTP)
--print-abstr—Print the forms after transforms instead of compiling

A project that uses its own transform and a precompiled one from a dependency:

[[targets]]
name = "app"
sources = ["src/*.erl"]

[targets.options]
source_search_paths = ["transforms"]  # transforms/pt_rewrite.erl, compiled for each build
parse_transforms = ["pt_dep"]         # applied to every module of the target
transform_paths = ["deps/*/ebin"]     # deps/ptdep/ebin/pt_dep.beam, built with erlc first