Distribution
How a Clause executable becomes a distributed Erlang node. Decided; the steps that implement it are in the project plan.
Today a program is a non-distributed node, nonode@nohost. Pids, ports and references of
other nodes exist as terms (terms), the node can be
named (this node), connection entries, node status messages, node monitors
and the controller data path follow OTP's process-controller interface without moving any signal
yet (connections). erl_epmd and net_adm:names/0,1 speak to an
epmd, and Clause's epmd server runs as clause-epmd (epmd; written beside programs
with -daemon, -names and -kill) or inside the program when a registration finds none
(--epmd bundled); --epmd external starts an epmd program, and --no-bundled-epmd leaves
the bundled epmd out. net_kernel:start/1,2 and stop/0 start and end distribution in a
running program (starting distribution), and --sname/--name before the
entry (node start options), with OTP's cookies (cookies).
Nodes connect through OTP's handshake with all its statuses, allow/1 and dynamic names (making
connections), over IPv4 or IPv6, messages move
between them (messages), processes monitor other nodes' processes and net_adm:ping/1
answers (process monitors), and they link and send exit signals to each other
(links and exit signals), and lead each other's processes (group
leaders) and spawn processes on each other (spawning), every
signal of a pair of processes in the order sent (signal order). Clause and OTP 29 nodes
connect, ping, monitor, message, link, lead and spawn on each other both ways on either kind of epmd
(testing). net_kernel changes tick times, reports and tunes connections, and
auth keeps cookies as OTP's (managing connections, cookies).
The other libraries answer as OTP does without distribution ({badrpc, nodedown}, not_alive). The
rest of this file is the contract for distribution. Each section becomes current behavior when its
step lands.
Scope
- Peers: OTP 29 nodes and other Clause nodes, in mixed clusters. Clause speaks OTP's protocol and keeps OTP's names, messages and inter-node protocols wherever another node observes them.
- Carriers:
- Node discovery: epmd (epmd): bundled inside the program, a standalone
clause-epmd, or OTP's; custom epmd modules through thekernelparameterepmd_module(theerl_epmdcallbacks); nodes without epmd (erl_epmd_node_listen_port). - Node kinds: hidden nodes, dynamic node names (
--name undefined), nodes that do not listen (--dist-listen false). - Left out unless a later step selects them:
- code loading over distribution (
-loader inet) anderl_boot_server; - remote shells (
-remsh); - tracing across nodes (
seq_tracetokens, trace ports); - C nodes (
erl_interface): they speak the same protocol but are not tested.
- code loading over distribution (
Protocol
- Handshake protocol version 6 (OTP 23 and later). The challenge digest is the MD5 of the cookie
and the challenge (
DFLAG_MANDATORY_25_DIGEST). - Packets carry a 2-byte length during the handshake and a 4-byte length after it, as with OTP's TCP carrier. Messages use the pass-through form (byte 112): the control message, then the payload, both in the external term format.
- Offered flags:
- OTP 26's mandatory set (
DFLAG_DIST_MANDATORY):EXTENDED_REFERENCES,FUN_TAGS,NEW_FUN_TAGS,EXTENDED_PIDS_PORTS,UTF8_ATOMS,NEW_FLOATS,MAP_TAG,EXPORT_PTR_TAG,BIT_BINARIES,BIG_CREATION,HANDSHAKE_23,V4_NC,UNLINK_ID. - Also
DIST_MONITOR,DIST_MONITOR_NAME,SPAWN,ALIAS,UNICODE_IO,SMALL_ATOM_TAGS,SEND_SENDER,BIG_SEQTRACE_LABELS,EXIT_PAYLOADandMANDATORY_25_DIGEST. PUBLISHEDunless the node is hidden, andNAME_MEwhen it asks for a dynamic name.
- OTP 26's mandatory set (
- Not offered for now, so peers never use them:
- The atom cache (
DIST_HDR_ATOM_CACHEand the older atom cache flags) and fragments (FRAGMENTS). Both are later optimizations: without fragments, a large message is sent in one piece and holds the connection until it is sent. ALTACT_SIG: Clause handles priority signals as ordinary ones (processes).NATIVE_RECORDS, untilterm_to_binary/1supports native records (native records).
- The atom cache (
- A peer that lacks a mandatory flag is refused, as OTP refuses it.
Runtime and libraries
The work is split as in OTP: the runtime moves signals, and Erlang code makes connections.
- Runtime (C++):
- this node's name and creation;
- pids, ports and references of other nodes as term kinds of their own;
- one entry per connected node: state, controller process, flags, creation and connection id;
- encoding of outgoing signals to other nodes, and decoding and delivery of incoming ones. Signals between two processes stay in order, as locally;
- the signals a lost connection causes (
noconnectionexits and'DOWN'messages,{nodedown, Node}); - OTP's process-controller interface:
erlang:setnode/2,3, thedist_ctrl_*functions,dist_get_stat/1,erts_internal:new_connection/1,abort_pending_connection/2,get_creation/0,dynamic_node_name/0,1andget_dflags/0; - the epmd server, shared by the bundled epmd and
clause-epmd(epmd).
- Libraries (Erlang, written from OTP's documentation, never from its sources):
net_kernel,authand theerl_epmdclient;dist_utiland the TCP carriersinet_tcp_distandinet6_tcp_dist(inlibrary/kernel). A carrier runs a controller process overgen_tcpthrough thedist_ctrl_*functions;- later
inet_tls_distandinet6_tls_distinlibrary/ssl.
- Registered processes:
net_sup,net_kernel,auth,erl_epmdandrexstart with distribution, as in OTP. Theglobalandpgservers keep their OTP names and protocols.
This node
Current (builtins/erlang/node.cpp, scheduler/signals.cpp), as in OTP:
- Naming:
erlang:setnode(Node, Creation)names the node.Nodeis a node name (letters, digits,-and_, an@, then a host without another@) other thannonode@nohost;Creationa non-negative small integer, cut to 32 bits and raised to at least 4. A live process must be registered asnet_kernel, and the node must have no name yet. Anything else isbadarg. - Readers:
node/0,1,nodes/1,2(this),ets:info/2(node), monitors and sends by{Name, Node},spawn/2,4and the encodings of this node's pids, ports and references use the name and creation.is_alive/0is true for a named node or a dynamic name (erts_internal:dynamic_node_name/1);erts_internal:get_creation/0is the creation, orundefinedwithout a name. A decoded identity naming this node is local only under its creation. - Monitoring nodes:
monitor_node/2,3raisenotalivefor a node other thannonode@nohostwhile the node has no name. Monitoring this node does nothing; another node's monitor is its connection entry's (node monitors). - End: when the
net_kernelprocess that named the node ends, its connections end (connections); after the last one the node isnonode@nohostwith creation 0 again, and can be named anew.setnode/2and this end sendnodeupandnodedownfor this node (node status messages).
Connections
Current (scheduler/connections.cpp, builtins/erlang/connections.cpp), as OTP's
process-controller interface. Signals move over it (messages); the controller
reads and writes through the controller data path. Node monitors hold entries
too (node monitors).
- Entries: one per other node, made by
erts_internal:new_connection(Node)(no name needed),monitor_node/2,3or asetnode/3attempt, and kept while the program runs. An entry is idle, pending or up; up holds the controller process, the flags and the creationsetnode/3gave. - Dist handles:
{ConnId, Ref}. Every handle of an entry holds the same reference, so handles of one entry compare equal. The connection id advances by one, within 24 bits, each time the entry turns pending. new_connection(Node): the entry's handle; an idle entry turns pending.badargunlessNodeis a node name other than this node's.abort_pending_connection(Node, DHandle):falsewhen the connection is up, elsetrue; a pending connection of the handle's id turns idle, and its node monitors getnodedown.badargunless the handle is one ofNode's entry.erlang:setnode(Node, DistCtrlr, {Flags, Creation}): brings the pending connection up and returns its handle. It sends{dist_ctrlr, DistCtrlr, Node, Caller}tonet_kernelandnodeupto the subscribers.badargunlessnet_kernelowns this node's name,Nodeis a node name other than this node's with a pending entry,DistCtrlris a live local process that controls no connection and is notnet_kernel,Flagsis a non-negative integer of at most 64 bits andCreationa non-negative small integer (cut to 32 bits). Flags without every mandatory one also write OTP's error report (... attempted to enable connection to node 'N' which does not support all mandatory capabilities.). A port controller isbadarg: Clause's carriers run controller processes.erts_internal:create_dist_channel/3answers{ok, DHandle}orbadarg.get_dflags/0:{erts_dflags, Default, Mandatory, Addable, Rejectable, StrictOrder}of the flags Clause offers (protocol).nodes/0,1,2:visibleandhiddenare up entries with and withoutDFLAG_PUBLISHED,connectedboth,knownevery entry and this node,thisthis node. The list is OTP's order: pending, idle, hidden, then visible entries, each oldest change first, then this node.nodes/2gives the connection id of an up entry (undefinedotherwise) and the node type (knownfor an entry that is not up).- Ends: a controller's end takes its connection down: the entry turns idle and the subscribers
get
nodedownwith the controller's exit reason ({shutdown, R}givesR,normalconnection_closed).net_kernel's end aborts the pending entries, the latest first, and kills the controllers, hidden before visible, the latest first (nodedownreasonkilled); after the last one this node's name ends with reasonnet_kernel_terminated, orno_networkwhen that wasnet_kernel's exit reason.
Node monitors
Current (scheduler/connections.cpp, builtins/erlang/node.cpp), as OTP's:
monitor_node(Node, true): the caller's monitor of the node once more, held by its entry (made idle when there is none). Each monitor gets one{nodedown, Node}when the connection goes down or its pending connection is aborted, before the node status messages.falseends one of the caller's monitors and makes no entry. A process's end ends its monitors.- Auto-connect: monitoring an idle entry makes it pending and sends
{auto_connect, Node, DHandle}tonet_kernel; without a process registered asnet_kernelthe connection is aborted at once, so the monitor gets itsnodedown. - Checks:
badargfor amonitor_node/3option other thanallow_passive_connector an improper option list, thennotalive(this node), thenbadargfor a flag that is not a boolean. allow_passive_connect: with a connection that is not up,monitor_node/3holds no monitor and asksnet_kernel:passive_cnct(Node)(a{passive_cnct, Node}call tonet_kernel,falsewithout one) inclause_dist:passive_monitor/3:truemonitors again,falsesends the caller{nodedown, Node}at once. An up connection takes the monitor as without the option.erlang:dmonitor_node(Node, Flag, Options): asksnet_kernelthe same way when the options holdallow_passive_connect; otherwise the caller gets{nodedown, Node}at once. It returnstruewhatever the other arguments.- Text of identities:
list_to_pid/1,list_to_port/1andlist_to_ref/1read another node's identity while its connection is up: the node part is the node name atom's number, as printed, and the identity gets the connection's creation. A pid's numbers are cut to 32 bits, a port's number to 64; a reference has one to five words below 2^32. Any other node part isbadarg.
Controller data path
Current (builtins/erlang/connections.cpp), as OTP's. A connection has an output
queue of encoded packets, which messages fill (messages), also while it
is pending, and an up connection an input handler, first the controller.
dist_ctrl_get_data(DHandle): the oldest queued packet as a list of binaries,{Size, Packet}withget_size, ornone.dist_ctrl_get_data_notification(DHandle):ok; the controller getsdist_dataat once when a packet is queued, else when the next one is.dist_ctrl_get_opt(DHandle, get_size)anddist_ctrl_set_opt(DHandle, get_size, Bool)(which returns the value before) choose whether sizes come. Only the controller calls these:notsupfor any other caller, thenbadargfor a handle that is not its connection's current one, another option or a value that is not a boolean.dist_ctrl_put_data(DHandle, Data): data from the other node. A list continues inerts_internal:dist_ctrl_put_data/2, which makes iodata one binary (badargotherwise). Checks:badargunless the handle is a pair of a small integer and the reference of an entry,notsupunless the caller is the input handler of an up connection,badargfor another connection's. Data of an earlier connection id is dropped; an empty binary is a tick. Other data is read as a packet (messages) and its message delivered, or the controller killed, before the put is counted as received;badargfollows for data that is not a binary.dist_ctrl_input_handler(DHandle, InputHandler): the controller makes any pid, of an ended process or of another node too, the processdist_ctrl_put_data/2takes data from; the controller then puts no more data.notsupunless the handle is a pair of a small integer and an entry's reference, elsebadargunless the caller controls that up connection, the id is its current one andInputHandleris a pid.dist_get_stat(DHandle):{ok, Received, Sent, PendingBytes}for any caller: the puts, the packets taken and the bytes queued, also while the connection is pending. OTP 29 counts 160 bytes more for each packet, 174 for one queued while the connection was pending, and Clause does the same (differences). Zeros while the connection is idle;badargfor a handle that is not its entry's current one. Each connection starts at zero.
Messages
Current (builtins/erlang/dist_messages.cpp), as OTP 29's, observed with a peer
speaking the protocol and with a test process as a connection's controller.
- Packets: 112 (pass-through), then the control message and the payload, each in the external
format with its 131. Clause sends
SEND_SENDER{22, From, To}to another node's pid,SEND{2, '', To}while its connection is not up yet,REG_SEND{6, From, '', Name}to{Name, Node}andALIAS_SEND{33, From, Alias}to another node's reference. - Sends:
!,send/2,3andsend_nosuspend/2,3queue the packet on the node's connection: an idle one auto-connects first (node monitors) and the packet waits untilsetnode/3; an aborted connection drops it. Withnoconnectan idle connection is left alone (noconnect,send_nosuspend/3false). A node without a name drops every such message, as does a pid or reference of an earlier incarnation of this node. Another node's port isbadarg. - The program's end (the entry returned,
halt/0,1) stops at once: packets still queued on a connection may be lost, as OTP 29'shaltloses them. Its flush runs no process and kills the distribution ports; ten messages sent just beforehalt()never arrived in OTP 29.1.1, while a pause orinit:stop/0, which takes the applications down first, let them all through. - Delivery:
dist_ctrl_put_data/2deliversSEND,REG_SEND,SEND_SENDER,ALIAS_SENDand their_TTforms (the trace token is ignored). The sender field is never checked; another node's receiver, a name no process has and a reference that is no alias get nothing. Process monitor signals (process monitors) and link signals (links and exit signals) are taken, group leaders set (group leaders), spawns served and answered (spawning), and the oldUNLINK(4) is an invalid distribution message, as since OTP 26. Bytes after the payload are ignored. - Unreadable packets kill the controller (
nodedownreasonkilled) after OTP's report,Nbeing the node's number in its printed pids:- fewer than three bytes: no report;
- a first byte other than 112, or a control message without 131: the error report
** Got message from incompatible erlang on channel N; - a control message that does not decode, or a payload that is missing, lacks its 131 or does not
decode: the warning report
'Me' got a corrupted external term from 'Node' on distribution channel N, the bytes from that term on (<<...,131,255>>, or>>for none) andATOM_CACHE_REF translations: none; - a control message of no known operation, arity or receiver, or a monitor signal naming what
OTP's
dist.crefuses (below): the error reportInvalid distribution message: T,Tprinted aserlang:display/1prints it.
net_kernellogsNet kernel got ~twfor a message it does not expect, as OTP's: a{dist_ctrlr, ...}of a process that set up no connection, such as a test process callingsetnode/3, among them.
Process monitors
Current (scheduler/remote_monitors.cpp), as OTP 29's dist.c, observed with a
test process as a connection's controller and between two nodes.
- Monitoring:
monitor/2,3of another node's pid or of{Name, Node}of another node queueMONITOR_P{19, Watcher, Pid | Name, Ref}on the node's connection, auto-connecting an idle one, also while it is pending. Options work as locally:{tag, Tag}replaces'DOWN',{alias, Mode}makes the reference an alias that another node's process can answer to (gen:call/3,4). - 'DOWN':
{'DOWN', Ref, process, Item, Reason}, Item the pid or{Name, Node}, comes from the node'sMONITOR_P_EXIT{21, Item, Watcher, Ref, Reason}orPAYLOAD_MONITOR_P_EXIT{28, Item, Watcher, Ref}with the reason as payload. It comes withnoconnectionwhen the connection goes down or cannot be made, at once on a node that is not alive ({Name, Node}of another node isbadargthere), and withnoprocat once for a pid of an earlier incarnation of this node. - Ending:
demonitor/1,2and the watcher's end sendDEMONITOR_P{20, Watcher, Pid | Name, Ref};flushalso finds a'DOWN'that came from another node. - Monitored here: another node's
MONITOR_Pwatches the process here it names by pid or registered name; one that is not there (no process, a port's name, a pid of an earlier incarnation) is answered withnoprocat once. The process's end sendsMONITOR_P_EXITnaming it as the monitor did, in the payload form when the node hasEXIT_PAYLOAD.DEMONITOR_Pends the monitor.process_info/2'smonitorsandmonitored_byshow the other node's pid or{Name, Node}. - Checks: a
MONITOR_PorDEMONITOR_Pwhose watcher is no pid of the connection's node, whose monitored process is no pid or atom here, or whose reference is no reference, and aMONITOR_P_EXITwhose monitored process is no pid or atom of another node or whose watcher is no pid here, are invalid distribution messages (messages). AMONITOR_P_EXITof a monitor that already ended, or for a pid of an earlier incarnation, does nothing. - Nodes without OTP's monitor flags (
DIST_MONITOR,DIST_MONITOR_NAME) get no monitor signals; their monitors watch only the connection. - Ping:
net_adm:ping/1calls{net_kernel, Node}with{is_auth, node()}, whichnet_kernelanswersyes:pong, elsedisconnect_node/1andpang(a wrong cookie fails the handshake with its log line first).auth:is_auth/1pings.
Links and exit signals
Current (scheduler/remote_links.cpp), as OTP 29's dist.c, observed with a test process as a
connection's controller and between two nodes.
- Linking:
link/1of another node's pid queuesLINK{1, From, To}on its connection, auto-connecting an idle one, also while it is pending; linking again sends nothing. A node that cannot be reached, also on a node that is not alive, exits the link at once:true, and the caller getsnoconnection({'EXIT', Pid, noconnection}when trapping exits). A pid or port of an earlier incarnation of this node raisesnoproc, or sends{'EXIT', Pid, noproc}when trapping; another node's port isbadargforlink/1,unlink/1andexit/2. - Unlinking:
unlink/1queuesUNLINK_ID{35, Id, From, To}and keeps the link, unseen byprocess_info/2, untilUNLINK_ID_ACK{36, Id, To, From}with that id; an exit that comes for it meanwhile is dropped. Linking again before the acknowledgement makes the link active again (LINKonce more), and the old acknowledgement leaves it. Ids count up from 1 for the whole program; OTP's count per process. - Exits: a linked process's end sends
EXIT{3, From, To, Reason}to each linked process of other nodes, also fornormal, orPAYLOAD_EXIT{24, From, To}with the reason as payload to a node withEXIT_PAYLOAD.exit/2,3of another node's pid sendsEXIT2{8, From, To, Reason}orPAYLOAD_EXIT2{26, From, To}; to another node's alias nothing, as the node lacksALTACT_SIG, andprioritychanges nothing. - Taken from another node: its
LINKlinks the process here, or is answered at once by an exitnoprocfor no process or a pid of an earlier incarnation; itsUNLINK_IDends a link not being unlinked here and is always acknowledged; its link exits (EXIT,PAYLOAD_EXITand their_TTforms) end the link and signal the process as a link does, dropped without a link; its exit signals (EXIT2,PAYLOAD_EXIT2,_TTforms) act asexit/2(killends a process that traps exits askilled), dropped for no process or an earlier incarnation. Exit signals reach a process running on another worker when its slice ends. - Connection loss: every link to the node's processes exits
noconnection, before the monitors''DOWN'messages and thenodedownof node monitors, as OTP orders them. - Checks: a link signal whose sender is no pid of the connection's node, whose receiver is no pid here (also a
pid of an earlier incarnation for
LINKandEXIT2), or anUNLINK_IDof id 0, is an invalid distribution message (messages). process_info/2'slinksshows the other node's pids.
Group leaders
Current (builtins/erlang/dist_messages.cpp), as OTP 29's.
- Another node's leader: a process's group leader may be another node's pid, kept in a buffer beside the
process's links:
group_leader/0andprocess_info/2return it, and the process's spawns inherit it.standard_iostill writes to this node's standard output. group_leader(Leader, Pid):Leadera pid of any node. For another node'sPidit queuesGROUP_LEADER{7, Leader, Pid}on that node's connection, auto-connecting an idle one, and returnstrue, also when the node cannot be reached or this node is not alive; a pid of an earlier incarnation of this node isbadarg.- Taken from another node:
GROUP_LEADERmakes its leader the group leader of the process here it names, once that process does not run on another worker; one naming another node's process does nothing. AGROUP_LEADERwhose leader or member is no pid is an invalid distribution message (messages).
Spawning
Current (scheduler/remote_spawns.cpp, builtins/erlang/dist_spawns.cpp,
library/kernel/clause_dist_spawn.erl), as OTP 29's dist.c and erlang.erl, observed with a test process as a
connection's controller and between two nodes.
- Requests:
spawn_request/2..5naming another node queueSPAWN_REQUEST{29, ReqId, From, GroupLeader, {Module, Function, Arity}, Options}with the argument list as payload on its connection, auto-connecting an idle one, and return the request id. The requester keeps{reply_tag, Tag},{reply, Mode},{monitor, MonitorOptions}and{link, LinkOptions}; the node gets the other options,monitorandlinkfirst for those tuple forms. An invalid kept option repliesbadoptat once; a node that cannot be reached, also from a node that is not alive, repliesnoconnection, and an up node without theSPAWNflagnotsup. Badarg for a node that is no atom, or an improper argument or option list. - Replies: the node's
SPAWN_REPLY{31, ReqId, To, Flags, Result}makes the link (flag 1) and the monitor (flag 2, the request id as its reference, with its tag and alias) active, then brings{Tag, ReqId, ok, Pid}or{Tag, ReqId, error, Reason}as{reply, Mode}asks. Until thendemonitor/1of the id does nothing. A lost connection fails pending requests withnoconnection. - Abandoned or orphaned:
spawn_request_abandon/1of a pending request returnstrue(falseonce replied or for another process's); its child, once spawned, gets the link's exitabandoned. A requester that ended has its child demonitored (DEMONITOR_P) and unlinked with the requester's exit reason. A reply to no pending request does nothing. - Blocking spawns:
spawn/2,4,spawn_link/2,4,spawn_monitor/2,4andspawn_opt/3,5naming another node continue inclause_dist_spawn:spawn/5, a request and its reply as OTP'sspawn_opt/5: the pid, or{Pid, Ref}withmonitor;badoptisbadarg, as are the reply options;noconnectionandnotsupgive OTP's useless pid, a process here that exits with that reason.spawn(Node, Fun)runserlang:apply(Fun, [])there. - Served here: another node's
SPAWN_REQUESTstartsModule:Function(Args...)with the request's group leader (group leaders), linked to and monitored by the requester as its options ask, and queues itsSPAWN_REPLYbefore the child can run, so the reply comes before anything the child sends. Options that are no proper list or not spawn options are answeredbadopt. ASPAWN_REQUESTwhose id is no reference of another node, whose requester is no pid of another node, whose group leader is no pid or whose{Module, Function, Arity}is malformed, and aSPAWN_REPLYwhose id or receiver is not of this node, whose flags are not below 2^27 or whose result is no pid of another node or atom, are invalid distribution messages (messages).
Signal order
Current, as OTP's. Every signal one process sends to another node (messages, links, unlinks,
monitors, exit signals, group leaders, spawn requests) is a packet on that node's one connection, queued in the
order sent, also while the connection is pending or being auto-connected; the packets of the other node are taken
in the order its controller puts them, and each is acted on before the next. So signals between a pair of
processes arrive in the order sent, as locally. While a connection is pending its packets use the forms of no
flags (SEND without the sender, EXIT2 without payload), as OTP's do.
- Aliases of
alias/0,1and ofmonitor/3's{alias, Mode}take messages from other nodes as they take local ones:replyends with the first,explicit_unaliaswithunalias/1,reply_demonitorends its monitor. - Send options:
erlang:send/3withnoconnectsends on an up or pending connection and returnsnoconnectfor a node not connected;nosuspendchanges nothing, as a connection is never busy here;send_nosuspend/2,3returntrue, orfalsefornoconnect.
Node status messages
Current, as OTP's: process_flag(monitor_nodes, Bool),
process_flag({monitor_nodes, Options}, Bool) and net_kernel:monitor_nodes/1,2 over them.
- Subscriptions: one per process and option set, counted:
trueadds one and returns the count before,falseends the subscription and returns its count. Messages come in the order the subscriptions were first made, as many copies as the count. A process's end ends its subscriptions. - Options:
[](or#{list => true}alone) gives{nodeup, Node}and{nodedown, Node}for visible nodes. A map gives{nodeup | nodedown, Node, Info}withInfoa map, or a list withlist => true, holdingconnection_id(undefinedfor this node),nodedown_reason(onnodedown) andnode_typewhen asked;node_type => visible | hidden | allchooses the nodes. The list holds them in that order. Anything else isbadarg. net_kernel:monitor_nodes/2: a list ofconnection_id,nodedown_reasonand{node_type, T}, or a map of them. Errors as OTP's:{bad_option_value, ...}first, then{unknown_options, ...},{option_value_mismatch, [New, Old]}for two node types and{invalid_options, Options}for anything but a list or a map;errorfor a flag that is not a boolean.
Carriers
User carriers are supported through OTP's interface, current
(library/kernel/dist_util.erl, include/dist.hrl, include/dist_util.hrl):
--proto-dist Protoselects the moduleProto_dist, as OTP's-proto_distdoes.- Clause's
dist_utilexports OTP's functions:handshake_we_started/1,handshake_other_started/1,strict_order_flags/0,rejectable_flags/0,start_timer/1,setup_timer/2,reset_timer/1,cancel_timer/1,is_node_name/1,split_node/1,is_allowed/2,shutdown/3,4andnet_ticker_spawn_options/0. kernel/include/dist_util.hrlandkernel/include/dist.hrlprovide the#hs_data{}record and theDFLAG_*macros with OTP's names and fields;kernel/include/net_address.hrlthe#net_address{}record a carrier'slisten/2returns. The header is written by Clause, and its fields follow OTP's because carriers set them.- Clause's own carriers are written over the same interface, so a third-party carrier and a built-in one run the same handshake code.
Making connections
Current (library/kernel/dist_util.erl, inet_tcp_dist.erl, net_kernel.erl), as OTP's; the
statuses other than ok, allow/1 and dynamic names.
- Handshake: protocol 6: the initiator's name (
N, flags, creation, name), the acceptor's status, its challenge, the initiator's reply and the acceptor's ack. A digest is the MD5 of the cookie and the challenge as decimal text; both sides take the cookieauth:get_cookie(Node)gives for the other node. - Statuses: the acceptor answers
okfor a node it holds no connection of;alivefor a node whose connection is up. The initiator answerstrue, sincenet_kernelonly sets up a connection it does not hold; the acceptor then ends the old connection with nodedown reasonwait_pendingand goes on. Any other answer closes the new one;- for a node it is itself setting up a connection to (simultaneous connects):
ok_simultaneouswhen the other node's name is greater, ending its own attempt and handing itsconnect_node/1callers to the accepted one, elsenok. An initiator toldnokwaits for the other node's attempt, which takes its callers over; the setup timer ends the wait; nokfor a node another accepted handshake is pending for, and for this node's own name, logging** Cannot get connection id for node N(after an empty line), as OTP does;not_allowedfor a node outside the allow list (below);named:with a name and a creation for a node asking for a dynamic name (below).
allow/1: adds atoms to the allow list in order, each once, and answersok;errorfor a non-list or a list with a non-atom, which adds nothing;ignoredwithout distribution.allowed/0gives{ok, Nodes}. While the list is not empty, the acceptor refuses a nodedist_util:is_allowed/2does not find in it (** Connection attempt from disallowed node N **), checking a dynamic name request by its host, andconnect_node/1refuses one before connecting (** Connection attempt with disallowed node N **).- Dynamic names: a node started as
--sname undefinedor--name undefinedis hidden, listens nowhere and makes no auto-connects. Its first handshake offersNAME_MEand sends only its host; the acceptor names itName@Host, withNamethe base-36 digits of a 64-bit number derived from a random seed of the acceptor, the host and an attempt counter: a node asking again from that host gets the same name, unless one of that name is connected, which moves on to the next attempt. The creation is random the first time and one more each time the acceptor gives that name again.net_kernelnames the node witherlang:setnode/2. When the connection that gave the name ends,net_kernelrestarts itsnet_supchild ofkernel_sup(net_sup_dynamicafternet_kernel:start/1,2), as OTP does: the other connections end withnet_kernel_terminated, the node is unnamed and the nextconnect_node/1names it again. - Flags: the offer is
get_dflags/0's default with the carrier'sadd_flags(of the addable ones) and without itsreject_flags(of the rejectable ones), andPUBLISHEDunless this node is hidden (net_kernel:publish_on_node/1) or the request ishidden. A peer'sMANDATORY_25_DIGESTstands for the OTP 25 set; a peer lacking a mandatory flag or one ofrequire_flagsis refused with an error report. The connection takes the flags both nodes offer, and is visible when both published. - Wrong cookie: the acceptor logs
** Connection attempt from node N rejected. Invalid challenge reply. **(error_logger:error_msg/2) and closes; the initiator'sconnect_node/1answersfalsewithout a log. - Up: the supervisor calls
f_setopts_pre_nodeup,erlang:setnode/3with the controllerf_getllgives, tellsnet_kernel, then callsf_setopts_post_nodeupandf_handshake_complete(arity 3, or 4 with#{creation, flags}), and supervises the connection. net_kernel:connect_node/1andhidden_connect_node/1answertruefor this node and an up connection. Otherwise the first carrier whoseselect/1takes the node sets the connection up, and the callers wait for its handshake:true, orfalseonce its process ended. A node no carrier takes and a name that is no node name answerfalse. A node monitor's auto-connect sets up the same way, and is aborted when it cannot, on a dynamic node and under thekernelparameterdist_auto_connectnever.- Accepted connections: the documented
{accept, Acceptor, Controller, Family, Protocol}from a carrier's acceptor startsaccept_connection/5of the listening carrier of that family and protocol. disconnect/1ends the connection's supervisor and controller with reasondisconnectand answerstrueonce both ended, so the connection is down;falsewithout a connection, and for this node.erlang:disconnect_node/1isnet_kernel:disconnect/1on an alive node andignoredon another, as OTP's.- Between
dist_utilandnet_kernel(Clause's own messages):{Pid, {accept_pending, Node | {name_me, Host}, Type}}answered by{Kernel, {accept_pending, Status}}(ok,ok_simultaneous,nok,aliveor{named, Node, Creation}); afteraliveandtrue,{Pid, {replace, Node, Type}}answered by{Kernel, {replace, ok | nok}}; on the initiator,{Pid, {set_name, Node, Creation}}answered by{Kernel, {set_name, ok | error}};{Pid, {nodeup, Node, Address, Type}}answered by{Kernel, {inserted, Intensity}}, and the ticks{Kernel, tick}. Carriers receiveallow/1's list asAllowed, and a node waiting for its dynamic name passes itself tosetup/5asundefined@Host.
- Ticks: a linked ticker spawned at
maxpriority sendsnet_kerneltickeverynet_ticktime× 1000 /net_tickintensitymilliseconds (the tick time rounded up to a multiple of the intensity, thekernelparameternet_tickintensitykept within 4 to 1000), andnet_kernelticks each up connection's supervisor. The supervisor readsmf_getstat:net_tickintensityticks without input end the connection with** Node N not responding **/** Removing (timedout) connection **and the reasonnet_tick_timeout; a tick without output since the last one callsmf_tick, as does a hidden connection's tick without input, and the tick sent counts as output. - Setup timer:
dist_util:start_timer/1links a process that ends the handshake withsetup_timer_timeoutunless a received handshake message resets it withinnet_kernel:connecttime/0(7 seconds, or thekernelparameternet_setuptimein seconds, at most 120); it is cancelled when the connection comes up. inet_tcp_dist:accept/1links an acceptor atmaxpriority;accept_connection/5andsetup/5spawn the supervisor withdist_util:net_ticker_spawn_options/0. Setup asks theepmd_module'sport_please/2and connects withinet_dist_connect_optionsunder the carrier's own family,binary,{active, false}and{packet, 2}, overnodelay({nodelay, false}whendist_nodelayisfalse); an address family ininet_dist_connect_optionsfails the connect, as in OTP. Host names the naming refuses log OTP's** System running to use fully qualified hostnames **lines. The controller is a process owning the socket: during the handshake it answers its supervisor's requests and switches to{packet, 4}before the connection is up; then it reads with{active, once}and puts each packet withdist_ctrl_put_data/2(an empty one is a tick), sends the queued packets ondist_dataand sends ticks as empty packets. A closed socket, a failed send or refused data ends it withconnection_closed, the other node'snodedownreason.mf_setoptsandmf_getoptsask the controller, which answers while the connection is up:setopts/2refusesactive,deliverandpacket({error, {badopts, Options}}) as for listen sockets, andgetopts/2reports{active, true}as OTP's port does while the controller reads with{active, once}.
Managing connections
Current (library/kernel/net_kernel.erl, dist_util.erl), as OTP's net_kernel:
- Tick time changes:
set_net_ticktime(T, Period)(Period60 seconds by default) answersunchangedfor the current tick time,change_initiated, or{ongoing_change_to, T}while a change runs;get_net_ticktime/0answersTor{ongoing_change_to, T},ignoredwithout distribution. The new tick time keeps the intensity and is rounded up to a multiple of it. An aux ticker sendsaux_tick(the supervisors tick unless a visible connection has output pending) at once and every shorter interval of the two untilPeriodis covered; a longer tick time takes effect at the start, a shorter one at the end of the period. A ticker that ends starts again. - Connection information:
node_info(Node)gives{ok, [{owner, Pid}, {state, up | pending}, {address, #net_address{}}, {type, normal | hidden}, {in, N}, {out, N}]}, the packet counts of an up connection read by its supervisor ({Kernel, get_status}),0for a pending one;node_info(Node, Key)one of them or{error, invalid_key};{error, bad_node}without a connection,badargwithout distribution (OTP'ssys_disttable is missing then).nodes_info/0lists every connection;i/0prints them as OTP's table with totals,i/1one connection. - Socket options:
getopts(Node, Options)andsetopts(Node, Options)go through the up connection's supervisor to the carrier'smf_getoptsandmf_setopts({error, enotsup}without them),{error, noconnection}without an up connection or when it ends first.setopts(new, Options)calls each listening carrier'ssetopts/2on its listen socket (its error is the answer;enotsupwhen the carrier has none), then puts each option in front ofinet_dist_listen_optionsandinet_dist_connect_optionsin place of options of its key (an option that is no pair crashesnet_kernel, as OTP's), setsdist_nodelayfor{nodelay, B}, and answers once each pending connection's supervisor applied them when it came up. A node name or option list of another type isfunction_clause, also without distribution, which answersignored. dist_auto_connect once: an up connection's end bars its node from auto-connects (node monitors abort at once);connect_node/1and an accepted connection lift the bar.- Passive connects:
{passive_cnct, Node}sets up a connection as an auto-connect does and answers once it is up,trueat once for an up connection or this node,falsewhen the setup fails or underdist_auto_connect never; for a barred nodepassive_connect_monitor/2waits up toconnecttime()for the node to connect. - Disconnects:
disconnect/1andasync_disconnect/1(a cast, not waiting) end up connections only; a pending connection answersfalse. - Other requests:
verbose(Level)answers the level before and logs OTP's info reports of connects, disconnects and exits at that level;kernel_apply/3answersnot_implemented. The old spawn protocol's{spawn, M, F, A, Leader},{spawn_link, ...}and{spawn_opt, M, F, A, Options, Link, Leader}calls spawnspawn_func/6, which answers the caller with its pid, sets the leader and applies the function (do_spawn/3).{From, registered_send, Name, Message}is passed to a registered name,{From, badcookie, ...}logs** Got OLD cookie from Nand disconnects the node. Other calls are never answered, as OTP's.
epmd
Clause can provide epmd in two forms, built from the same C++ epmd code in the runtime. Both serve
OTP's epmd protocol: ALIVE2 (with the ALIVE2_X reply), PORT_PLEASE2, NAMES, DUMP,
KILL and STOP. A registration ends when its node's connection to epmd closes, and STOP and
KILL are accepted only with -relaxed_command_check, as in OTP's epmd. ERL_EPMD_PORT (default
4369) and ERL_EPMD_ADDRESS work as for OTP's epmd.
- Bundled epmd (built in by default): the program's executable carries the epmd. It runs inside
the program, on a runtime thread with no Erlang processes, and only while the program runs.
clau --no-bundled-epmd, orbundled_epmd = falsein[targets.options], leaves it out.- It serves every node on the host, not only its own program. When the program exits, the other
nodes lose their registrations, and new connections to them fail until they register again.
Clusters that outlive one program use
externalwithclause-epmd.
clause-epmdexecutable (optional):clau --epmd-executable, orepmd_executable = truein[targets.options], also writesclause-epmd(clause-epmd.exeon Windows) beside the output.- It is a small standalone program with no Erlang code.
- It takes a subset of OTP's
epmdflags:-daemon,-port P,-address LIST,-relaxed_command_check,-namesand-kill. -daemondetaches it. On Windows that means a new process group, no console and no inherited handles; on POSIX,setsidafter a double fork, with standard I/O on/dev/null.-namesand-killprint OTP'sepmdmessages and exit with its statuses.
The runtime option --epmd MODE chooses what happens when distribution starts, at startup or by
net_kernel:start/1,2:
| Mode | When an epmd already listens | When none listens |
|---|---|---|
bundled (default) | Register with it | Run the bundled epmd inside the program, then register |
external | Register with it | Start clause-epmd -daemon (the program's directory first, then PATH) or OTP's epmd -daemon from PATH, then register; distribution fails to start if neither is found |
none | Register with it | Distribution fails to start |
- A program built without the bundled epmd treats
bundledasnoneand prints a warning once. - When two programs start an epmd at the same time, the one that cannot bind gives way. The nodes retry their connection to epmd for a short time.
- Nodes without epmd stay possible through the
kernelparametersepmd_module(a customerl_epmdcallback module) anderl_epmd_node_listen_port.
Current (runtime/src/epmd/, library/kernel/erl_epmd.erl):
- Server:
registrykeeps the names andrequestsanswers each request as OTP's epmd does;serverlistens (every address of both families, or-addressplus loopback) and moves the bytes. Observed from OTP 29's epmd and kept:- a registration whose highest version is 6 or more gets
ALIVE2_X_RESP, othersALIVE2_RESPwith a creation of 1 to 3; a name registered again gets its old creation plus one; - a refused registration (name taken) is answered with result 1 and creation 99, and its connection stays open; any byte on a registered connection closes it;
NAMESlists the newest name first;DUMPends each line with a NUL byte and also lists remembered names (old/unused name);KILLanswersNOwhile names are registered,STOPis ignored, and other hosts may only askPORT_PLEASE2andNAMES, unless-relaxed_command_check; with it,STOPof an unknown name answersNOEXISTSTOPPED, as OTP's;- malformed requests, unknown codes and bytes past a packet close the connection without reply; a connection without a registration closes after 60 seconds.
- a registration whose highest version is 6 or more gets
clause-epmd(targetclause_epmd) runs it in the foreground with-port,-addressand-relaxed_command_check; the test build makes it for the multi-node runner. It also has the rest below.erl_epmd: registers withALIVE2_REQ(versions 6 to 5, type 77) and registers again a second after the connection breaks.names/0,1list the oldest name first, as OTP's client does.
Current (runtime/src/epmd/bundled.cpp, runtime/src/builtins/epmd.cpp):
- Bundled epmd: every executable carries it (the
clause_runtime_epmdarchive). It runs on a thread of its own, opened by the first registration that needs it, and it ends with the program; aKILLstops it, and the next registration that finds no epmd opens it again. It listens asclause-epmdwould under the program'sERL_EPMD_PORT,ERL_EPMD_ADDRESSandERL_EPMD_RELAXED_COMMAND_CHECK. --epmd bundled|none: whenerl_epmd's registration (also the re-registration after a lost epmd) finds no epmd (econnrefused), it asksclause_dist:start_epmd/0. Underbundledthe bundled epmd opens; when another epmd holds the port, it gives way. The registration then connects again for up to a second. Undernonethe registration fails witheconnrefused, as OTP's.names/0,1andport_please/2,3never start an epmd, as in OTP.
Current (runtime/src/epmd/main.cpp, client.cpp, daemon_*.cpp,
compiler/src/linking/link.cpp):
- Beside programs:
clau --epmd-executableandepmd_executable = truelinkclause-epmdfor the program's target from the archiveclause_epmd_programbeside the runtime archive, and publish it beside the executable with it. -namesand-kill: as OTP's, only as the last argument (after-port). They ask the epmd on the loopback (IPv6, then IPv4) atERL_EPMD_PORTor-port.-namesprintsepmd: up and running on port P with data:and the epmd's lines;-killprintsKilled, orKilling not allowed - living nodes in database.with status 1. Without an epmd both printepmd: Cannot connect to local epmdon standard error with status 1.-daemon: POSIX forks twice aroundsetsid, ignoresSIGHUP, moves to/and puts standard I/O on/dev/null; the parents exit with status 0 at once. Windows starts the program again with its arguments and-detached(DETACHED_PROCESS,CREATE_NEW_PROCESS_GROUP, no inherited handles), and that process serves with standard I/O onNUL. A daemon that cannot listen exits silently, as OTP's.
Current (runtime/src/epmd/extension.cpp, launch_*.cpp,
library/kernel/clause_dist.erl):
- Without the bundled epmd: the bundled epmd and the server are the
clause_runtime_epmdarchive beside the runtime archive, behind the startup descriptor's extension entryCLAUSE_epmd_extension_v1(as the crypto archive's). Every program lists it and links the archive unlessclau --no-bundled-epmdorbundled_epmd = false; such a program answers a registration that finds no epmd as--epmd nonedoes, after one warning on standard error (clau: warning: this program was built without the bundled epmd ...). CMake consumers ofClause::generated_programlink the archive too. --epmd external:erl_epmdlooks forclause-epmdin the program's directory (init:script_name/0), then forclause-epmdand OTP'sepmdonPATH(os:find_executable/1,2), and starts it with-daemon(POSIX: standard I/O on/dev/null, waiting for its first process; Windows: detached, a new process group, not waited for). The registration then connects again for up to a second; without a program, or when it does not start, it fails witheconnrefused. The epmd outlives the program, as OTP's does.
Starting distribution
Current (library/kernel/net_kernel.erl, erl_distribution.erl, inet_tcp_dist.erl), as
OTP's net_kernel:start/1,2 and stop/0. The started node connects to others as
making connections says.
- Processes:
net_kernel:start/1,2starts thenet_supsupervisor (erl_distribution, one-for-all, no restarts) with theerl_epmdserver (thekernelparameterepmd_module, when that module hasstart_link/0), theauthserver (cookies) and then thenet_kernelserver, askernel_sup'snet_sup_dynamicchild (kernel_sup), and returns{ok, NetSup};{error, {already_started, NetSup}}while it runs. - Options:
start/2takesname_domain(longnamesby default),net_ticktime(thekernelparameter, 60 by default; a value below 1 is 1),net_tickintensity(4),dist_listenandhidden; an invalid one raises{invalid_option, Key, Value}, a name that is no atominvalid_name, options that are no mapinvalid_options.start/1takes OTP's documented lists ([Name],[Name, NameDomain],[Name, NameDomain, TickTime],net_ticktimebeingTickTime× 4 in seconds, rounded up). - Names: a name part of letters, digits,
-and_. A host part is taken as given; withshortnamesit may hold no dot (Can't set short node name!). Without one (aora@) the node takes this host's name, its first label forshortnames, its fully qualified name forlongnames(Can't set long node name!when it has no dot). Other names logInvalid node name!. Each log line is anerror_logger:info_msg/1,2report, followed by the failed server's crash report andnet_sup's start error, as in OTP. - Listening: the first
--proto-distcarrier (inet_tcpby default) checks the name withis_node_name/1(Protocol 'inet_tcp': invalid node name: ...), thenlisten/2registers the port with epmd under the--epmdmode (epmd); a taken name logs... the name N seems to be in use by another Erlang node, other errors... register/listen error: Reason.erlang:setnode/2then names the node with the creation epmd gave.dist_listen => falselistens nowhere and registers no name, takes a random creation and is hidden; the nameundefinednames no node yet (erts_internal:dynamic_node_name/1). inet_tcp_dist:listen/1,2listens withgen_tcp(IPv4,{packet, 2}) on the first free port frominet_dist_listen_mintoinet_dist_listen_max(any port without them), oninet_dist_use_interface, withinet_dist_listen_optionsunder the carrier's own options (an address family there isbadarg) and over a backlog of 128;address/0,select/1(both parts and an IPv4 address for the host),is_node_name/1(an atom with one@) andclose/1as OTP's. Also (library/kernel/inet6_tcp_dist.erl):--proto-dist inet6_tcpselectsinet6_tcp_dist, the same carrier over IPv6: it listens on IPv6, registers with epmd asinet6_tcp, selects nodes whose host has an IPv6 address and givesinet6addresses. It runs overinet_tcp_dist'sgen_*functions, which take the driverinet_tcporinet6_tcp, andfam_address/1andfam_select/2, which take the family. OTP's undocumentedfam_listen/4,fam_setup/4,getstat/1andtick/2are stubs.setopts/2andgetopts/2set and read options of future connections on a listen socket throughinet;setopts/2refusesactive,deliverandpacketoptions with{error, {badopts, Refused}}.net_kernel:setopts/2andgetopts/2call them.merge_options(Options, Forced)givesForced, then the options ofOptionswhose keyForcedhas not. A key is a pair's first element,modeforbinaryandlist, andfamilyforinet,inet6andlocal; other options have none and always stay.merge_options/3appends the options of its third argument whose keyOptionshas not.
- Failures: a start that fails returns OTP's
{error, {{shutdown, {failed_to_start_child, net_kernel, {'EXIT', nodistribution}}}, Child}},kernel_sup's answer with thenet_sup_dynamicchild record. - Requests:
nodename/0,longnames/0,get_net_ticktime/0andget_state/0(started => dynamic,name,name_type,name_domain) ask the server; without one they answerignored(#{started => no}). stop/0: terminates and deletesnet_sup_dynamic, whosenet_kerneltakes the name with it and whoseerl_epmdcloses the registration;{error, not_found}without distribution,{error, not_allowed}for a node the node start options named (node start options).
kernel_sup
Current (library/kernel/kernel.erl, clause_kernel.erl). OTP's node starts kernel_sup at boot;
Clause starts it at the first use of a kernel service: rex (rpc), global_name_server (global),
global_group, or distribution (net_kernel:start/1,2, --sname, --name). A program that uses none has none.
- Children:
kernel:init/1's, in OTP's start order:rex,global_name_server,net_sup(distribution of the node start options; otherwiseerl_distribution:start_link/0answersignoreand the child stays without a process),global_group(a server without groups) andkernel_safe_sup, an empty supervisor other libraries add children to. One-for-all without restarts, as OTP's. OTP's other children (code_server,file_server_2,inet_db,logger_sup,user, ...) are runtime services in Clause (differences). net_kernel:start/1,2adds thenet_sup_dynamicchild, andstop/0terminates and deletes it.- Opting out (user choice):
exclude_kernel_services = [...]in[targets.options]orclau --exclude-kernel-service IDleavesrex,global_name_server,global_grouporkernel_safe_supout ofkernel_sup(net_supcannot be excluded).claugeneratesclause_kernel_services, whoseservices/0tellskernel:init/1the included services and links their modules; an excluded one is not linked forkernel_sup. This mostly saves processes, not program size:gen_serveralready linksglobalandrpc, so an excluded service usually stays linked and is only not started. An excluded server the program uses anyway (global,rpc) starts unlinked at its first use, as beforekernel_sup. - End: a keeper process links to
kernel_sup; whenkernel_supends, for example afternet_kernelwas killed, the program halts with OTP's sloganKernel pid terminated (application_controller) ("{application_terminated,kernel,shutdown}"), exit 1. A first use racing another waits until the children started.
Cookies
Current (library/kernel/auth.erl, runtime/src/builtins/erlang/node.cpp), as OTP's auth:
- Without distribution:
erlang:get_cookie/0,1andauth:get_cookie/0,1answernocookie;erlang:set_cookie/1,2andauth:set_cookie/1,2raisedistribution_not_started, anderlang:set_cookie(node(), C)raisesbadarg. A cookie or node that is no atom isbadarg. - This node's cookie: the
authserver undernet_supstarts with--setcookie COOKIE, else--nocookie(nocookie), else.erlang.cookiein the home directory (init:get_argument(home):HOME, on WindowsHOMEDRIVEandHOMEPATH, else the user profile), else infilename:basedir(user_config, "erlang"). The cookie is the file's printable characters (space to~) up to the first other one, blanks included; only line ends and blanks may follow (Bad characters in cookieotherwise,Too short cookie stringfor none), as OTP checks it. A missing file is created in the home directory (the configuration directory without a home) with 20 random lettersAtoZand made readable by its owner alone (mode 400; read-only on Windows). On Unix, a file others may read stopsauthwithCookie file F must be accessible by owner only, and a start without a home whose configuration directory cannot be read withNo home for cookie file, so distribution fails to start.--setcookie NODE COOKIEforms, where the runtime passes them, set other nodes' cookies as OTP's (a node given twice keeps none). - Alive: the
erlangfunctions continue inauth's.get_cookie/0is this node's cookie;get_cookie(Node)the cookie set forNode, else this node's.set_cookie(C)andset_cookie(node(), C)set this node's cookie,set_cookie(Node, C)the cookie ofNode; each answerstrue.auth:cookie/0,1,node_cookie/1,2andsync_cookie/0(true; a cookie given for this node's name becomes its cookie) as OTP's;is_auth/1andnode_cookie/1,2ping the node (process monitors). The server takes OTP's requests (get_our_cookie,{get_cookie, Node},{set_our_cookie, C},{set_cookie, Node, C},sync_cookie,echoansweredhello),auth:set_cookie/1,2andget_cookie/1take any term as OTP's (theerlangfunctions check atoms), andprint(Node, Format, Arguments)casts{print, Format, Arguments}toNode'sauth, which logs it witherror_logger:error_msg/2. OTP's answers tobadcookiemessages are left out: no runtime sends them. global: checks thekernelparametersconnect_allandprevent_overlapping_partitionsat its start as OTP's ({invalid_parameter_value, Name, Value}for a value that is no boolean); an unsetconnect_allbecomesfalseonly under the emulator flag-connect_all false.- End: the cookies end with
net_sup; a later start reads the file again.
Node start options
Runtime options (executables) start distribution before the entry runs. They can be given:
- on the command line;
- in
CLAUSE_FLAGS; - in the manifest, as
runtime_options = [...]in[targets.options]. These are built into the executable as its defaults.
The command line wins over CLAUSE_FLAGS, which wins over the manifest.
| Option | OTP counterpart | Effect |
|---|---|---|
--sname NAME | -sname | Short node name NAME@Host; NAME@HOST is taken as given |
--name NAME | -name | Long node name with the fully qualified host; undefined asks the first peer for a dynamic name |
--setcookie COOKIE | -setcookie | This node's cookie; otherwise ~/.erlang.cookie is read, or created as OTP creates it |
--nocookie | -nocookie | No cookie until erlang:set_cookie/1,2 |
--hidden | -hidden | Hidden node |
--proto-dist PROTO | -proto_dist | Carrier module PROTO_dist; default inet_tcp |
--epmd bundled|external|none | -start_epmd | What to do when no epmd runs (epmd); default bundled |
--dist-listen true|false | -dist_listen | Accept incoming connections; false implies --hidden |
--env APP KEY VALUE | -App Key Value | Sets APP's environment parameter KEY to the term VALUE before applications start, over the .app resources (for example --env kernel net_ticktime 20). It works for every application, not only for distribution |
--envvalues are parsed as OTP parses-App Key Value: whenapplication_controllerloadsAPP(kernel and stdlib when it starts), over the.appresource, merged in OTP's order. A value that is no term fails the load with{bad_environment_value, Text}after OTP's log line, and stops the program for kernel or stdlib.init:get_argument/1andinit:get_arguments/0report these options under OTP's flag names:sname,name,setcookie,hidden,proto_dist,dist_listen, andstart_epmd("false"for--epmd none,"true"otherwise);--env kernel net_ticktime 20askernelwith[["net_ticktime", "20"]].
- The
kernelparameters distribution reads, as in OTP:net_ticktime,net_tickintensity,net_setuptime;dist_auto_connect,connect_all;inet_dist_listen_min,inet_dist_listen_max,inet_dist_use_interface,inet_dist_listen_options,inet_dist_connect_options;epmd_module,erl_epmd_node_listen_port;prevent_overlapping_partitions,global_groups;distributed,sync_nodes_mandatory,sync_nodes_optional,sync_nodes_timeout.
net_kernel:start/1,2starts distribution in a running program, as in OTP, under the same--epmdmode.
Current (library/kernel/clause_dist_start.erl, erl_distribution.erl,
runtime/src/startup/startup.cpp):
- Linking (user choice, 2026-10-31): the distribution modules join a program that calls
net_kernel, or one built withclau --distribution, the manifest optiondistribution = true, or manifestruntime_optionsholding--snameor--name. Other programs stay small; given--snameor--name, they stop before the entry withclau: runtime failure: --sname and --name need the distribution modules (link the program with --distribution), exit 70 (differences). - Start: with
--snameor--name, the runtime runsclause_dist_start:boot(Module, Function, Arguments)as the main process instead of the entry. It startskernel_sup, whosenet_supchilderl_distribution:start_link/0starts as OTP's kernel does at boot: the name of--sname(shortnames) or--name(longnames),--hidden,--dist-listen,--proto-distand thekernelparameters as fornet_kernel:start/2;undefinedasks the first peer for a name. It then calls the entry in its place, so the entry runs in the main process with its usual reports. - Such a node:
get_state/0saysstarted => static,stop/0answers{error, not_allowed}andstart/1,2{error, {already_started, NetSup}}.net_supiskernel_sup'snet_supchild. - Failure: a start that fails logs OTP's lines and reports, then halts with OTP's slogan
Kernel pid terminated (application_controller) ("{application_start_failure,kernel,{{shutdown, {failed_to_start_child,net_sup,...}},{kernel,start,[normal,[]]}}}"), exit 1, before the entry runs.
Testing
- Multi-node cases:
- A case's
golden.jsonlists its nodes. Each node has a name, its arguments, its kind (clauseorotp), its expected stdout and exit status, and a stderr pattern. All nodes run the same program, with roles chosen by argument. - The runner gives each run its own epmd port (
ERL_EPMD_PORT), so cases run in parallel and never touch a user's epmd. It starts a foregroundclause-epmd -port N(built once by the test build; OTP'sepmdfor a run of"epmd": "otp") and starts the nodes at once. It collects each node's output and stops everything at the end or on a timeout. Clause nodes run with--epmd none, and with a fixed cookie. - Single-node runs get an
ERL_EPMD_PORTwhere nothing listens, so they never ask a user's epmd. - A node's own
env(Clause) andotp_flags(OTP, asERL_FLAGS) give it node start options. - Cases of the bundled and external modes start no runner epmd and check which epmd the nodes
find. A run of
"epmd": "bundled"also checks that no epmd listens on its port after the nodes ended. - Test programs and epmds listen on loopback only, so the host's firewall never asks about them: every epmd and every program gets
ERL_EPMD_ADDRESS=127.0.0.1,::1, and every node of a multi-node run, Clause's and OTP's alike,inet_dist_use_interfaceof 127.0.0.1 (::1under--proto-dist inet6_tcp;nodes.loopback_flags). Nodes that connect are named@localhost, and fixtures askerl_epmd:names(localhost): a lookup by this host's name reaches its network address, where no epmd listens ({error, address}, as in OTP). - Host names in output are masked with
stdout_masks. - A program linked with
--epmd-executablefinds theclause-epmdbeside it inCLAUSE_TEST_EPMD; under the oracle that names OTP'sepmd. The oracle gives single-node runs anERL_EPMD_PORTwhere nothing listens, as the runner does.
- A case's
- Goldens come from OTP nodes on OTP's
epmdon the same kind of epmd port, generated once (tests/compiler/executables/nodes.py, used byrun.pyandregenerate.py). - Interop cases mix
clauseandotpnodes, or use OTP'sepmd. Their golden says"host_otp": true: they carry the CTest labelhost_otpand skip when noescriptis found. Clause-only cases need no OTP:epmd_namesruns onclause-epmdand on bundled epmds (two nodes racing to start one), andepmd_interopruns the same program on OTP'sepmdand with an OTP node. - OTP peers:
dist_ping,dist_connectanddist_signalsrepeat their first run with nodeaan OTP node onclause-epmdand with nodebone on OTP'sepmd; each node's stdout is the oracle's. An OTP node's stderr pattern is in OTP's own log format (aTime level: messageline per report), and a node that waits for another's node status messages waits for its epmd marker name first, since an OTP node boots slower than a Clause program.
Clause