ADR 0020: Connection Security — mTLS, Proxies, and Network Overlays

Status

Implemented (2026-02-18)

Update (2026-02-17): Phase 0 (WebSocket transport + cookie auth) implemented in BT-683. ADR accepted based on validated implementation. Update (2026-02-18): Phases 0–2 complete (BT-683, BT-691, BT-692). Phase 3 (SPIFFE/SPIRE) remains future. Update (2026-03-15): TLS distribution support removed (PR #1401). beamtalk workspace attach and beamtalk transcript moved under beamtalk workspace subcommand. References below reflect the original design; current commands are beamtalk workspace attach and beamtalk workspace transcript.

Context

Beamtalk has several TCP communication channels that need security consideration:

ChannelTransportCurrent SecurityUsers
REPL ↔ WorkspaceWebSocket ws://127.0.0.1:{port} (OS-assigned ephemeral)Loopback + cookie handshake (BT-683)beamtalk repl
CLI ↔ CompilerOTP Port (stdin/stdout)Process isolation (ADR 0022)beamtalk build, LSP
Distributed ErlangErlang distribution protocolCookie file (chmod 600)Workspace-to-workspace
Web terminalNot yet implementedBrowser-based REPL
Remote attachNot yet implementedbeamtalk workspace attach prod@host

Threat Model

Local development (single machine, single user):

Local development (shared machine, CI runner, Docker --net=host):

Web terminal (browser on same or different machine):

Remote attach (production debugging, multi-machine clusters):

Why WebSocket (Not Unix Socket or Raw TCP) for the REPL

Update (2026-02-17): The REPL now uses WebSocket over TCP (BT-683), not raw TCP. The original reasoning for choosing TCP over Unix sockets still applies — WebSocket inherits TCP's network capability while adding HTTP-upgrade compatibility for browser access and standard proxy support.

  1. The workspace is an Erlang/OTP process. Cowboy (HTTP/WebSocket server) is native to the BEAM ecosystem and well-supported. Unix socket support in Erlang requires workarounds (afunix or NIF-based solutions) and isn't portable to Windows.
  2. Remote attach needs network transport. The REPL protocol must work over the network for beamtalk workspace attach. WebSocket is HTTP-upgrade-compatible, so standard proxies (Caddy, nginx) can terminate TLS without custom bridging code.
  3. Browser access requires HTTP. Raw TCP cannot be reached from a browser. WebSocket provides bidirectional communication over HTTP, enabling browser-based workspaces (ADR 0017).
  4. The compiler is now an OTP Port (ADR 0022). The Unix socket daemon was eliminated — the compiler runs as a child process of the workspace, communicating via stdin/stdout.

Trade-off acknowledged: WebSocket on localhost is less secure than Unix sockets on shared machines. We mitigate this with cookie-based authentication on WebSocket connect (see Layer 1 below).

Design Principles

  1. Local stays local. 127.0.0.1 binding is the default and cannot be overridden by accident. No "listen on all interfaces" flag.
  2. Security at the right layer. Don't reinvent TLS inside the beamtalk protocol. Use the transport layer (OTP ssl, network overlays, reverse proxies).
  3. Zero-config for dev, explicit opt-in for remote. Local REPL should Just Work™. Remote access requires deliberate setup.
  4. Composable with existing infrastructure. Support Tailscale, WireGuard, SPIFFE, and standard reverse proxies — don't mandate any single approach.

Decision

Layer 1: Local connections (127.0.0.1 + cookie handshake)

Local REPL connections remain loopback-only. The compiler uses an OTP Port (stdin/stdout, no network). The REPL protocol uses WebSocket as its sole transport, with a cookie handshake on connection to authenticate the client on shared machines.

┌──────────────┐   WS ws://127.0.0.1:{port}  ┌──────────────────┐
│  beamtalk    │ ──────────────────────────── │  Workspace Node  │
│  repl (CLI)  │  1. WebSocket connect        │  (BEAM + cowboy)  │
│              │  2. Send cookie message       │                  │
│              │  3. Server validates          │                  │
│              │  4. JSON protocol begins      │                  │
└──────────────┘                              └──────────────────┘

Single transport: WebSocket. The workspace uses cowboy (standard Erlang HTTP/WebSocket server) instead of gen_tcp. The REPL protocol is the same JSON messages, carried over WebSocket frames instead of newline-delimited TCP. This means:

Why WebSocket only (not TCP + WebSocket): One transport means one implementation to test and maintain. WebSocket is HTTP-upgrade-compatible, so standard proxies (Caddy, nginx, envoy) can terminate TLS, handle auth headers, and forward without custom bridging code. This follows nREPL's principle of transport-agnostic protocol design — the JSON messages are the protocol; WebSocket is the transport.

Cookie handshake protocol:

→ {"type": "auth", "cookie": "<workspace cookie>"}
← {"type": "auth_ok"}

or on failure:

← {"type": "auth_error", "message": "Invalid cookie"}
→ [connection closed by server]

The workspace cookie already exists at ~/.beamtalk/workspaces/{id}/cookie with chmod 600. The CLI reads it from disk; the workspace validates it on first WebSocket message. This adds one message to connection setup.

Why this matters: On single-user machines, loopback binding is sufficient. On shared machines (CI runners, Docker --net=host, multi-user servers), any local process can connect to the port. The cookie handshake ensures only processes with read access to the cookie file can eval code.

Rationale for cookie (not mTLS): Any process that can read cert files can also read the cookie file. Cookie auth is simpler and provides equivalent security to mTLS for the localhost threat model. The OS filesystem ACL (chmod 600) is the actual authentication boundary.

Cookie lifecycle: The workspace cookie persists across workspace restarts, following the Erlang cookie model (~/.erlang.cookie). Cookie rotation is a separate concern — a future beamtalk workspace rotate-cookie command can regenerate it when needed, but automatic rotation on restart would break reconnecting clients unnecessarily.

Layer 2: Web terminal via standard reverse proxy

Since the workspace speaks WebSocket natively, the browser can connect directly for local dev. For remote access or TLS, use a standard reverse proxy (Caddy, nginx, envoy) — zero custom proxy code.

Local dev (no proxy needed):

┌────────────┐  ws://localhost:{port}     ┌────────────────────┐Browser   │ ────────────────────────── │  Workspace Node    │
│  (xterm.js)│    (WebSocket, direct)     │  (cowboy)          │
└────────────┘                            └────────────────────┘

The browser loads a static xterm.js page (served by beamtalk web from http://localhost) and connects to ws://localhost:{port} (port discovered from the workspace port file). The cookie handshake authenticates the connection. The page must be served over HTTP (not loaded from file://) so that the Origin header is present for WebSocket validation.

Remote/TLS (standard proxy):

┌────────────┐    wss://              ┌───────┐    ws://127.0.0.1    ┌────────────┐Browser   │ ────────────────────── │ Caddy │ ────────────────── │  Workspace │
│            │  (TLS, encrypted)      │       │  (loopback)         │  (cowboy)   │
└────────────┘                        └───────┘                     └────────────┘
# Caddyfile — entire proxy config
localhost:8443 {
    reverse_proxy localhost:{port}  # port from ~/.beamtalk/workspaces/{id}/port
}

# Or with Tailscale:
my-laptop.tail12345.ts.net:8443 {
    reverse_proxy localhost:{port}
}

Authentication options:

MethodWhenHow
Cookie handshakeDefault (local)Browser sends workspace cookie in first WebSocket message (same as CLI). User provides cookie via the xterm.js page UI.
Session tokenConveniencebeamtalk web generates a one-time URL token, exchanged for HttpOnly; Secure; SameSite=Strict cookie. Prevents leakage via browser history.
Proxy-level authRemote/teamCaddy handles OAuth/OIDC, mTLS, or basic auth before forwarding to workspace.

DNS rebinding mitigation: The cowboy WebSocket handler in the workspace must:

  1. Validate the Origin header on WebSocket upgrade — reject cross-origin connections
  2. Require the cookie handshake before accepting any eval commands

The workspace always binds to 127.0.0.1 by default. Network exposure is handled by the reverse proxy, not by changing the workspace bind address. For overlay networks (Tailscale), the workspace can optionally bind to the overlay IP:

# NOTE: The following commands illustrate proposed CLI UX for Phases 1-2.
#       Some subcommands/flags (e.g. `beamtalk web`, `--bind tailscale`) are not
#       available in the current CLI yet.

# Default: local only (browser connects directly, no proxy needed)
beamtalk repl
# → Workspace listens on ws://127.0.0.1:{OS-assigned port}

# Serve xterm.js web terminal (proposed UX — Phase 1, not yet implemented)
beamtalk web
# → Opens browser to http://localhost page that connects to ws://localhost:{port}

# Remote via Caddy (workspace stays on loopback)
caddy reverse-proxy --from :8443 --to localhost:{port}

# Remote via Tailscale (workspace binds to overlay IP) — proposed UX, Phase 2
beamtalk run server.bt --bind tailscale
# → ws://100.64.x.x:{port}, only reachable by Tailscale peers

# Safety check for non-loopback binding — proposed UX, Phase 2
beamtalk run server.bt --bind 0.0.0.0
# ERROR: Binding to all interfaces exposes the workspace to the network.
#        Use --confirm-network to proceed, or use --bind tailscale for secure remote access.

Layer 3: Remote attach via mTLS or network overlay

For beamtalk workspace attach prod@host, two approaches are supported. The user chooses based on their infrastructure.

Option A: Network overlay (Tailscale, WireGuard, VPN)

The simplest path. The overlay provides encryption and identity. Beamtalk treats the overlay network as "local":

┌──────────────┐     WireGuard tunnel      ┌──────────────────┐
│  Developer   │ ────────────────────────── │  Production      │
│  100.64.0.1  │     (encrypted)            │  100.64.0.2      │
└──────────────┘                            └──────────────────┘
        │                                            │
beamtalk workspace attach prod@100.64.0.2               beamtalk repl
(connects to 100.64.0.2:{port})               (listens on 100.64.0.2:{port})

Changes needed:

# Start workspace listening on Tailscale interface
beamtalk run server.bt --bind tailscale
# → Binds to Tailscale IP (100.64.x.x), only reachable by Tailscale peers

# Connect from another machine on same tailnet
beamtalk workspace attach my-server.tail12345.ts.net

Tailscale-specific integration:

Option B: mTLS on Erlang distribution

For environments without a network overlay, enable OTP's built-in TLS for distributed Erlang:

┌──────────────┐     TLS (mTLS)            ┌──────────────────┐
│  Developer   │ ────────────────────────── │  Production      │
│  BEAM node   │  (client+server certs)     │  BEAM node       │
└──────────────┘                            └──────────────────┘

Erlang's ssl_dist module provides mTLS for the Erlang distribution protocol with minimal configuration:

%% vm.args (or generated by beamtalk CLI)
-proto_dist inet_tls
-ssl_dist_optfile ~/.beamtalk/tls/ssl_dist.conf
%% ssl_dist.conf
[{server, [
    {certfile, "server.pem"},
    {keyfile,  "server-key.pem"},
    {cacertfile, "ca.pem"},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true}
]},
 {client, [
    {certfile, "client.pem"},
    {keyfile,  "client-key.pem"},
    {cacertfile, "ca.pem"},
    {verify, verify_peer}
]}].

Certificate management — auto-generated per workspace:

~/.beamtalk/tls/
├── ca.pem              # Self-signed CA (generated once)
├── ca-key.pem          # CA private key
├── workspaces/
│   └── {id}/
│       ├── server.pem      # Workspace server cert (signed by CA)
│       ├── server-key.pem  # Workspace server key
│       ├── client.pem      # Client cert (for attaching)
│       └── client-key.pem  # Client key
# Auto-generate CA on first use
beamtalk tls init
# → Creates ~/.beamtalk/tls/ca.pem (self-signed, long-lived)

# Auto-generate workspace certs
beamtalk workspace create my-feature --tls
# → Creates per-workspace server + client certs signed by CA

# Attach with mTLS
beamtalk workspace attach prod@host --tls
# → Uses client cert from ~/.beamtalk/tls/workspaces/{id}/

Option C: SPIFFE/SPIRE (future — cloud-native deployments)

For Kubernetes and service-mesh environments, SPIFFE provides automatic workload identity:

┌──────────────┐                           ┌──────────────────┐
│  SPIRE Agent │ ─── issues x509-SVID ──── │  Workspace Node  │
│  (per node)  │     (auto-rotated)         │  SPIFFE ID:      │
└──────────────┘                            │  spiffe://bt/    │
                                            │  workspace/prod  │
                                            └──────────────────┘

This is deferred to a future ADR when Kubernetes deployment becomes a priority. The mTLS infrastructure from Option B provides a natural integration point — SPIRE-issued certs can replace the self-signed CA certs with zero protocol changes.

Summary: What secures what

ScenarioTransport SecurityIdentity/AuthConfig
beamtalk repl (local)Loopback bindingCookie handshakeDefault, zero config
Browser (local)Direct WebSocketCookie handshakebeamtalk web serves xterm.js
Browser (remote)Caddy/nginx TLSCookie + proxy auth (OAuth, mTLS)Standard reverse proxy
beamtalk workspace attach via TailscaleWireGuardTailscale identity + cookie--bind tailscale
beamtalk workspace attach via mTLSTLS (ssl_dist)Client certificate--tls flag
Future: K8s/SPIFFEmTLS (SVID)SPIFFE workload identitySPIRE agent

Prior Art

Jupyter Notebook

Erlang/OTP ssl_dist

Tailscale / WireGuard

SPIFFE/SPIRE

LiveBook (Elixir)

nREPL (Clojure)

User Impact

Newcomer

Smalltalk developer

Erlang/BEAM developer

Operator (production)

Steelman Analysis

"Just use mTLS everywhere, including localhost"

"Use Unix sockets for local REPL, TCP only for remote"

"Tailscale is a vendor dependency"

"SPIFFE should be the primary approach"

Alternatives Considered

Do nothing (status quo)

Unix sockets for local REPL

SSH tunneling for remote attach

Token-only auth (no TLS)

Keep TCP + add WebSocket (dual transport)

Custom encryption protocol

Consequences

Positive

Negative

Neutral

Implementation

Phase 0: WebSocket transport + cookie handshake

Phase 1: Network binding + standard proxy documentation

Phase 2: mTLS for Erlang distribution

Phase 3: SPIFFE/SPIRE integration (future)

Migration Path

The WebSocket migration (Phase 0) is a breaking change to the REPL transport layer. Since beamtalk is pre-1.0, there is no backward compatibility obligation, but the transition should be handled cleanly:

Phase 0 migration (WebSocket transport + cookie handshake)

  1. CLI and workspace must be updated together. A new CLI using tungstenite cannot connect to a workspace still using gen_tcp, and vice versa. Both sides ship in the same release.
  2. No version negotiation. The old TCP protocol has no version handshake, so there is no way to gracefully negotiate transport. If a user runs a new CLI against an old workspace (or vice versa), the connection will fail immediately with a clear error message (e.g., "Connection refused — workspace may be running an older version. Run beamtalk workspace restart to upgrade.").
  3. Running workspaces must be restarted. Existing workspace processes speak the old TCP protocol and cannot be hot-upgraded to WebSocket. Users must stop and restart workspaces after updating: beamtalk workspace stop && beamtalk repl.
  4. Cookie file creation is automatic. If a workspace has no cookie file (~/.beamtalk/workspaces/{id}/cookie), one is generated on first startup. Existing workspaces get a cookie on next restart.

User-visible steps

# After updating beamtalk CLI:
beamtalk workspace stop          # Stop old workspace
beamtalk repl                    # Starts new workspace with WebSocket + cookie

Later phases (no migration needed)

Phases 1–3 (network binding, mTLS, SPIFFE) are additive features. They do not change the core protocol and require no migration from users already on WebSocket transport. Browser UI is handled by ADR 0017.

Scope Boundaries

This ADR does not cover:

Implementation Tracking

Epic: BT-685 Status: Phases 0–2 Done

PhaseIssueTitleSizeStatus
0BT-683Migrate REPL transport from TCP to WebSocket with cookie authLDone
1BT-691Network bind options and reverse proxy documentationMDone
2BT-692mTLS for Erlang distributionLDone
3BT-693SPIFFE/SPIRE workload identity integrationXLFuture

References