Secure Access Tunneling
The internal/tunnel package provides platform-mediated access to mesh nodes through WireGuard tunnels without exposing services to the public internet. The control plane declares a node's live sessions in the sessions block of the reconciliation state pull; the node agent holds its listeners level with that block, opening a listener bound to the mesh IP for each entry it can provision, and reporting activity back to the control plane. A tcp entry gets a TCP listener that forwards connections to the target host. On Linux and macOS, an ssh entry gets an SSH listener that starts a shell or one command for the session's user (see SSH Sessions).
The block is desired state, not a delivery queue: an entry stands for as long as its session is valid, and its disappearance is the teardown signal. Revocation and hard expiry both reach the node as that same absence, and there is no terminal status to report. A revocation the control plane also announces in a verified session_revoked event is closed when the event arrives, unless a pull drained it first (see SSE Event Stream); the absence stays the teardown for every revocation whose event never arrives.
Data Flow
Control Plane
│
│ GET /v1/nodes/{id}/state → sessions block
▼
┌─────────────────┐ ┌───────────────────┐
│ Reconciler │──▶│ tunnel.Dispatcher │
│(internal/recon- │ │ Handle │
│ cile) │ └─────────┬─────────┘
└─────────────────┘ │
▼
┌───────────────┐
│ SessionManager │
│ CreateSession │
└───────┬───────┘
│
┌───────┴───────┐
│ Session │
│ │
│ ┌───────────┐ │
│ │ Listener │ │ ← bound to meshIP:0
│ │ (TCP) │ │
│ └─────┬─────┘ │
│ │ │
│ ┌─────┴─────┐ │
│ │ Forwarder │ │ ← bidirectional io.Copy
│ └─────┬─────┘ │
└───────┼───────┘
│
▼
Target Host
(e.g. sshd)Reconciliation Sequence
- The reconciler pulls
GET /v1/nodes/{node_id}/stateon its cadence, or immediately when an SSE event —session_setupamong them — triggers a cycle Dispatcher.Handleruns on every successful pull, before the empty-diff short-circuit, so an unchanged snapshot still reconciles sessions- Teardown pass: every live session whose entry is no longer in the block is closed, with the reason derived from its capped local expiry
- Provision pass, in block order: each entry not yet settled is validated and, if it is a
tcpentry or, on Linux and macOS, ansshentry, handed toSessionManager.CreateSession, which opens a listener onmeshIP:0 - Report pass: once the block is provisioned,
SessionActivityReporter.ReportSessionStartedposts asession_startedrow per listener whose row is still outstanding — thetcprow with the target host and port, or thesshlifecycle row, each with thelistener_endpointthe listener actually bound. A row that does not land is re-posted on the next pull from the same listener, solistener_endpointstays stable across attempts. The posts run concurrently and each is bounded, so a stalled control plane cannot hold the reconciler's single goroutine for the length of the block. A row answered with a terminal-state verdict (a409session_already_revokedorsession_expired, classified byapi.IsSessionRevokedOrExpired) is settled rather than re-posted and is logged at Info (session no longer live server-side; awaiting the drain); the listener stays up until the block drain, or a verifiedsession_revokedevent naming the session, closes it, and every pull that still carries the settled entry warns that the listener is forwarding for a session the platform has withdrawn. A404session_not_foundis not a terminal state — the block and the session store are separate reads that can disagree — so it stays on the re-post path - A client connects through the mesh to that listener;
Sessionforwards to the target - A session ends when its entry drains from the block, when a verified
session_revokedevent names it (Dispatcher.SessionRevokedHandler), when its capped expiry fires, when its idle window elapses, or onShutdown - The
SessionManager's on-closed callback fires the production reporter'sReportSessionEnded, posting asession_endedrow with aterminated_byreason for every close reason,Shutdownincluded: thetcprow carries the byte counters, thesshrow picks its shape fromClosedSessionInfo.Kind. For a close caused by the drain or by asession_revokedevent the platform already holds the session as revoked or expired, and it records the ending row of the session's current listener once instead of refusing it. A409to the ending row means the listener's end is already recorded, or a control plane that predates that rule; the row is fire-once and nothing re-posts it, so plexd warns (tunnel session ended row refused; its counters are recorded only here, orssh session ended row refused; its terminating reason is recorded only herefor an ssh session) and carriesterminated_by, plus atcprow's byte counters, into that line, which is where they survive
Config
Config holds secure access tunneling parameters passed to the SessionManager constructor.
| Field | Type | Default | Description |
|---|---|---|---|
Enabled | bool | true | Whether tunneling is active |
MaxSessions | int | 10 | Maximum concurrent tunnel sessions |
DefaultTimeout | time.Duration | 30m | Default/maximum session timeout |
SSHListenAddr | string | — | SSH mesh server listen address (empty = no SSH server) |
HostKeyDir | string | — | Directory for SSH host key (empty = transient key) |
SSHSessionsEnabled | *bool | true | ssh_sessions_enabled: whether the node serves mediated ssh sessions; read through SSHSessionsAllowed(), where nil means true |
SessionSigningPublicKey | string | — | session_signing_public_key: standard-base64 Ed25519 key the session helper verifies tokens against; empty means the helper pins the identity's key. Read only by Linux's socket-activated session helper; unused on macOS (see Trust anchor) |
cfg := tunnel.Config{
MaxSessions: 5,
}
cfg.ApplyDefaults() // Enabled=true, DefaultTimeout=30m, MaxSessions stays 5
if err := cfg.Validate(); err != nil {
log.Fatal(err)
}Default Heuristic
ApplyDefaults uses zero-value detection: on a fully zero-valued Config, MaxSessions == 0 triggers all defaults including Enabled = true. If MaxSessions is already set (indicating explicit construction), Enabled is left as-is. This allows Config{Enabled: false} to disable tunneling after ApplyDefaults.
Validation Rules
| Field | Rule | Error Message |
|---|---|---|
MaxSessions | Must be > 0 when enabled | tunnel: config: MaxSessions must be positive when enabled |
DefaultTimeout | Must be >= 1m when enabled | tunnel: config: DefaultTimeout must be at least 1m when enabled |
SessionSigningPublicKey | Empty, or standard base64 decoding to 32 bytes | tunnel: config: session_signing_public_key: ... |
The key is checked whatever Enabled says, because the session helper reads it either way. The other rules are skipped when Enabled is false.
Session
Represents an active tunnel session with a local TCP listener that forwards connections to a target host through the mesh.
Fields
| Field | Type | Description |
|---|---|---|
SessionID | string | Unique session identifier |
TargetHost | string | Target host to forward connections to |
TargetPort | int | Target port |
MeshIP | string | Mesh IP to bind the listener to |
Constructor
func NewSession(sessionID, targetHost string, targetPort int, meshIP string, expiresAt time.Time, logger *slog.Logger) *SessionMethods
| Method | Signature | Description |
|---|---|---|
Start | (ctx context.Context) (string, error) | Opens TCP listener on meshIP:0, starts accept loop and, when an idle window is armed, the idle monitor |
Close | () error | Idempotent shutdown: cancels context, closes listener and connection, then waits (bounded) for the forwarding goroutines so the byte counters are complete |
IdleFor | () time.Duration | How long the session has gone without observed byte flow, measured monotonically |
ListenAddr | () string | Returns listener address or empty string if not started |
Connection Lifecycle
Startbinds a TCP listener tomeshIP:0(ephemeral port, mesh-only interface) and derives the child context every goroutine it starts runs on, soClosecancels all of themacceptLoopruns in a goroutine, accepting one connection at a time- Single-connection enforcement: if a connection is already active, new connections are rejected
forwarddials the target, sets the active connection under mutex, and runs bidirectionalio.Copywithsync.Oncecleanup andsync.WaitGroupfor completionCloseis idempotent viasync.Mutex+closedflag; cancels context, closes listener and active connection
Security
- Listener binds to mesh IP only, never
0.0.0.0orlocalhost.SessionManager.CreateSessionparses the mesh IP and refuses to provision a session at all unless it is a bindable unicast address, becausenet.Listenon an empty or unspecified host binds every unicast address the host has. The check runs on the addressnet.Listenwould actually bind — zone stripped and IPv4-mapped form unwrapped — so::ffff:0.0.0.0and::%eth0are refused as the wildcards they bind to, not accepted as the distinct addresses they parse to - At most one active forwarded connection per session
- Context cancellation propagates to listener and active connection
- The forward itself is not authenticated: whichever client reaches the listener first takes the single connection slot. Reachability to the node's mesh IP is the only control (see WireGuard Mesh)
SessionManager
Central coordinator for tunnel session lifecycle.
Constructor
func NewSessionManager(cfg Config, meshIP string, logger *slog.Logger) *SessionManager- Applies config defaults via
cfg.ApplyDefaults() - Logger is tagged with
component=tunnel
Methods
| Method | Signature | Description |
|---|---|---|
CreateSession | (ctx context.Context, sess api.NodeStateSession) (string, error) | Validates one entry of the sessions block, creates and starts its session, and returns the bound listener address |
SetSSHSessionDeps | (deps SSHSessionDeps) | Wires the host key, signing keys, command reporter and session launcher ssh sessions need; plexd up calls it on Linux and macOS |
CloseSession | (sessionID string, reason string) *ClosedSessionInfo | Closes and removes a session by ID; returns session info |
Shutdown | () | Closes all active sessions, reporting each through the on-closed callback |
ActiveSessions | () map[string]time.Time | Snapshot of live session IDs and their capped local expiry; the dispatcher's teardown input |
ActiveCount | () int | Returns number of active sessions |
CreateSession Validation
tcp entries are provisionable everywhere, ssh entries on Linux and macOS once SetSSHSessionDeps wired them. The dispatcher filters the block before calling in; the kind guard repeats the check so the manager is safe to call on its own. Both kinds share one session map, so MaxSessions and the DefaultTimeout cap count them together.
| Check | Condition | Error |
|---|---|---|
| Tunneling disabled | cfg.Enabled == false | ErrTunnelingDisabled (tunnel: tunneling is disabled) |
| Not provisionable | Kind is neither tcp nor ssh, or the entry lacks the target of its kind | tunnel: session kind is not provisionable |
Missing fields (tcp) | Empty ID, empty host, or port outside 1-65535 | tunnel: invalid session setup: ... |
Unsupported (ssh) | Neither Linux nor macOS | ErrSSHSessionsUnsupported (tunnel: ssh sessions are not supported on this platform) |
Refused (ssh) | ssh_sessions_enabled: false | ErrSSHSessionsDisabled (tunnel: ssh sessions are disabled on this node) |
Missing user (ssh) | Empty target.ssh.user | tunnel: invalid session setup: ssh target user is required |
Not wired (ssh) | Any SSHSessionDeps member is nil | tunnel: ssh sessions are not wired: host key, signing keys, command reporter and session launcher are required |
Missing ID (ssh) | Empty session_id | tunnel: invalid session setup: session_id is required |
| Already expired | ExpiresAt in the past | tunnel: session already expired |
| Duplicate ID | Session ID already exists | tunnel: duplicate session ID: {id} |
| Capacity | len(sessions) >= MaxSessions | tunnel: max sessions reached ({n}) |
| Invalid idle window | IdleTimeoutSeconds negative or above 86400 | tunnel: invalid idle_timeout_seconds: {n} |
| Unbindable mesh IP | meshIP empty, not an IP address, multicast, or unspecified once the zone is stripped and the IPv4-mapped form unwrapped (0.0.0.0, ::, ::ffff:0.0.0.0, ::%eth0) | tunnel: mesh IP {ip} is not a bindable unicast address; refusing to bind a session listener |
ErrTunnelingDisabled and ErrSSHSessionsDisabled are sentinels: the dispatcher matches them with errors.Is to settle an entry permanently instead of retrying it on every pull.
Expiry
ExpiresAtis capped atDefaultTimeoutfrom now (never exceeds maximum)time.AfterFuncschedules automaticCloseSession(id, "expired")at the capped expiry time. The timer closes over the session, not just its ID, so a timer armed for an earlier session cannot close the one that later took its ID- A cap that truncates the granted
expires_atis logged at warn: the control plane does not learn the node's local maximum from the pull, so a session granted more time thanDefaultTimeoutends early — with attl_expiredsession_endedrow — while its entry still stands in the block, and it is not provisioned a second time
Idle Enforcement
An entry with idle_timeout_seconds > 0 arms an idle monitor; 0 or an absent value means the session has no idle window.
idle_timeout_secondsis validated on the way in: a negative value, or one large enough to overflow the seconds-to-Durationmultiplication, is rejected rather than silently read as "no idle window"- Byte flow re-arms the window: every forwarded chunk of a tcp session, and every byte read from or written to any connection of an ssh session, stamps the session's last activity, so the monitor either closes the session or waits out the remaining window
- Activity is stamped as a monotonic offset, not a wall-clock time: an NTP step or a VM snapshot resume cannot stretch or collapse the window
- The listener bind counts as the first activity, so a listener no connection ever reaches idles out one window after
Start - The close runs through
CloseSession(id, "idle")like any other, so thesession_endedrow still carries the byte counters and, via the reason, aterminated_byofidle_timeout - The monitor is started by
Session.Starton its own child context and exits when that context is cancelled, which every close reaches
Lifecycle
logger := slog.Default()
mgr := tunnel.NewSessionManager(tunnel.Config{}, "10.0.0.1", logger)
// Create a session from one entry of the pull's sessions block
addr, err := mgr.CreateSession(ctx, api.NodeStateSession{
SessionID: "sess-abc",
JTI: "sess-abc",
Kind: api.SessionKindTCP,
Target: api.SessionTarget{
TCP: &api.SessionTargetTCP{Host: "127.0.0.1", Port: 22},
},
ExpiresAt: time.Now().Add(10 * time.Minute),
IdleTimeoutSeconds: 900,
})
// Close specific session
mgr.CloseSession("sess-abc", "drained")
// Graceful shutdown (closes all sessions)
mgr.Shutdown()Dispatcher
Dispatcher consumes the sessions block of the reconciliation pull and holds the node's listeners level with it. Handle matches reconcile.DispatchHandler, so it runs on every successful pull, before the diff.
func NewDispatcher(manager *SessionManager, reporter SessionActivityReporter, logger *slog.Logger) *Dispatcher
func (d *Dispatcher) Handle(ctx context.Context, desired *api.NodeStateSnapshot)
func (d *Dispatcher) SessionRevokedHandler(trigger api.ReconcileTrigger) api.EventHandlerHandle is not safe for concurrent use: it is invoked only from the reconcile goroutine, one cycle at a time. The handler SessionRevokedHandler returns (see SSE Event Stream) runs on the SSE goroutine, concurrently with Handle; it touches only the SessionManager, which locks for itself, and the revoked set, which a mutex guards.
Unpopulated Block
Handle returns immediately, touching nothing, when the snapshot's sessions block is absent or null — api.NodeStateSnapshot.Sessions is a *[] so that "the control plane says this node has no sessions" ([]) stays distinguishable from "the control plane did not populate the block". Emptiness is destructive here: reading a rolled-back control plane, a fleet instance predating the block, or one degraded response as an empty block would kill every live operator forward on every node that pulls from it. The refusal is logged once per such pull.
Teardown Pass
The ID set of the whole block is collected first — absence from the block is what drives teardown, so the set has to be complete before the first close. Every live session whose ID is not in that set is closed — concurrently, because each close blocks on the session drain and on a bounded activity post, and this pass runs inline on the reconciler's single goroutine ahead of the diff and every reconcile handler. The reason is derived from the session's capped local expiry, the only discriminator the node has for why an entry left:
| Condition at teardown | Close reason | Reported terminated_by |
|---|---|---|
| Capped expiry is still in the future | drained | plexd_close |
| Capped expiry has passed | expired | ttl_expired |
A session a verified session_revoked event already closed is no longer live here, so this pass does not close it again: its session_ended row went out with the terminated_by the event's revoke_reason maps to.
A drain does not report operator_revoke. Only a lapsed local expiry is self-evident; an entry that simply is not there may be a revocation, a lagging read replica, a control plane mid-rollout, or a filtering gateway, and the node cannot tell them apart. Asserting a human action in the audit trail on that evidence would answer "who revoked this session?" confidently and wrongly. The node reports operator_revoke only for a close caused by a verified session_revoked event, which is signed and names the session. A pull that drains the entry before that event arrives still reports plexd_close, and the late event finds no live session to close.
Provision Pass
Entries are handled in block order. A known set records every session ID the dispatcher has already acted on while its entry stands — provisioned, observed live, or permanently settled — so a settled entry is a no-op on every later pull:
| Entry state | Outcome | Settled |
|---|---|---|
Empty session_id | Error log (session entry carries no session_id; dropping), entry dropped | n/a (never enters known) |
Already in known | No-op | already |
In the revoked set (a verified session_revoked event named it) | Info (session was revoked by a signed event; not provisioning it); not provisioned, and an outstanding session_started row is dropped | yes |
Valid entry the event revoked while CreateSession was binding its listener | Info (session was revoked by a signed event while its listener came up; closing it); the listener is closed at once with the event's reason and no session_started row is posted | yes |
Live session with an outstanding session_started row | The row is re-posted from the address the listener already holds; no second setup | no |
Live session not yet in known | Recorded as known; the listener is already bound, so no second setup | yes |
expires_at is the zero time | Warning (session carries no expires_at; refusing to provision) | yes |
expires_at is not in the future | Info (session expired; waiting for the control plane to drain the entry) | yes |
kind is k8s, or ssh off Linux and macOS | One warning (unsupported session kind; no listener provisioned); no listener, no activity row | yes |
kind is none of the three the contract names | Warning (unrecognised session kind; no listener provisioned) — the target is fine, the kind is not one this build knows | yes |
tcp or ssh entry without the target of its kind, or with more than one target member set | Warning (session target does not match kind) | yes |
Valid tcp or ssh entry, CreateSession succeeds | Listener bound; ReportSessionStarted posts the session_started row with listener_endpoint | yes |
The session_started row does not reach the control plane | Warning; the listener stays bound and the entry is left unsettled so the next pull re-posts the row from the same listener_endpoint | no |
The session_started row is answered 409 session_already_revoked / session_expired | Info (session no longer live server-side; awaiting the drain); the listener stays bound, the entry is settled, the teardown waits for the block drain or a verified session_revoked event, and each later pull still carrying the entry warns | yes |
CreateSession returns ErrTunnelingDisabled | Warning (tunneling is disabled; no listener provisioned) — configuration, not weather | yes |
CreateSession returns ErrSSHSessionsDisabled | Warning (ssh sessions are disabled on this node; no listener provisioned) | yes |
CreateSession fails otherwise | Warning (session listener setup failed); left unsettled so the next pull retries | no |
k8s entries are part of the contract but not of this agent's mediation, and ssh entries are served on Linux and macOS only. Both are settled rather than retried, so the warning is emitted once per entry rather than once per pull. An ssh entry with an empty user, or one reaching a manager nobody wired, is a setup failure and is retried.
The revoked set holds the IDs a verified session_revoked event named. The control plane never makes a revoked session live again, but a pull rendered before the revocation committed can still list its entry; that entry is settled rather than provisioned, so the listener the event closed is not bound again.
An entry draining from the block also prunes its ID from known, from the outstanding-row set and from the revoked set, which bounds all three by the size of the block. The pass carries no dispatch budget: it is bounded by the live sessions MaxSessions caps plus the length of the block, and its only network call is a single bounded activity post per session whose started row is still outstanding.
Registration
mgr := tunnel.NewSessionManager(tunnel.Config{}, meshIP, logger)
sessionDispatcher := tunnel.NewDispatcher(mgr, reporter, logger)
reconciler.RegisterDispatchHandler(sessionDispatcher.Handle)
sseMgr.RegisterHandler(api.EventSessionRevoked, sessionDispatcher.SessionRevokedHandler(reconciler))SessionActivityReporter
Interface for reporting a session's session_started row to the control plane. Abstracted for testability.
type SessionActivityReporter interface {
ReportSessionStarted(ctx context.Context, entry api.NodeStateSession, listenerEndpoint string) error
}
type SSHCommandReporter interface {
ReportSSHCommandStarted(ctx context.Context, sessionID, command string, startedAt time.Time) error
ReportSSHCommandExited(ctx context.Context, sessionID, command string, exitCode int, startedAt, completedAt time.Time) error
}The production implementation, controlPlaneSessionReporter in cmd/plexd/cmd/up.go, posts an api.SessionActivityRequest to api.ControlPlane.ReportSessionActivity (POST /v1/nodes/{node_id}/sessions/{session_id}). For a tcp entry ReportSessionStarted emits a tcp session_started row with the target and listenerEndpoint, the address the listener actually bound, so the control plane has somewhere to send the operator; for an ssh entry it emits the ssh lifecycle row with listener_endpoint. ReportSessionEnded(ctx, sessionID, info *ClosedSessionInfo, terminatedBy) emits a session_ended row: the tcp row with the byte counters, or, when info.Kind is ssh, the ssh row with terminated_by only. ReportSessionEnded is not on the interface: it is fired from the SessionManager's on-closed callback that plexd up wires with SetOnClosed. The same type implements SSHCommandReporter, whose two rows the ssh listener posts around every exec (see Commands and activity rows); both return the post's error.
ReportSessionStarted returns the post's error because the row is not merely an audit record: listener_endpoint is the operator's only route to the listener, so the dispatcher has to know whether the row landed. A terminal-state answer (api.IsSessionRevokedOrExpired: a 409 session_already_revoked or session_expired) settles the entry and is logged at Info; the listener stays bound until the block drain, or a verified session_revoked event naming the session, closes it. Every other error (a transport failure, a 400, a 404 session_not_found, a 501 access_session_not_provisioned, a 5xx) keeps the listener bound, leaves the entry unsettled, and re-posts the row on the next pull from the same listener_endpoint, logged at Warn. ReportSessionEnded returns nothing and is fire-once from the on-closed callback: a 409 is logged at Warn carrying terminated_by and, for a tcp row, the byte counters (the control plane records the ending row of a revoked or expired ssh or tcp session's current listener once, unless its newest lifecycle row is already a session_ended, so a 409 to it means the listener's end is already recorded, or a control plane that predates that rule; nothing re-posts the row, so the line is where that payload survives), any other failure at Error, and nothing is retried.
API Types
Types defined in internal/api for tunnel communication with the control plane.
NodeStateSession
One entry of the sessions block of GET /v1/nodes/{node_id}/state. JTI equals the session ID: it is carried as an opaque value and never evaluated — the node does not authorize connections by it. ExpiresAt is an absolute UTC timestamp, not a relative timeout. IdleTimeoutSeconds 0 or absent means the session has no idle window.
type NodeStateSession struct {
SessionID string `json:"session_id"`
JTI string `json:"jti"`
Kind string `json:"kind"`
Target SessionTarget `json:"target"`
ExpiresAt time.Time `json:"expires_at"`
IdleTimeoutSeconds int `json:"idle_timeout_seconds,omitempty"`
}SessionTarget
Exactly one member is set, matching Kind — SessionKindSSH (ssh), SessionKindK8s (k8s), or SessionKindTCP (tcp). This agent provisions tcp entries, and ssh entries on Linux and macOS; k8s entries, and ssh entries on Windows, are decoded and settled as unsupported. The ssh listener takes the allowed-command list from the session token, not from SessionTargetSSH.AllowedCommands.
type SessionTarget struct {
SSH *SessionTargetSSH `json:"ssh,omitempty"`
K8s *SessionTargetK8s `json:"k8s,omitempty"`
TCP *SessionTargetTCP `json:"tcp,omitempty"`
}
type SessionTargetSSH struct {
User string `json:"user"`
AllowedCommands []string `json:"allowed_commands,omitempty"`
}
type SessionTargetK8s struct {
User string `json:"user"`
ImpersonateGroups []string `json:"impersonate_groups,omitempty"`
}
type SessionTargetTCP struct {
Host string `json:"host"`
Port int `json:"port"`
}SessionActivityRequest
Posted by the node agent per session event. Exactly one of SSH, K8s, or TCP is set, selecting the session kind. A tcp session sets TCP and an ssh session sets SSH; the K8s variant is carried by the type and accepted by the control plane but not emitted.
type SessionActivityRequest struct {
SSH *SSHActivity `json:"ssh,omitempty"`
K8s *K8sActivity `json:"k8s,omitempty"`
TCP *TCPActivity `json:"tcp,omitempty"`
}Endpoint: POST /v1/nodes/{node_id}/sessions/{session_id} (success: 204 No Content)
TCPActivity
The tcp member: a session lifecycle row. Phase is session_started or session_ended. ListenerEndpoint is the node's bound listener address and is set on session_started rows only. BytesIn (operator→target) and BytesOut (target→operator) are pointers so a session_ended row carries explicit zeros while a session_started row omits the byte counters. TerminatedBy, when set, is one of the TerminatedBy* values.
type TCPActivity struct {
Phase string `json:"phase"`
TargetHost string `json:"target_host,omitempty"`
TargetPort int `json:"target_port,omitempty"`
ListenerEndpoint string `json:"listener_endpoint,omitempty"`
BytesIn *int64 `json:"bytes_in,omitempty"`
BytesOut *int64 `json:"bytes_out,omitempty"`
TerminatedBy string `json:"terminated_by,omitempty"`
}SSHActivity
The ssh member takes one of two shapes. A lifecycle row carries Phase (SSHPhaseSessionStarted or SSHPhaseSessionEnded) plus ListenerEndpoint on session_started or TerminatedBy on session_ended, and no command fields. A command row carries Command (1 to 1024 bytes), with StartedAt on the row posted before the command runs and ExitCode, StartedAt and CompletedAt on the row posted after it exited. plexd never posts an empty Command: the control plane refuses a row that mixes the shapes.
type SSHActivity struct {
Command string `json:"command,omitempty"`
ExitCode *int `json:"exit_code,omitempty"`
StartedAt *time.Time `json:"started_at,omitempty"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
Phase string `json:"phase,omitempty"`
ListenerEndpoint string `json:"listener_endpoint,omitempty"`
TerminatedBy string `json:"terminated_by,omitempty"`
}terminated_by mapping
TerminatedByFromReason maps the internal session close reason to the wire terminated_by value reported on a session_ended row:
| Close reason | Wire value (terminated_by) | Produced when |
|---|---|---|
expired | ttl_expired | The capped expiry timer fired, the entry drained after it lapsed, or a session_revoked event carried revoke_reason ttl_expired |
idle | idle_timeout | The idle window elapsed with no byte flow, or a session_revoked event carried revoke_reason idle_timeout |
revoked | operator_revoke | A session_revoked event carried an operator-driven revoke_reason, or an empty or unrecognised one |
drained | plexd_close | The entry left the block while the session still had time on it |
shutdown | plexd_close | The node is going down |
The reasons are constants in internal/tunnel, not bare literals: they are the only input to this mapping, so producer and consumer share one vocabulary. operator_revoke is produced only for revoked, which only the session_revoked handler passes; a drain never produces it.
closeReasonForRevocation maps the revoke_reason of a verified session_revoked event (see SessionRevocation) to the close reason the handler passes to CloseSession:
revoke_reason | Constant | Close reason | Wire value (terminated_by) |
|---|---|---|---|
operator_request | api.RevokeReasonOperatorRequest | revoked | operator_revoke |
policy_violation | api.RevokeReasonPolicyViolation | revoked | operator_revoke |
identity_compromise | api.RevokeReasonIdentityCompromise | revoked | operator_revoke |
resource_decommissioned | api.RevokeReasonResourceDecommissioned | revoked | operator_revoke |
ttl_expired | api.RevokeReasonTTLExpired | expired | ttl_expired |
idle_timeout | api.RevokeReasonIdleTimeout | idle | idle_timeout |
| Empty or unrecognised | n/a | revoked | operator_revoke, with a Warn (session_revoked carries an unrecognised revoke_reason; reporting operator_revoke) |
The first four are operator-driven, supplied on the control plane's explicit revoke call. The last two are set by the control plane's sweeper when the session reaches its hard expiry or its idle timeout, so they keep the terminated_by the node's own timers would report. An unrecognised value still maps to revoked, because the close is still caused by the event.
The session_ended row is emitted by the SessionManager's on-closed callback for every close reason: TTL expiry, idle timeout, revocation (closed by a verified session_revoked event or by the drain), and node shutdown alike. The row is the only carrier of a session's byte counters and terminated_by, so skipping it on shutdown would leave the control plane's audit record for every live session truncated — the node going offline says the session ended, but not how much traffic it carried.
SessionRevocation
The payload of the session_revoked event: the revoked session, the revoked token's jti (equal to the session ID), and when and why the control plane revoked it. The contract names six RevokeReason values, the RevokeReason* constants (see terminated_by mapping).
type SessionRevocation struct {
SessionID string `json:"session_id"`
JTI string `json:"jti"`
RevokedAt time.Time `json:"revoked_at"`
RevokeReason string `json:"revoke_reason"`
}Integration Points
Reconciler (internal/reconcile)
Dispatcher.Handle is registered with RegisterDispatchHandler, alongside the actions dispatcher. Dispatch handlers run on every successful pull and before the empty-diff short-circuit, so a session is provisioned or torn down even when nothing else in the snapshot moved.
SSE Event Stream (internal/api)
The tunnel package provides one SSE handler, the one Dispatcher.SessionRevokedHandler returns, which plexd up registers for session_revoked in place of a plain reconcile trigger. Two event types relate to sessions:
| Event Type | Tier | Effect |
|---|---|---|
session_setup | contract | A plain reconcile trigger whose payload plexd does not parse; the resulting pull carries the session in its sessions block |
session_revoked | contract | SessionRevokedHandler closes the session the payload names, reporting the terminated_by its revoke_reason maps to, then triggers a reconcile |
The SSE stream verifies the envelope's signature and freshness before any handler runs. The session_revoked handler then:
- Decodes the payload as
api.SessionRevocation. A payload that does not decode, an emptysession_id, or ajtithat differs fromsession_idcloses nothing: the handler triggers a reconcile and returns the error, which the event dispatcher logs at Error (event handler failed). - Maps
revoke_reasonto a close reason withcloseReasonForRevocation(see terminated_by mapping). An empty or unrecognised value is logged at Warn (session_revoked carries an unrecognised revoke_reason; reporting operator_revoke) and still closes the session asrevoked. - Records the session ID in the dispatcher's
revokedset, with the close reason, so a pull rendered before the revocation committed cannot provision the session again (see Provision Pass). When such a pull is already binding the listener as the event lands, the handler's close finds no live session;Handlechecks the set again onceCreateSessionreturns and closes the listener it just bound with the event's reason, without posting itssession_startedrow (session was revoked by a signed event while its listener came up; closing it). - On its own goroutine, calls
SessionManager.CloseSessionwith that reason. The close blocks on the session's drain and on the boundedsession_endedpost, and the SSE stream dispatches its handlers one after another, so it does not run on the SSE goroutine. Thesession_endedrow goes out through the on-closed callback like that of any other close. The handler logs Infosession_revoked closed the session(session_id,revoke_reason,terminated_by), or Infosession_revoked names no live session; nothing to close(session_id,revoke_reason) when the drain or a local timer closed the session first, or it was never provisioned on this node. - Triggers the reconcile only after the close returned, so the pull it starts cannot drain the session first and report it as
plexd_close.
The event and the drain race. A pull that drains the entry before the event arrives closes the session as drained and reports plexd_close, and a session whose capped local expiry fires first reports ttl_expired; either way the late event finds no live session and only triggers a pull.
The envelope verifier keeps no nonce, so a captured session_revoked can be replayed. A replay can only name a session the control plane already revoked, which never becomes live again, so it finds nothing to close and only triggers a pull.
Neither event is required for correctness: a node that misses one converges on its own reconcile cadence, and the drain in the sessions block tears down a revoked session whose event never arrives. session_setup exists only to shorten the window; session_revoked also shortens it, and lets the node report why the session ended.
Control Plane API (internal/api)
The node agent reports session activity via a single endpoint on api.ControlPlane:
| Method | Endpoint | When Called |
|---|---|---|
ReportSessionActivity | POST /v1/nodes/{id}/sessions/{sid} | Listener ready (session_started), session close (session_ended), and around every ssh exec (the two command rows) |
WireGuard Mesh (internal/wireguard)
Tunnel listeners bind to the mesh IP assigned by the WireGuard interface. Connections arrive through the encrypted mesh — no ports are exposed on the public network. The meshIP parameter in NewSessionManager comes from registration.NodeIdentity.MeshIP, threaded through NewMeshServer(cfg, meshIP, hostKey, verifier, logger).
That bind address is the whole of a session listener's reachability boundary: the forward itself performs no authentication, accepting the first connection that arrives. The address is copied verbatim from the control plane's registration response into identity.json, so CreateSession parses it rather than trusting it and refuses to provision anything unless it is a bindable unicast address — on an empty, unparseable, or unspecified value (0.0.0.0, ::) net.Listen would bind every unicast address on the host and publish a forward to an internal service on the node's public and LAN interfaces.
Trust boundary
Reaching a session listener means being routable to the node's mesh IP, which every peer in the mesh is by construction. Nothing binds a forwarded connection to the operator the control plane authorized: the session JWT is carried in the block as an opaque value and is never presented on the forwarded connection. Treat a live session as reachable by any mesh peer for its duration.
Graceful Shutdown
Call SessionManager.Shutdown() on context cancellation to close all active sessions:
<-ctx.Done()
mgr.Shutdown()Access Flows
Mediated TCP Access Flow
- User requests access through the platform UI/CLI.
- Control plane verifies RBAC permissions and issues the session, adding a
tcpentry to the node'ssessionsblock. It may also emit asession_setupevent to pull the node's next reconcile forward. - On its next pull, plexd sees the entry, opens a TCP listener on the mesh interface, and reports a
session_startedrow carrying the boundlistener_endpoint. - Connections arriving on that listener are forwarded through the encrypted mesh to the target host and port.
- The session ends when the control plane drops the entry from the block (
plexd_close), when a verifiedsession_revokedevent names it (operator_revokefor an operator-drivenrevoke_reason,ttl_expiredoridle_timeoutfor the sweeper's), when its capped expiry fires (ttl_expired), when its idle window elapses (idle_timeout), or on node shutdown (plexd_close). Thesession_endedrow carries the byte counters and the reason.
Kubernetes Sessions
k8s entries are part of the sessions contract and plexd decodes them, but this agent provisions no listener for them: each is settled with a single unsupported session kind; no listener provisioned warning and produces no activity row, so there is nothing to tear down when the entry later drains. No kube-apiserver proxy contract exists yet. ssh entries get the same treatment on Windows.
SSH Sessions
On Linux and macOS, an ssh entry of the sessions block becomes an SSH-2 listener that a client logs in to with the session's token, as the ## SSH session listener section of plexsphere's agent contract specifies. Windows settles ssh entries as unsupported. A node owner refuses ssh sessions, and keeps serving tcp ones, with tunnel.ssh_sessions_enabled: false.
Listener
- One listener per session, on an ephemeral port of the mesh address, like a tcp session. The
session_startedrow carries it aslistener_endpoint. - It presents the node's SSH host key (
ssh_host_ed25519_keyin the data dir), whose fingerprint the capability manifest carries. Clients do not pin it (contract §d): the token authenticates the pair. - It identifies as
SSH-2.0-plexd. - A session holds at most 4 concurrent connections (a fifth is closed before its handshake) and each connection at most 10 open session channels (an eleventh is rejected with
resource shortage). A handshake has 30 s, and a connection 3 authentication attempts. - The mesh
SSHServer(tunnel.ssh_listen_addr) is a separate server that keeps servingdirect-tcpip.
Authentication
The client logs in as the entry's target.ssh.user, with the session token as password; password is the only method offered. VerifySessionToken checks the token against the keys the envelope verifier trusts at that moment: the current signing key (registration's, or the one a later state pull or signing_key_rotated event installed), plus the previous one during a rotation's grace window. A pulled key that differs from the current one replaces it and ends any grace window, because a node trusts exactly one key of its Domain. The checks run in this order:
- at most 8192 bytes of three raw-base64url segments whose header and claims are JSON objects
algisEdDSA;kidis not consulted, since the contract binds the key and not its id- the Ed25519 signature
kindandtarget.kindare bothsshjtiis the session idnbf <= now < exp, where a missingexpcounts as expired; there is no clock-skew leewaytarget.useris the login user
A refusal fails the handshake and is logged at Warn (ssh login refused). No log line and no error carries the token. iss and aud are not checked.
Requests
| Request | Answer |
|---|---|
session channel | Accepted; one process per channel |
Any other channel type (direct-tcpip, x11, auth-agent@openssh.com) | Rejected with administratively prohibited |
Global requests (tcpip-forward among them) | Refused |
pty-req | Allocates a pty sized from the request; terminal modes are ignored. Refused once a process started or a pty exists. A failed allocation answers false, and a later shell or exec runs without a pty |
window-change | Resizes the pty |
shell | Starts the user's login shell. Refused when the token lists allowed commands |
exec | Runs the command through the user's shell. Refused when it is empty, over 1024 bytes, or, under a non-empty allowed-command list, not byte-equal to one of its entries |
env, subsystem (sftp included), x11-req, auth-agent-req@openssh.com, signal, anything else | Refused |
The allowed-command list comes from the verified token's target claim, not from the pull entry, so the list plexd checks is the list the session helper enforces.
Commands and activity rows
The session posts two lifecycle rows, {"ssh":{"phase":"session_started","listener_endpoint":"…"}} when the listener is up and {"ssh":{"phase":"session_ended","terminated_by":"…"}} on close, and two command rows per exec. A shell posts no command rows.
An exec fails closed. plexd first posts the command row {"ssh":{"command":"…","started_at":"…"}} and runs the command only once the control plane accepted it. When the post fails, the client's stderr reads plexd: the control plane did not record this command; it was not run: <error> and the exit status is 1; a 409 session_already_revoked or session_expired also closes the connection. After the command exits plexd sends its exit-status and posts the command row again with exit_code, started_at and completed_at. A lost exit row is logged at Warn and leaves the client's exit status as it was. A launcher error reaches the client as a plexd: <error> stderr line with exit status 1.
Session helper
On Linux, plexd's unit bounds it to CAP_NET_ADMIN CAP_NET_RAW and hides /home, so a shell plexd forked could not become the login user. Every process therefore starts in the session helper, plexd session-helper:
- When
/run/plexd-session-helper.sockexists, plexd connects to it. Theplexd-session-helper.socketunit (Accept=yes) then starts one root instance ofplexd-session-helper@.serviceper connection, outside plexd's sandbox. A connection the socket refuses is an error: plexd never falls back to a child helper there, because that would run the shell inside the sandbox. - Without the socket, plexd starts its own binary as
plexd session-helper --child --trusted-key <key>…over a socketpair, trusting the keys plexd trusts. The child has plexd's own privileges, which is no boundary, so it pins nothing; hosts without a sandbox (OpenWrt, containers) need no extra unit. Under systemd plexd warns once that the socket is not listening, since shells then cannot switch users.
On macOS there is no helper socket. plexd always starts its own binary as plexd session-helper --child --trusted-key <key>… over a socketpair, trusting the keys plexd trusts. plexd runs as root under launchd with no sandbox, so the child switches users itself, and its token check is a consistency check, not a boundary. When plexd runs as the launchd daemon, the child's log lines go through the same writer as plexd's own, to /Library/Logs/plexd/plexd.log, so they follow a newsyslog rotation.
The protocol is one request per connection. plexd sends one JSON line of at most 16 KiB, with the process's file descriptors as SCM_RIGHTS: the pty slave, or stdin, stdout and stderr. The helper answers with one JSON line, {"exit_status":N} or {"error":"<reason>"}, and closes. plexd closing its end means "stop the process".
Whatever plexd checked, the helper:
- verifies the token again: the socket-activated helper against its own trust anchor, a child helper (always on macOS) against the keys plexd passed it;
- enforces the allowed-command list and the mode rules;
- resolves the user:
- on Linux from
/etc/passwd, with supplementary groups from/etc/group; NSS sources such as LDAP are not consulted, and an empty shell field means/bin/sh; - on macOS from the local directory node, with
dscl -plist . -read /Users/<name> RecordName UniqueID PrimaryGroupID NFSHomeDirectory UserShell. A name that is empty,.,..or contains/is refused beforedsclruns.dsclalso finds a record by any of its names and in any case, so the name must be the record's firstRecordNameexactly:ROOT, orwwwfor_www, is refused as not existing. An account absent from the local node, such as an Active Directory or Open Directory network account, is refused as not existing. A missingUserShellmeans/bin/sh. Supplementary groups come fromgetgrouplist, and at most the first 16 apply (macOS'sNGROUPS_MAX);
- on Linux from
- starts a login shell, or
<shell> -c <command>, as that user in a new session and process group, in the home directory, with exactlyHOME,USER,LOGNAME,SHELLandPATHin the environment, plusTERMwith a pty, which it first hands to that user, as sshd does; - sends
SIGHUPto the process group when plexd hangs up, andSIGKILL2 s later; - once the process exits, kills what is left of its group and answers with the exit code, or 128 plus the signal for a death by signal.
No PAM session, no utmp, wtmp or lastlog record and no resource limit apply: plexd links no libpam.
Trust anchor
This section applies to Linux's socket-activated helper. On macOS the child helper trusts the keys plexd passes and pins nothing, and tunnel.session_signing_public_key is not read.
The socket-activated helper verifies tokens against one key:
tunnel.session_signing_public_keyin/etc/plexd/config.yaml, when set; the pin file is then neither read nor written;- otherwise the key pinned at
/etc/plexd/session-signing-key. On its first run the helper copiessigning_public_keyfromidentity.jsonthere (mode 0644) and logspinned the session signing keywith the key's SHA-256; later runs trust only that file.
Both units make /etc read-only for plexd, so a compromised plexd cannot replace the anchor, and without a token the control plane signed it cannot start a shell. A malformed pin fails every login until it is fixed. The helper does not follow signing-key rotations: once the Domain key rotates, it refuses every token until an operator sets tunnel.session_signing_public_key to the new key or deletes the pin. plexd rewrites identity.json's signing key whenever a state pull carries a new one, so a helper that has not pinned yet pins the key plexd last persisted, and deleting the pin makes the next run pin that key. The same holds once the node re-registers with another Domain: plexd deregister leaves the pin in place, so set the new Domain's key there as well.
The trust model does not move: plexd could always rewrite its own identity.json, and the boundary is the pin in read-only /etc. plexd logs each key a state pull installs at Info as installed signing key from state pull, with key_id and public_key; that public_key is the value to set in tunnel.session_signing_public_key.
Nodes enrolled before the per-Domain signing key
A node enrolled before plexsphere/plexsphere#918 stored a static signing key that no signer holds, and a helper that ran before plexd persisted the Domain key pinned that key. plexd learns the Domain key from the state pull and accepts the login, but the helper refuses every shell and command: the client's stderr reads plexd: tunnel: session helper refused: token refused: tunnel: session token: signature does not verify against a trusted key. Set tunnel.session_signing_public_key to the public_key of plexd's installed signing key from state pull log line, or delete /etc/plexd/session-signing-key so that the next run pins the key plexd persisted. Delete the pin only once plexd has persisted the key, that is, once it no longer logs signing key not persisted; retrying on the next pull.
Lifetime
An ssh session ends like a tcp one: when its entry drains (plexd_close), when a verified session_revoked event names it (operator_revoke for an operator-driven revoke_reason, ttl_expired or idle_timeout for the sweeper's), when its capped expiry fires (ttl_expired), when its idle window elapses (idle_timeout), or on shutdown (plexd_close). Any byte on any of its connections is activity. The close ends every connection within the drain bound and stops each running process through its helper; under socket activation systemd also kills the instance's cgroup.
The helper cannot see the sessions block, so it cannot honour a revocation: a compromised plexd can reuse a token a client presented until that token's exp. The listener itself closes on the drain, or earlier on a verified session_revoked event.
Deployment notes
- A Linux host installed before the helper units keeps its old
plexd.serviceuntilplexd installruns again;service.upgradedoes not rewrite units. Until then plexd uses the child helper and warns, and every login as a user other than root fails.plexd installwrites the units and reloads systemd but neither enables nor starts the socket, so runplexd install && systemctl restart plexd: plexd pulls the socket in throughWants=when it starts. - Nodes without a usable shell, such as the distroless Kubernetes DaemonSet and container images, should set
tunnel.ssh_sessions_enabled: false: their listeners would come up, and every launch would fail. - On macOS the pf policy governs mesh traffic addressed to the node itself and ends in a default deny (see What the rules govern). A session listener, ssh or tcp, is therefore reachable only where the node's compiled policy allows inbound mesh TCP to the node. Otherwise the
session_startedrow is posted, but clients time out. plexd opens no port itself. - macOS privacy protection (TCC) may keep processes plexd starts from reading protected folders such as Desktop, Documents and Downloads, unless plexd has Full Disk Access. Grant it in System Settings or through an MDM privacy profile.
- On macOS an
execruns withPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, so Homebrew's/opt/homebrew/binis not on it. A login shell rebuildsPATHthrough/etc/zprofile(path_helper). - A Mac upgraded to this release serves ssh sessions unless it sets
tunnel.ssh_sessions_enabled: false.
Logging
All log entries use component=tunnel. Session-scoped entries add session_id.
| Level | Event | Keys |
|---|---|---|
Info | Session started | listen_addr, target |
Info | Session created | session_id, kind, listen_addr, expires_at |
Info | Session closed | session_id, kind, reason, duration |
Info | ssh session started | listen_addr, user |
Info | ssh login accepted | remote_addr, user |
Info | ssh request refused | request, reason |
Info | ssh process exited | mode, exit_status (a number, never a signal name) |
Info | ssh session revoked or expired server-side; closing the connection | — |
Info | ssh session closed | duration |
Warn | ssh login refused | remote_addr, user, error |
Warn | ssh pty allocation failed | error |
Warn | ssh command not recorded by the control plane; refusing to run it | command, error |
Warn | ssh command exit not recorded | command, exit_code, error |
Warn | ssh session launch failed | mode, error |
Warn | ssh sessions are disabled on this node; no listener provisioned | session_id |
Warn | plexd-session-helper.socket is not listening; ssh sessions run inside plexd's own sandbox and cannot switch users | socket |
Warn | Timed out waiting for ssh handlers to drain | — |
Debug | ssh connection refused: session connection limit reached | remote_addr |
Debug | ssh handshake failed | remote_addr, error |
Debug | ssh request not served | request |
Info | All tunnel sessions closed | — |
Info | Session expired; waiting for the control plane to drain the entry | session_id, kind, expires_at |
Warn | Session carries no expires_at; refusing to provision | session_id, kind |
Warn | Unsupported session kind; no listener provisioned | session_id, kind |
Warn | Session target does not match kind | session_id, kind |
Warn | Tunneling is disabled; no listener provisioned | session_id |
Warn | Session listener setup failed | session_id, error |
Info | Session was revoked by a signed event; not provisioning it | session_id |
Info | Session was revoked by a signed event while its listener came up; closing it | session_id |
Info | session_revoked closed the session | session_id, revoke_reason, terminated_by |
Info | session_revoked names no live session; nothing to close | session_id, revoke_reason |
Warn | session_revoked carries an unrecognised revoke_reason; reporting operator_revoke | session_id, revoke_reason |
Debug | Connection rejected (duplicate) | — |
Debug | Session not found for close | session_id |
Error | Session entry carries no session_id; dropping | kind |
Error | Failed to dial target | target, error |
The session helper logs to its own unit's journal (journalctl -u 'plexd-session-helper@*') under socket activation. In child mode it logs to plexd's stderr, which on macOS is the daemon log file (/Library/Logs/plexd/plexd.log) when plexd runs as the launchd daemon. Its lines are text: session helper started process (session_id, user, mode, command, pid), session helper process exited (session_id, exit_status) and pinned the session signing key (path, sha256) at Info, and session helper refused a request (session_id, reason) at Warn.