Heartbeat Service

The internal/agent package implements the HeartbeatService, which sends periodic heartbeat requests to the control plane and processes directive flags from the response.

Config

Field	Type	Default	Description
Interval	time.Duration	30s	Heartbeat send interval
NodeID	string	—	Node identifier (required)

Heartbeat Loop

HeartbeatService.Run(ctx) operates as follows:

1. Send one heartbeat immediately on start
2. Start a ticker at the configured interval (default 30s)
3. On each tick, build and send a HeartbeatRequest
4. Process the HeartbeatResponse directive flags
5. Continue until context is cancelled

Run() always returns nil.

Request Payload

The heartbeat request is built by an optional buildRequest function. If not set, a zero-valued HeartbeatRequest is sent. The builder typically collects runtime state:

type HeartbeatRequest struct {
    ClientNow      time.Time      `json:"client_now"`
    BinaryChecksum string         `json:"binary_checksum"`
    BinaryVersion  string         `json:"binary_version"`
    NATSummary     map[string]any `json:"nat_summary"`
}

NATSummary is always a non-nil map so it marshals as a JSON object: {} before NAT discovery has a result, otherwise {"endpoint", "nat_type"}. A null value is rejected by the control plane.

Response Handling

The control plane returns a HeartbeatResponse with directive flags:

type HeartbeatResponse struct {
    AcceptedAt time.Time `json:"accepted_at"`
    Reconcile  bool      `json:"reconcile"`
    RotateKeys bool      `json:"rotate_keys"`
}

On acceptance, accepted_at is logged at debug level alongside the local send time so operators can estimate the clock skew between the node and the control plane.

Flag	Action
reconcile	Call ReconcileTrigger.TriggerReconcile()
rotate_keys	Call the onRotateKeys callback

Error Handling

401 Unauthorized

When the heartbeat receives a 401 error (api.ErrUnauthorized), the onAuthFailure callback is invoked. In production (plexd up), this triggers re-registration:

1. Call registrar.Register(ctx) to obtain a new identity
2. Update the control plane client auth token via client.SetAuthToken()
3. Log the re-registration result

If re-registration fails, the error is logged and the heartbeat continues retrying on the next tick.

Clock Skew

When the heartbeat is rejected with the clock_skew problem code — the control plane found client_now more than 60s from its own time — the service logs at error level, instructing the operator to synchronize the system clock via NTP. There is no extra retry; the next attempt is the next 30s tick.

Other Errors

Non-401 errors are logged at error level. The heartbeat loop continues on the next tick interval.

Callbacks

Method	Signature	Description
SetReconcileTrigger	ReconcileTrigger	Reconciler to trigger on reconcile=true
SetOnAuthFailure	func()	Called on 401 Unauthorized
SetOnRotateKeys	func()	Called on rotate_keys=true
SetBuildRequest	func() HeartbeatRequest	Custom request builder

Integration Wiring

In plexd up, the heartbeat service is wired as follows:

HeartbeatService
├── client: ControlPlane (sends heartbeat RPCs)
├── reconcileTrigger: Reconciler (triggers state reconciliation)
├── onAuthFailure: re-registers → updates auth token
└── onRotateKeys: triggers reconcile (key material itself arrives via the signing_key_rotated SSE event)

Interfaces

type HeartbeatClient interface {
    Heartbeat(ctx context.Context, nodeID string, req HeartbeatRequest) (*HeartbeatResponse, error)
}

type ReconcileTrigger interface {
    TriggerReconcile()
}

Both interfaces are small and testable. The HeartbeatClient is satisfied by *api.ControlPlane, and ReconcileTrigger is satisfied by *reconcile.Reconciler.