Systemd E2E Test

Validates that plexd runs correctly as a systemd service by deploying it inside a privileged Ubuntu container with systemd as PID 1. The test installs the plexd binary and the production plexd.service unit file, starts the service via systemctl, polls the mock-api assertion endpoint to verify registration, heartbeat, state retrieval, capabilities reporting, state convergence, and observability forwarding (metrics, logs, audit), then verifies clean shutdown.

Container Topology

┌──────────────┐       ┌─────────────────────────┐
│   mock-api   │◄──────│   systemd container     │
│  :8080 (HTTP)│       │  (Ubuntu 24.04)         │
│  :8443 (TLS) │       │  systemd PID 1          │
│  /v1/health  │       │  plexd.service managed   │
└──────────────┘       └─────────────────────────┘
        │                          │
        └──── plexd-systemd-e2e ───┘
              (Docker network)

Container	Image	Source	Purpose
plexd-e2e-mockapi	mockapi:e2e-systemd	test/e2e/mockapi/Dockerfile	Fixture-based mock Central API, tracks call counters
plexd-e2e-systemd	plexd-systemd:e2e	test/e2e/systemd/Dockerfile	Ubuntu 24.04 with systemd as PID 1, runs plexd as a real service

Test Phases

1. Pre-flight checks

Verifies that docker, curl, and jq are available on $PATH. Exits immediately if any tool is missing.

2. Pre-cleanup

Removes any leftover containers and network from a previous run (handles the case where a prior run was killed with SIGKILL and the trap did not fire).

3. Image build

Builds the mock-api image, the systemd container image, and cross-compiles the plexd binary using a Go Alpine container. The Go version is extracted from go.mod.

4. Network and container startup

Creates a Docker bridge network and starts both containers on it. The systemd container runs with --privileged, --cgroupns=host, and cgroup volume mounts for full systemd functionality.

5. Service installation

Copies the plexd binary, the production deploy/systemd/plexd.service unit file and the two session helper units (plexd-session-helper.socket, plexd-session-helper@.service) into the systemd container. Writes config.yaml pointing at the mock-api container and the bootstrap token to /etc/plexd/bootstrap-token (mode 0600), before the service starts so the unit can grant write access to it, and creates /etc/plexd/hooks as plexd install would. Runs systemctl daemon-reload and systemctl enable --now plexd, then checks that systemctl is-active plexd-session-helper.socket prints active: plexd.service wants the socket and names it in Also=.

6. Assertion polling

Polls GET http://localhost:18080/test/assertions every 3 seconds for up to 60 seconds (configurable via TIMEOUT).

7. Request body validation

Uses GET /test/last-request/{endpoint} to verify the content of request payloads:

Endpoint	Validated Fields
register	token (non-empty), hostname (non-empty)
heartbeat	Valid JSON with timestamp field
capabilities	binary_version (non-empty), binary_checksum (32-byte base64 digest), eleven builtin_actions; binary and plexd_hooks absent

8. Spent token cleanup

The unit keeps /etc read-only and grants write access to /etc/plexd/bootstrap-token alone, so plexd cannot unlink the token it spent and empties the file instead. The phase fails unless stat -c %s /etc/plexd/bootstrap-token prints 0, journalctl -u plexd contains token file emptied, it could not be removed, and the journal contains neither failed to delete token file nor hook watcher stopped.

9. Periodic loop verification

Waits up to 60 seconds for heartbeat_count, metrics_count, and logs_count to reach >= 2, proving that periodic loops run continuously. Logs work via journald in the systemd container. Audit is tested separately via restart.

10. Audit forwarding via service restart

The ProcessSource uses sync.Once to emit a single process_start audit entry per process lifetime. The test restarts the plexd service via systemctl restart plexd, creating a new process with a fresh ProcessSource, and verifies audit_count increases.

11. Local endpoint delivery

Polls GET /test/assertions until local_metrics_count, local_logs_count, and local_audit_count are all >= 1 (timeout: 60s). Validates that the local endpoint credential chain works with systemd: NSK from registration → secret fetch → AES-256-GCM decryption → Bearer token → HTTPS POST to plexd-e2e-mockapi:8443.

12. Mediated ssh session

Drives a mediated ssh session through the sessions block, the socket-activated session helper and a real OpenSSH client. The listener binds the mesh address the mock assigns, 10.99.0.1, so the client runs inside the container (docker exec … sshpass -e ssh), and the phase relies on WireGuard coming up in the privileged container. The image carries openssh-client, sshpass and an unprivileged user e2e. Each login uses -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o PreferredAuthentications=password -o PubkeyAuthentication=no -o NumberOfPasswordPrompts=1; the last option makes a refused token end ssh with 255 instead of sshpass giving up on a second prompt. Tokens come from POST /test/session-token.

1. Configures an ssh entry for root and waits up to 60 s for a session_started row whose listener_endpoint starts with 10.99.0.1:; without one it prints journalctl -u plexd and fails.
2. Runs id -un as root and expects root, then a last session activity body with ssh.command == "id -un" and ssh.exit_code == 0.
3. Checks that /etc/plexd/session-signing-key has mode 644 and holds signing_public_key from identity.json, the key the helper pinned on its first run.
4. Adds an entry for e2e, waits for its own session_started row, and expects id -un to print e2e, which proves the helper switched uid.
5. Expects exit 3 to exit 3, and a token minted for another session id to end with 255 and Permission denied.
6. Drains the block, waits up to 60 s for session_activity_count to rise by 2, and requires session_activity_rejected_count unchanged: both session_ended rows post for sessions the mock already holds as revoked, and the mock, like the control plane, records the ending row of such a session once, so both are accepted, as the tcp drain is in the docker suite. The last body is an ssh session_ended row with terminated_by plexd_close.

13. Start after a reboot

A reboot empties /run, which takes /run/plexd with it. The phase stops plexd, removes /run/plexd and starts plexd again, then waits up to 30 s for /var/run/plexd/api.sock and expects systemctl is-active plexd to print active. Every ReadWritePaths entry of the unit is optional, so systemd skips the missing directory instead of failing with 226/NAMESPACE, and plexd recreates it for the node API socket.

14. Shutdown verification

Stops the service with systemctl stop plexd, verifies inactive state and exit code 0, checks journalctl -u plexd for absence of crash indicators (core dumped, segfault, SIGABRT, SIGKILL), and verifies the shutdown message was logged.

15. Cleanup

The cleanup function runs on EXIT trap (both success and failure). It prints diagnostics on failure, then removes both containers and the Docker network.

Assertion Logic

The test polls GET http://localhost:18080/test/assertions which returns JSON counters:

{
  "registration_count": 1,
  "heartbeat_count": 1,
  "state_count": 1,
  "capabilities_count": 1,
  "metrics_count": 1,
  "logs_count": 1,
  "audit_count": 1,
  "local_metrics_count": 1,
  "local_logs_count": 1,
  "local_audit_count": 1
}

The test passes when all seven platform counters are >= 1 (initial assertions), and separately verifies that all three local endpoint counters are >= 1 (Phase 11):

Counter	Meaning
registration_count	plexd called POST /v1/register
heartbeat_count	plexd called POST /v1/nodes/{id}/heartbeat
state_count	plexd called GET /v1/nodes/{id}/state
capabilities_count	plexd called PUT /v1/nodes/{id}/capabilities
metrics_count	plexd called POST /v1/nodes/{id}/metrics
logs_count	plexd called POST /v1/nodes/{id}/logs
audit_count	plexd called POST /v1/nodes/{id}/audit
local_metrics_count	plexd sent metrics to POST /local/metrics (TLS)
local_logs_count	plexd sent logs to POST /local/logs (TLS)
local_audit_count	plexd sent audit to POST /local/audit (TLS)

plexd Configuration

The config is written directly into the container (not via the installer script):

api:
  base_url: http://plexd-e2e-mockapi:8080
registration:
  data_dir: /var/lib/plexd
node_api:
  data_dir: /var/lib/plexd
heartbeat:
  node_id: e2e-systemd-node
metrics:
  enabled: true
  collect_interval: 5s
  report_interval: 10s
  local_endpoint:
    url: https://plexd-e2e-mockapi:8443/local/metrics
    secret_key: local-bearer-token
    tls_insecure_skip_verify: true
log_fwd:
  enabled: true
  collect_interval: 5s
  report_interval: 10s
  file_patterns:
    - "/var/log/plexd/*.log"
  local_endpoint:
    url: https://plexd-e2e-mockapi:8443/local/logs
    secret_key: local-bearer-token
    tls_insecure_skip_verify: true
audit_fwd:
  enabled: true
  collect_interval: 5s
  report_interval: 10s
  local_endpoint:
    url: https://plexd-e2e-mockapi:8443/local/audit
    secret_key: local-bearer-token
    tls_insecure_skip_verify: true

The bootstrap token is written to /etc/plexd/bootstrap-token (mode 0600) before the service starts; no environment file is written.

Configuration Variables

Variable	Default	Description
TIMEOUT	60	Assertion polling timeout in seconds

Usage

make test-e2e-systemd

Or directly:

bash test/e2e/systemd/test.sh

Override configuration:

TIMEOUT=120 make test-e2e-systemd

Prerequisites

• Docker
• curl and jq on the host

Debugging Failures

systemd does not boot inside the container:

The container requires --privileged, --cgroupns=host, and cgroup volume mounts. Verify the host supports cgroup v2:

mount | grep cgroup2

plexd service fails to start:

docker exec plexd-e2e-systemd journalctl -u plexd --no-pager
docker exec plexd-e2e-systemd systemctl status plexd

Common causes:

• Binary not executable — check that chmod +x was applied to /usr/local/bin/plexd.
• Config YAML parse error — field names must use snake_case YAML tags (e.g. base_url, not baseurl).
• Missing bootstrap token — /etc/plexd/bootstrap-token must hold the token before the service starts.

Assertions not met (counters stay at 0):

docker logs plexd-e2e-mockapi
docker exec plexd-e2e-systemd journalctl -u plexd --no-pager -n 100

Common causes:

• Network connectivity — both containers must be on the plexd-systemd-e2e bridge network.
• mock-api not ready — the script polls /v1/health for readiness but the health timeout may be too short.

Leftover containers from previous run:

The script performs pre-cleanup, but if issues persist:

docker rm -f plexd-e2e-mockapi plexd-e2e-systemd
docker network rm plexd-systemd-e2e

Diagnostics Output

On any failure, the print_diagnostics function outputs:

Command	Purpose
docker network inspect plexd-systemd-e2e	Network state and connected containers
docker logs plexd-e2e-mockapi	Mock API server logs
systemctl status plexd	Service state and recent log lines
journalctl -u plexd -n 100	Recent plexd service logs
journalctl -u 'plexd-session-helper@*' -n 100	Recent session helper logs
journalctl -n 50	Recent system-wide journal entries
ps aux	Process list inside systemd container
ls -la /etc/plexd/	Configuration files
ls -la /usr/local/bin/plexd	Binary presence and permissions

Key Files

File	Purpose
test/e2e/systemd/Dockerfile	Ubuntu 24.04 systemd container image
test/e2e/systemd/test.sh	Orchestration script (build, install, assert, shutdown, cleanup)
test/e2e/systemd/.dockerignore	Prevents unnecessary files in Docker context
test/e2e/mockapi/Dockerfile	Mock API image
test/e2e/mockapi/mockapi.go	Mock API server with /test/assertions endpoint
deploy/systemd/plexd.service	Production unit file (copied verbatim into container)
deploy/systemd/plexd-session-helper.socket, deploy/systemd/plexd-session-helper@.service	Session helper units (copied verbatim into container)
Makefile	test-e2e-systemd target

See also

• Docker E2E Test — Docker Compose-based E2E test
• Kubernetes E2E Test — kind-based Kubernetes E2E test