Agent Deployment Guide¶

Installation, Configuration, and Operation of MazeVault Agents

Document Version: 1.1.0
Last Updated: 2026-10-26

1. Overview¶

The MazeVault Agent is a lightweight Go binary deployed on infrastructure nodes. It provides:

Secret synchronization — Automatic delivery of secrets to local applications
Certificate discovery — Scanning infrastructure for certificates (filesystem, JKS, Windows cert store)
Certificate lifecycle — Automated renewal and deployment of managed certificates
Real-time updates — WebSocket connection for instant push notifications
Local proxy — Secure proxy for application access to secrets without direct API calls
Offline resilience — Local caching ensures operation during network interruptions

2. Supported Platforms¶

Platform	Architecture	Notes
Linux	x86_64, ARM64	Ubuntu, RHEL, Rocky Linux, Debian, Alpine
Windows	x86_64	Windows Server 2019+
Kubernetes	—	No agent container image; run the agent on nodes/VMs (see 3.3)

3. Installation¶

3.1 Linux — Installation Methods¶

Method	Use it for
Install script (Deployment → Deploy)	Single hosts; downloads and verifies the agent, installs the service and enrolls the host
Packages `.deb` / `.rpm` (section 3.6)	Fleets managed with your own tooling (Ansible, Puppet, images)
Manual installation (section 3.2)	Only where neither of the above can be used

All methods install the same signed binary with the layout of section 3.2.1.

3.2 Linux — Manual Installation¶

Use the layout of section 3.2.1: the binary must stay in /opt/mazevault-agent/bin, which the agent user can write for self-update. The manual download is not signature-verified; the install script and the packages verify it.

# Service user and directories
sudo useradd --system --no-create-home --shell /usr/sbin/nologin mazevault-agent
sudo install -d -o mazevault-agent -g mazevault-agent -m 0750 \
  /opt/mazevault-agent/bin /etc/mazevault /etc/mazevault/certs /var/lib/mazevault /var/log/mazevault

# Agent binary (linux-arm64 on ARM)
curl -fsSL https://vault.example.com/api/v1/agents/download/linux-amd64 -o mazevault-agent
sudo install -o mazevault-agent -g mazevault-agent -m 0755 mazevault-agent /opt/mazevault-agent/bin/mazevault-agent

Create /etc/systemd/system/mazevault-agent.service (the same unit as the install script and the packages):

[Unit]
Description=MazeVault Certificate Management Agent
Documentation=https://mazevault.com
After=network-online.target
Wants=network-online.target
# A package install without configuration stays idle until agent.yaml or an enrollment file exists.
ConditionPathExists=|/etc/mazevault/agent.yaml
ConditionPathExists=|/etc/mazevault/enroll.conf

[Service]
Type=simple
User=mazevault-agent
Group=mazevault-agent
ExecStart=/opt/mazevault-agent/bin/mazevault-agent -config /etc/mazevault/agent.yaml
Restart=always
RestartSec=5s
# Exit code 78 is an invalid configuration: a restart cannot fix it.
RestartPreventExitStatus=78
LimitNOFILE=65536

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/opt/mazevault-agent/bin /var/lib/mazevault /var/log/mazevault /etc/mazevault
PrivateTmp=true

Environment=MAZEVAULT_LOG_LEVEL=info

[Install]
WantedBy=multi-user.target

Then enroll the host (section 3.5); enroll writes agent.yaml and starts the service:

sudo systemctl daemon-reload
sudo systemctl enable mazevault-agent
sudo /opt/mazevault-agent/bin/mazevault-agent enroll --server https://vault.example.com --token mvbt_xxxxxxxxxxxxx

3.2.1 Linux — Layout Installed by the Deployment Script¶

The install script generated under Deployment → Deploy (and the rpm/deb package) uses this layout. It is designed for hardened hosts (CIS/STIG baselines) where /var and /tmp are mounted noexec and SELinux is enforcing.

Path	Purpose	Owner / mode
`/opt/mazevault-agent/bin/mazevault-agent`	Agent binary. Outside `/var` (`noexec`), default SELinux label `bin_t`. The directory is writable by the agent user because self-update stages the new binary next to the running one and the remote-uninstall runner executes a copy from it	`mazevault-agent`, `0750` dir / `0755` file
`/etc/mazevault/agent.yaml`	Configuration and state (`agent.crt`, `agent.key`)	`mazevault-agent`, `0640`
`/etc/mazevault/certs`	Deployed certificates (`cert_store_path`)	`mazevault-agent`, `0750`
`/var/log/mazevault`	Logs	`mazevault-agent`, `0750`
`/etc/systemd/system/mazevault-agent.service`	Unit: `ProtectSystem=strict`, `ReadWritePaths=/opt/mazevault-agent/bin /var/lib/mazevault /var/log/mazevault /etc/mazevault`	`root`, `0644`

The script is idempotent and safe to re-run: it verifies the host before enabling the service and migrates hosts installed with the previous layout (binary under /var/lib/mazevault/bin, ExecStart drop-in overrides).

Step	Behaviour
Preflight	Required tools present; `/opt/mazevault-agent/bin` not on a `noexec` mount; SELinux mode detected
Download	Certificate validation is never disabled. With a MazeVault CA (`MAZEVAULT_CA_FILE` on the server) the download uses it and falls back to the system trust store; without one, the system trust store is used and no `ca.crt` is written (`server_ca_file` stays empty)
Verification	ELF check and size check; the cosign signature the server sends (`X-Signature`, `X-Signature-Key-Id`) is verified against the signing keys embedded in the script and the `X-SHA256` is compared with the download, so an unsigned, tampered or truncated transfer is rejected. A failed download (including a transfer that broke off after HTTP 200) prints the HTTP status and the server's response body
Exec probe	The new binary is staged as `mazevault-agent.new` next to the final path and run as `mazevault-agent` (`mazevault-agent version`) before it replaces the installed binary and before the unit is written — a `noexec` mount, an SELinux denial or a binary built for a newer glibc stops the installer, leaves a previously installed agent untouched and never puts systemd into a restart loop
SELinux	When enforcing and `semanage` is available, `/opt/mazevault-agent/bin` is labelled `bin_t` explicitly
Start	`systemctl enable` + `restart`; the final status and the last journal lines are printed with tokens and UUIDs redacted

Exit codes: 2 preflight failed, 3 download or binary verification failed, 4 the service did not start.

Hardened hosts

Do not move the binary to a root-owned directory such as /usr/local/bin: self-update needs a directory writable by the agent user and listed in ReadWritePaths.
Agents before 1.0.61 were dynamically linked and required glibc ≥ 2.34; they do not run on RHEL/Rocky/AlmaLinux 8 (GLIBC_2.34 not found). From 1.0.61 the agent is statically linked.
Remote uninstall (Deployment → Agents) requires the agent to run as root; script-installed agents run as mazevault-agent, so use the manual steps in section 9, Agent Lifecycle.

3.3 Kubernetes¶

MazeVault does not publish an agent container image, and the sidecar agent mode is not supported. To manage certificates used by Kubernetes workloads, run the agent on the nodes or VMs that hold them (section 3.1), or use the MazeVault integrations for Kubernetes secrets. The agent binary always comes from your own MazeVault backend (/api/v1/agents/download/...), which serves signed binaries from the MazeVault Harbor.

3.4 Windows — Service¶

On Windows the agent runs as the Windows service MazeVaultAgent (automatic start, LocalSystem, automatic restart on failure). The install script generated under Deployment → Deploy uses this layout:

Path	Purpose
`C:\Program Files\MazeVault Agent\mazevault-agent.exe`	Agent binary
`C:\ProgramData\MazeVault\`	Configuration (`agent.yaml`, `ca.crt`), certificates and state
`C:\ProgramData\MazeVault\logs\agent.log`, `audit.log`	Logs

Run the script from an elevated PowerShell. It is safe to re-run: running it again upgrades the agent in place, and an installation from the earlier script layout (binary in ProgramData) is migrated. On a host where the agent is installed from the MSI package the script refuses to run.

The script and mazevault-agent enroll restrict C:\ProgramData\MazeVault\ to SYSTEM and Administrators (inheritance from ProgramData is removed), because it holds the bootstrap token, the agent's private key and the pre-seed file. Re-running the script on an existing host applies this to the files already there.

The agent binary also has the install and uninstall subcommands, which register and remove the service; run them from an elevated prompt.

Earlier versions

The install script of earlier versions failed with an unknown install command and the agent could not run as a Windows service. Re-run the new script on such hosts.

3.5 Enrolling an Agent (`enroll`)¶

mazevault-agent enroll registers a host with the server in one step:

sudo /opt/mazevault-agent/bin/mazevault-agent enroll --server https://vault.example.com --token mvbt_xxxxxxxxxxxxx --ca-file ./ca.crt

The full path is needed because /opt/mazevault-agent/bin is not on the sudo search path. On Windows run the command from an elevated (Run as administrator) prompt:

& "C:\Program Files\MazeVault Agent\mazevault-agent.exe" enroll --server https://vault.example.com --token mvbt_xxxxxxxxxxxxx --ca-file .\ca.crt

On Windows the service must already be registered (by the MSI or by mazevault-agent.exe install); otherwise add --no-start and register the service afterwards.

Linux packages

The .deb/.rpm packages (section 3.6) do not create agent.yaml, so enroll and the pre-seed file work right after the package install. Packages of earlier versions created a placeholder agent.yaml; on such a host run enroll with --force.

The command writes agent.yaml (mode 0640) with the platform default paths, copies the CA file to the configuration directory as ca.crt when --ca-file is given (otherwise the system trust store is used), and then enables and restarts the service.

Option	Description
`-config <path>`	Global option placed before `enroll`: path of the `agent.yaml` to write
`--server <url>`	MazeVault server URL; must be `https`
`--token <token>`	Bootstrap (deployment) token
`--ca-file <path>`	CA certificate of a private CA that signed the server certificate
`--force`	Overwrite an existing installation (see below)
`--no-start`	Write the configuration only; do not enable or restart the service

enroll refuses to run when agent.yaml already exists. With --force it deletes the configuration, the client certificate and key, and the agent state, so the agent enrolls again with a new identity (certificate, key and recovery token). The server keeps the same agent record when the hostname and the machine ID match. --force does not delete ca.crt or the pre-seed file.

Pre-seed file for automated installs¶

For unattended installs (images, configuration management) place an INI file next to the agent instead of running enroll:

Platform	Path
Linux	`/etc/mazevault/enroll.conf`
Windows	`C:\ProgramData\MazeVault\enroll.ini`

[enroll]
server_url = https://vault.example.com
bootstrap_token = mvbt_xxxxxxxxxxxxx
server_ca_file = /etc/mazevault/ca.crt

server_ca_file is optional. Save the file as ASCII or UTF-8 (a byte order mark is accepted); UTF-16, the default of some Windows editors and of PowerShell Out-File in Windows PowerShell 5.1, is not supported. The agent reads the file at start only when agent.yaml does not exist, turns it into agent.yaml and deletes it; next to an existing agent.yaml the file is deleted at start without being read (it would keep a token on disk). An invalid or incomplete file is kept and the agent stops with a configuration error (exit code 78 on Linux; the Windows service stops cleanly and is not restarted).

3.6 Installing from Packages (`.deb`, `.rpm`, `.msi`)¶

For installation with your own tooling (Ansible, Intune, SCCM, GPO, dpkg -i) download the packages under Deployment → Deploy → Install from packages (or GET /api/v1/agent-packages, permission deployment:read). Every package contains the same signed agent binary as the install script; MazeVault verifies the package signature before offering it. The packages contain no server address and no token, so one package serves all hosts. The panel offers the latest agent version (or says that it shows the bundled one when the latest has no packages available) and warns when the version policy (early or stable channel: target or ceiling) would switch agents installed from it to another version right after enrollment; review the policy before a rollout.

Package	Platforms
`mazevault-agent_<version>_amd64.deb`, `_arm64.deb`	Debian, Ubuntu
`mazevault-agent-<version>.x86_64.rpm`, `.aarch64.rpm`	RHEL, Rocky Linux, AlmaLinux, SUSE
`mazevault-agent-<version>-x64.msi`	Windows Server 2019+ (x64). Windows on ARM: use the install script

Linux:

sudo apt-get install ./mazevault-agent_1.4.0_amd64.deb        # or: sudo dnf install ./mazevault-agent-1.4.0.x86_64.rpm
sudo /opt/mazevault-agent/bin/mazevault-agent enroll --server https://vault.example.com --token mvbt_xxxxxxxxxxxxx

The package creates the user mazevault-agent and the layout of section 3.2.1 (the unit goes to /usr/lib/systemd/system/mazevault-agent.service). Without agent.yaml or a pre-seed file the service is enabled but stays idle. When /etc/mazevault/enroll.conf exists before the package is installed, the install starts the agent and it enrolls by itself; a pre-seed file placed after the install is read at the next start, so run sudo systemctl start mazevault-agent (or reboot).

Windows (silent install):

msiexec /i mazevault-agent-1.4.0-x64.msi /qn SERVER_URL=https://vault.example.com BOOTSTRAP_TOKEN=mvbt_xxxxxxxxxxxxx

MSI property	Meaning
`SERVER_URL`	MazeVault server URL (`https`)
`BOOTSTRAP_TOKEN`	Deployment token; hidden from the MSI log
`SERVER_CA_FILE`	Optional path of the CA certificate of a private CA

The MSI writes these into the pre-seed file C:\ProgramData\MazeVault\enroll.ini only when both SERVER_URL and BOOTSTRAP_TOKEN are given, restricts C:\ProgramData\MazeVault\ to SYSTEM and Administrators and starts the service MazeVaultAgent. Without the properties the service stops cleanly for lack of configuration; an enroll.ini placed later is read when the service is started again (Start-Service MazeVaultAgent). An upgrade or repair with properties changes nothing on an enrolled host (the agent deletes the enroll.ini it writes at its next start); re-point it with enroll --force.

MSI not Authenticode-signed

The MSI and the agent .exe are not Authenticode-signed yet: Windows SmartScreen warns when the MSI is opened interactively, and AppLocker or WDAC may need a hash or path rule. Their integrity is protected by the MazeVault signature, which the backend verifies before offering the package.

One token for many hosts: create the deployment with the lifetime and use limit you need (Token expires, Max uses; API expires_in_days, max_uses; Terraform mazevault_deployment). A token that never expires and has no use limit lets anyone who holds it enroll agents; the UI warns about it.

Updates and package managers: after installation the agent updates itself according to the version policy (section 6), replacing the binary in place. The package manager still records the installed package version, so rpm -V / dpkg --verify report the binary as modified; this is expected. Reinstalling or upgrading the package may briefly put back the package's binary; the agent returns to the version of the policy at its next update check.

Uninstall: sudo apt-get remove mazevault-agent (purge also deletes configuration, identity, logs and the user), sudo dnf remove mazevault-agent (configuration and identity stay), msiexec /x mazevault-agent-<version>-x64.msi on Windows. Remote uninstall from Deployment → Agents uses Windows Installer for MSI installs; on Linux it is not available for package installs (the agent does not run as root), see section 9.

4. Configuration¶

The agent reads agent.yaml (Linux /etc/mazevault/agent.yaml, Windows C:\ProgramData\MazeVault\agent.yaml, or the path given with -config). enroll, the pre-seed file and the install script write it with the settings needed to connect; everything else is optional. After a change, restart the service (sudo systemctl restart mazevault-agent, Restart-Service MazeVaultAgent); mazevault-agent -config <path> -validate checks a file without starting the agent. An invalid file stops the agent with a configuration error (exit code 78 on Linux).

4.1 `agent.yaml` Reference¶

Defaults are for Linux; the Windows value follows where it differs.

Connection and identity

Key	Default	Description
`server_url`	— (required)	MazeVault backend or gateway URL. Must be `https://`; `http://` only for `localhost` and loopback addresses
`server_ca_file`	— (system trust store)	CA certificate that signed the server certificate (private CA)
`cert_pins`	—	List of SHA-256 hashes of the server certificate's public key (SPKI), Base64. When set, the TLS connection is accepted only for a listed key, in addition to the CA check
`bootstrap_token`	—	Deployment (bootstrap) token for the first registration
`client_cert_file`, `client_key_file`	`/etc/mazevault/agent.crt`, `agent.key` (Windows `C:\ProgramData\MazeVault\…`)	Agent identity (mTLS) issued at registration
`license_key_file`	—	JSON file with OAuth client credentials (`oauthClientId`, `oauthClientSecret`), an alternative to the bootstrap token
`agent_name`	hostname	Display name in the UI

At least one of bootstrap_token, client_cert_file + client_key_file or license_key_file must be set.

Storage and writes

Key	Default	Description
`cert_store_path`	`/var/lib/mazevault/certs` (`enroll` and the install script write `/etc/mazevault/certs`; Windows `C:\ProgramData\MazeVault\certs`)	Directory for deployed certificates; always writable for the agent. Must be absolute
`allowed_write_paths`	see Allowed write directories (section 6)	Directories or files the backend may make the agent write to; setting it replaces the defaults. Absolute paths only
`state_dir`	`/etc/mazevault` (Windows `C:\ProgramData\MazeVault`)	Agent state (identity, recovery token, version floor)
`install_chain_to_trust_store`	`true`	Install the CA and intermediate certificates of deployed certificates into the OS trust store
`trust_store_path`	auto-detected	Linux trust-anchor directory used by `install_chain_to_trust_store`
`allowed_commands`	—	Extra regular expressions for post-install commands the backend may run after a deployment (on top of the built-in list: `systemctl restart\|reload <unit>`, `service <name> restart\|reload`, `nginx -s reload`, `apachectl graceful\|restart`, `iisreset`, `Restart-WebAppPool`); invalid expressions are ignored with a warning

Behaviour

Key	Default	Description
`sync_interval_seconds`	`3600`	Interval of the periodic certificate sync; at least `10`
`min_rollback_version`	—	Local rollback floor (`[v]X.Y.Z[-pre]`): the agent never installs a lower version, whatever the version policy says
`orchestrator_mode`	`false`	After each sync also detect certificate changes made outside MazeVault and report them

Discovery (discovery: block)

Key	Default	Description
`filesystem_paths`	all local drives (fallback `/etc`, `/opt`, `/home`, `/var/lib`, `/srv`; Windows `C:\`)	Directories scanned for certificate and key files
`exclude_paths`	—	Paths skipped by the filesystem scan
`jks_paths`	as `filesystem_paths`	Directories scanned for Java keystores
`jks_passwords`	—	Passwords tried to open discovered keystores
`windows_stores`	`My`, `Root`, `CA`, `TrustedPublisher`	Windows certificate stores scanned (Windows only)

Logging, health and metrics

Key	Default	Description
`log_level`	`info`	`debug`, `info`, `warn` or `error`; on Linux the unit's `MAZEVAULT_LOG_LEVEL` overrides it (section 4.2)
`log_format`	`text`	`text` or `json`
`log_file`	— (stderr, i.e. the journal; Windows `C:\ProgramData\MazeVault\logs\agent.log`)	Log file; rotated at 100 MB, 5 files kept
`enable_audit_log`	`false`	Write a local audit log of file access and backend operations
`audit_log_path`	`/var/log/mazevault/audit.log` (Windows `C:\ProgramData\MazeVault\logs\audit.log`)	Audit log file
`health_listen_addr`	`127.0.0.1:8081`	Address of the local health endpoint (section 8)
`metrics_enabled`	`false`	Serve Prometheus metrics
`metrics_addr`	`127.0.0.1:9090`	Address of the metrics endpoint

Keys accepted but without effect in this version (kept for older configuration files): project_id, windows_store_location, gateway_discovery (enabled, mode, static_urls, dns_pattern, prefer_local), discovery.max_depth, discovery.scan_interval_minutes.

Example of optional settings:

server_url: https://vault.example.com
server_ca_file: /etc/mazevault/ca.crt
log_level: info
log_format: json
sync_interval_seconds: 1800
allowed_write_paths:
  - /etc/ssl
  - /opt/tomcat/conf
discovery:
  filesystem_paths: [/etc, /opt]
  exclude_paths: [/opt/backup]
  jks_paths: [/opt/tomcat/conf]
metrics_enabled: true

4.2 Environment Variables¶

These variables override the matching agent.yaml keys. The systemd unit sets MAZEVAULT_LOG_LEVEL=info, so on Linux log_level in agent.yaml has no effect until you override the variable. Add or change variables with sudo systemctl edit mazevault-agent ([Service] Environment=…) rather than editing the unit, which an install or package upgrade replaces.

Variable	Overrides
`MAZEVAULT_SERVER_URL`	`server_url`
`MAZEVAULT_SERVER_CA_FILE`	`server_ca_file`
`MAZEVAULT_BOOTSTRAP_TOKEN`	`bootstrap_token`
`MAZEVAULT_LICENSE_KEY_FILE`	`license_key_file`
`MAZEVAULT_CLIENT_CERT_FILE`, `MAZEVAULT_CLIENT_KEY_FILE`	`client_cert_file`, `client_key_file`
`MAZEVAULT_AGENT_NAME`	`agent_name`
`MAZEVAULT_PROJECT_ID`	`project_id` (no effect)
`MAZEVAULT_CERT_STORE_PATH`	`cert_store_path`
`MAZEVAULT_STATE_DIR`	`state_dir`
`MAZEVAULT_LOG_LEVEL`, `MAZEVAULT_LOG_FORMAT`, `MAZEVAULT_LOG_FILE`	`log_level`, `log_format`, `log_file`
`MAZEVAULT_HEALTH_ADDR`	`health_listen_addr`
`MAZEVAULT_AGENT_INSTALL_CHAIN_TO_TRUSTSTORE`	`install_chain_to_trust_store` (`true`/`1`)
`MAZEVAULT_AGENT_TRUST_STORE_PATH`	`trust_store_path`

5. Bootstrap & Registration¶

The agent uses a two-phase authentication model:

Phase 1: Bootstrap (One-Time)¶

Generate a bootstrap token in the MazeVault web UI: Settings → Deployment → Agents → Generate Bootstrap Token
Configure the agent with MAZEVAULT_SERVER_URL and MAZEVAULT_BOOTSTRAP_TOKEN
Start the agent — it registers automatically via POST /api/v1/agents/register

Phase 2: mTLS (Ongoing)¶

After successful registration, the agent receives:

Client certificate — For mutual TLS authentication
CA certificate — For verifying the server
Heartbeat interval — Recommended reporting frequency
Sync interval — Secret synchronization frequency

All subsequent communication uses mTLS. The bootstrap token is no longer needed and can be revoked.

┌──────────┐          ┌───────────────┐
│  Agent   │──────────│  MazeVault    │
│          │ Bootstrap│  Backend      │
│          │  Token   │               │
│          │◀─────────│               │
│          │ mTLS Cert│               │
│          │          │               │
│          │──────────│               │
│          │ mTLS     │               │
│          │ (ongoing)│               │
└──────────┘          └───────────────┘

Registration Rules¶

A deployment token cannot re-register an agent that belongs to another organization.
Agents send a hashed machine ID. Two machines with the same hostname but different machine IDs become separate agents (for agents of this version).
An agent that lost its local state can still re-register with its original deployment token after the token expired, but only on the same machine and within the same organization.
An agent's recovery token identifies it even when its machine ID changed (for example after a rollback to an older agent version).
A record of an older agent (registered without a machine ID) in another organization does not block the hostname: an agent of this version that registers with a deployment token of its own organization becomes a new agent.

6. Periodic Operations¶

The agent runs several background tasks automatically:

Operation	Interval	Description
Secret sync	Configurable (default 5m)	Synchronizes secrets from backend to local storage
Heartbeat	60 seconds	Reports health, metrics (CPU, memory, disk), and version to backend
Certificate discovery	1 hour	Scans configured paths for certificates (filesystem, JKS, Windows cert store)
Configuration discovery	With the filesystem scan	Scans policy-defined paths for configuration files (`.env`, YAML, JSON, INI, properties, TOML, XML), detects embedded secrets, and reports candidates — requires the `config_management` license feature; see Configuration Management
Update check	24 hours	Checks for newer agent versions
License heartbeat	5 minutes	Sends telemetry to license server (when configured)
K8s reconciliation	5 minutes	Reconciles Kubernetes Secret certificates (K8s only)
WebSocket reconnect	Exponential backoff	Maintains real-time push connection (1s → 2s → 4s → ...)

All intervals are fixed except secret sync (SYNC_INTERVAL_SECONDS).

Updates and Rollback Floor¶

On start and at every update check the agent asks its backend which version the agent version policy offers it (see Environment Variables § Agent Binaries). It installs a newer version, or a lower one only when the backend marks it as a downgrade and it is not below the agent's rollback floor. Every binary is verified (hash, signature, reported version) before the swap. The floor is the highest of:

the floor built into the agent binary (raised by MazeVault releases),
the highest min_rollback_version the agent has accepted from its backend, persisted in <state_dir>/min-rollback-version (the backend can only raise it),
min_rollback_version in agent.yaml.

# /etc/mazevault/agent.yaml
min_rollback_version: v1.2.0   # never install a version below v1.2.0

A refused version is logged with the running version, the offered version and the floor. Lowering the floor is a local action only: lowering agent.yaml has no effect below the persisted value; to clear the value persisted from the backend run

mazevault-agent -config /etc/mazevault/agent.yaml reset-min-rollback-version

Certificate Deployment Requests¶

Besides the periodic tasks, the backend pushes certificate deployment work to the agent over the WebSocket connection when a rotation runs. The agent handles these request types:

Request	What the agent does
`certificate_sync`	Installs certificate + key into a keystore (`jks`, `pkcs12`, `windows_my`) and the issuer chain into the platform trust store. `windows_root` / `windows_ca` targets receive public certificates only — a private key is never placed in a trust store.
`certificate_file_sync`	Writes certificate files according to the target's content mode (`public_only`, `public_chain`, `full_chain_with_key`, `key_only`): leaf, chain, key and an optional combined bundle (PEM or PKCS#12). Private material is written `0600`; existing files are kept as `.BAK`.
`agent_command`	Runs a command for a deployment step (service restart, WLST restart script, `orapki` wallet import, `keytool` trust-store import). Only allowlisted commands execute. Arguments the backend flags as sensitive (store and wallet passwords) are redacted in the agent's audit log, response and error messages.

Command allowlist¶

The agent ships with a built-in allowlist (systemctl restart|reload, nginx -s reload, apachectl graceful, iisreset, IIS app-pool recycle, orapki wallet import_pkcs12 of a MazeVault-staged PKCS#12 and its cleanup, keytool -importcert of a MazeVault-staged issuer PEM into a JKS/PKCS12 trust store and its cleanup). Anything else — including the WLST restart script of a WebLogic target — must be allowed explicitly as a regular expression in the agent configuration:

allowed_commands:
  - '^/opt/weblogic/scripts/restart-managed-server\.sh\s+[\w\-\./:= ]+$'

Patterns are matched against the full command line. Keep them narrow (absolute paths, no shell metacharacters) — a rejected command fails the deployment step with a clear message instead of running.

Allowed write directories¶

The agent writes files the backend names (certificate_sync keystores, certificate_file_sync files, secret_sync targets, keytab_deploy keytabs, renewed certificates, file and JKS targets of the periodic sync) only inside cert_store_path and the directories listed in allowed_write_paths. When allowed_write_paths is not set, it defaults to /etc/ssl, /etc/pki, /etc/letsencrypt, /etc/nginx/ssl, /etc/nginx/certs, /etc/apache2/ssl, /etc/httpd/ssl, /etc/haproxy/certs, /etc/krb5.keytab, /var/lib/mazevault/certs and /etc/mazevault/certs on Linux, C:\ProgramData\MazeVault\certs on Windows. An entry may also name a single file. Paths must be absolute and must not contain ..; symlinks are resolved before the check, so a link inside an allowed directory cannot point the write elsewhere. The agent's own configuration, identity files and state directory are never writable this way. Add every other directory your deployment targets use:

allowed_write_paths:
  - /etc/ssl
  - /opt/tomcat/conf
  - /etc/jira-sync/config.json

Setting the list replaces the defaults; entries and cert_store_path must be absolute or the agent does not start. On Windows, junctions and mount points inside the path are refused as well. A refused path fails the request with an error naming allowed_write_paths; the periodic certificate sync (file and JKS targets) only logs the refusal in the agent log, so search the log for allowed_write_paths after upgrading.

Agent version requirements

certificate_file_sync, agent_command and the public-only semantics of windows_root/windows_ca require this agent version or newer. An older agent answers such requests with an explicit unsupported message type error, and the deployment step fails fast instead of installing material in the wrong place.

7. Certificate Discovery¶

The agent discovers certificates across infrastructure and reports them to the backend:

Discovery Sources¶

Source	Platform	Description
Filesystem	Linux/Windows	Scans configured directories for PEM, DER, PFX files
Java KeyStore (JKS)	All	Parses `.jks` and `.p12` files
Windows Certificate Store	Windows	Enumerates system and user certificate stores
Kubernetes Secrets	K8s	Watches `kubernetes.io/tls` secrets

Discovery Configuration¶

Discovery paths are configured via the MazeVault web UI under:

Project → Settings → Agent Discovery → Paths

Results appear in the Discovered Certificates view in the dashboard.

Configuration file discovery is driven by organization-level discovery policy templates rather than by the paths above; results appear under Enterprise Functions → Configuration Management → Discovered Configurations. See Configuration Management. Agents can also deliver externally-issued API tokens to configuration files as agent_file deployment targets; see Token Management.

8. Monitoring¶

Health Endpoint¶

The agent reports health via heartbeat. Monitor agent status in the web UI:

Deployment → Agents

Key Metrics Reported¶

cpu_percent — Agent CPU usage
memory_percent — Agent memory usage
secrets_synced — Number of secrets synchronized
certificates_managed — Number of managed certificates
last_sync_time — Timestamp of last successful sync
uptime_seconds — Agent uptime

Prometheus Metrics¶

The backend exposes agent metrics at /metrics:

mazevault_agents_online{environment="production"} 12
mazevault_agent_sync_operations_total{agent_id="agt_xyz"} 1842
mazevault_agent_discovery_findings_total{agent_id="agt_xyz"} 37

9. Troubleshooting¶

Agent Won't Start¶

# Check logs
journalctl -u mazevault-agent -f

# Verify connectivity
curl -sk https://vault.example.com/api/v1/health

Problem	Cause	Solution
`connection refused`	Backend unreachable	Verify `MAZEVAULT_SERVER_URL` and firewall rules
`certificate expired`	Agent cert expired	Re-register with new bootstrap token
`bootstrap token invalid`	Token expired or used	Generate a new bootstrap token in the web UI
`TLS handshake error`	CA certificate mismatch	Verify CA certificate chain
`status=203/EXEC` in `systemctl status`	systemd cannot execute the binary: `noexec` mount (typically `/var`), SELinux label, or missing execute bit	Re-run the install script (installs to `/opt/mazevault-agent/bin`, see 3.2.1); check `findmnt -T <binary dir>` and `ls -Z`
`GLIBC_2.34' not found`	Agent older than 1.0.61 on a host with glibc < 2.34 (RHEL/Rocky/Alma 8)	Install agent 1.0.61 or later (statically linked)
Installer exits with `2`, `3` or `4`	Preflight, download/verification, or service start failed	Read the printed reason; `3` with `HTTP 503` means the server has no agent binaries configured

Agent Lifecycle (Deployment → Agents)¶

Registered agents are managed from Deployment → Agents via the per-row actions menu. These operations act on the MazeVault control-plane identity of an installed agent and are distinct from deleting a generated install-script record under Deployment → Deploy.

Action	What it does	When to use
Unregister	Retires the agent, invalidates its mTLS identity and recovery tokens, deactivates project bindings, and blocks all future runtime use. History is preserved. Choose a security reason category (compromised / lost credential / stolen host) to record it as a revoke.	Decommissioned, lost, offline, or manually removed hosts.
Remote uninstall	Asks a connected agent to uninstall itself from the host and reports the real outcome back before finalizing. Supported on Linux (systemd package install, agent running as root) and Windows (MSI service install; the `MazeVaultAgent` service is removed via `msiexec`). The agent must be currently connected. The agent is retired only after it confirms a successful removal; if the removal fails the agent stays active and the console shows Uninstall failed so you can retry or intervene.	Cleanly remove the agent from a reachable Linux or Windows host.
Re-register / Renew identity	Retires the old identity and issues a single-use replacement bootstrap token (valid 24 h) for re-enrolling the host.	mTLS certificate expired or compromised; replacing a host.

An impact preview (managed certificates, project bindings, rotation targets, pending keytab work) is shown before unregistering. Unresolved work such as pending keytab deployments or actively managed certificates requires the Force option to proceed.

Unregister does not uninstall the host binary. For offline hosts, or where remote uninstall is unsupported (non-packaged/manual installs, macOS), unregister the agent and remove it locally.

Linux:

sudo systemctl stop mazevault-agent && sudo systemctl disable mazevault-agent
sudo apt remove mazevault-agent   # or: sudo rpm -e mazevault-agent
sudo rm -rf /etc/mazevault /var/lib/mazevault /var/log/mazevault /opt/mazevault-agent

Windows (PowerShell, elevated):

$p = Get-WmiObject Win32_Product -Filter "Name='MazeVault Agent'"
if ($p) {
    $p.Uninstall()    # installed from the MSI package
} else {
    # installed by the script: stops and deletes the MazeVaultAgent service
    & "C:\Program Files\MazeVault Agent\mazevault-agent.exe" -config C:\ProgramData\MazeVault\agent.yaml uninstall
    Remove-Item -Recurse -Force "C:\Program Files\MazeVault Agent" -ErrorAction SilentlyContinue
}
Remove-Item -Recurse -Force "$env:ProgramData\MazeVault" -ErrorAction SilentlyContinue

Older Windows installations keep the binary under C:\ProgramData\MazeVault\; run uninstall from there instead.

Re-Registration¶

If the agent's mTLS certificate expires or is compromised:

In Deployment → Agents, open the agent's actions menu and choose Re-register / Renew identity (or Unregister with a security reason for a compromised host).
Copy the single-use replacement bootstrap token from the dialog.
Remove local state on the host: rm -rf /etc/mazevault/agent.state (Windows: the agent.state file in C:\ProgramData\MazeVault\). Alternatively run mazevault-agent enroll --force with the new token, which deletes the configuration, client certificate, key and state — the agent gets a new identity (certificate, key and recovery token) and the server keeps the same agent record when the hostname and the machine ID match.
Restart the agent with the new MAZEVAULT_BOOTSTRAP_TOKEN.

The token must belong to the same organization as the agent; a token of another organization is rejected. An agent that lost its local state can also re-register with its original deployment token after it expired, but only on the same machine and within the same organization (see Registration Rules in section 5).

Debug Logging¶

Enable verbose logging for diagnostics:

MAZEVAULT_LOG_LEVEL=debug mazevault-agent

10. Security Considerations¶

Bootstrap tokens are single-use and time-limited (default: 24 hours)
mTLS certificates are scoped to the specific agent and project
Secret data is encrypted in transit (TLS 1.2+) and cached encrypted locally
Minimal permissions — agents only access secrets assigned to their project
No inbound ports — the agent initiates all connections (outbound only)

Agents API — REST API reference for agent endpoints
FAQ — Frequently asked questions
Scaling — Agent scaling guidance
Authentication — Agent authentication details
Monitoring — Prometheus metrics for agents