Certificate Backup

Certificate Backup

Mantis uses mTLS certificates for all inter-component communication. Losing certificates means components cannot authenticate with each other, requiring re-enrollment. This guide covers how to back up and restore certificates safely.

What to Back Up

Certificates follow the *-cert.pem / *-key.pem naming convention in /etc/mantis/certs/:

File	Location	Purpose	Sensitivity
CA certificate	/etc/mantis/certs/ca-cert.pem	Root of trust for all components	Public (can be shared)
CA private key	/etc/mantis/certs/ca-key.pem	Signs new certificates	Critical — protect this
Component certificates	/etc/mantis/certs/<component>-cert.pem	Identity for each component	Public
Component private keys	/etc/mantis/certs/<component>-key.pem	Authentication for each component	Sensitive

Component certificate/key pairs include mandible-cert.pem/mandible-key.pem, client-cert.pem/client-key.pem (the shared mesh client cert used by Tarsus agents and Mandible’s orchestration client), and the Thorax-side server-cert.pem/server-key.pem. Mandible also holds dispatch-cert.pem/dispatch-key.pem — its dispatch identity (CN “mandible”) that Thorax trusts for cross-tenant deployment calls. Production stores the RS256 JWT signing pair jwt-private.pem/jwt-public.pem in /etc/mantis/keys/ (a separate directory from /etc/mantis/certs/; include it in any backup that covers /etc/mantis/certs/).

Danger

The CA private key is the most critical file. If compromised, an attacker can issue valid certificates for any component. If lost, you cannot sign new certificates without re-initializing the entire CA.

Backup Procedures

Manual Backup

# Create a dated backup directory
BACKUP_DIR="/opt/mantis/backups/certs/$(date +%Y%m%d)"
mkdir -p "$BACKUP_DIR"

# Copy certificate files and JWT signing keys
cp -a /etc/mantis/certs/ "$BACKUP_DIR/"
cp -a /etc/mantis/keys/ "$BACKUP_DIR/"

# Set restrictive permissions on the backup
chmod -R 600 "$BACKUP_DIR"
chown -R root:root "$BACKUP_DIR"

# Verify the backup
ls -la "$BACKUP_DIR/"
diff -r /etc/mantis/certs/ "$BACKUP_DIR/"

Encrypted Backup

For off-site or cloud storage, encrypt the backup:

# Create a tarball and encrypt with a passphrase
# Includes both /etc/mantis/certs/ (mTLS certs) and /etc/mantis/keys/ (JWT signing keys)
tar czf - /etc/mantis/certs/ /etc/mantis/keys/ | \
  gpg --symmetric --cipher-algo AES256 \
  -o "/opt/mantis/backups/certs-$(date +%Y%m%d).tar.gz.gpg"

# Or encrypt with a public key (preferred for automation)
tar czf - /etc/mantis/certs/ /etc/mantis/keys/ | \
  gpg --encrypt --recipient "ops@example.com" \
  -o "/opt/mantis/backups/certs-$(date +%Y%m%d).tar.gz.gpg"

Restore from Encrypted Backup

# Decrypt and extract
gpg --decrypt "/opt/mantis/backups/certs-20260115.tar.gz.gpg" | \
  tar xzf - -C /

# Verify restored files
openssl x509 -in /etc/mantis/certs/ca-cert.pem -noout -subject -dates

Automated Backup Script

/opt/mantis/scripts/backup-certs.sh

#!/bin/bash
set -euo pipefail

CERT_DIR="/etc/mantis/certs"
KEYS_DIR="/etc/mantis/keys"   # JWT RS256 signing pair lives here
BACKUP_BASE="/opt/mantis/backups/certs"
RETENTION_DAYS=90
DATE=$(date +%Y%m%d-%H%M%S)

# Create backup (certs + JWT keys)
mkdir -p "$BACKUP_BASE"
tar czf "$BACKUP_BASE/certs-$DATE.tar.gz" \
  -C /etc/mantis certs keys
chmod 600 "$BACKUP_BASE/certs-$DATE.tar.gz"

# Verify backup integrity
tar tzf "$BACKUP_BASE/certs-$DATE.tar.gz" > /dev/null

# Clean old backups
find "$BACKUP_BASE" -name "certs-*.tar.gz" -mtime +$RETENTION_DAYS -delete

echo "Certificate backup completed: $BACKUP_BASE/certs-$DATE.tar.gz"

Schedule with cron:

# Weekly certificate backup
0 2 * * 0 /opt/mantis/scripts/backup-certs.sh >> /var/log/mantis/cert-backup.log 2>&1

CA Key Storage Best Practices

The CA private key deserves extra protection beyond standard backups:

Offline Storage

After initial setup and certificate signing, move the CA key to offline storage:

1. Copy the CA key to an encrypted USB drive
2. Store the USB drive in a physical safe or secure location
3. Remove the CA key from the server (keep only the CA certificate)
4. Only bring the CA key online when signing new certificates

Secrets Management

For automated environments, store the CA key in a secrets manager:

Tool	Command
HashiCorp Vault	vault kv put secret/mantis/ca key=@ca-key.pem
AWS Secrets Manager	aws secretsmanager create-secret --name mantis-ca-key --secret-binary fileb://ca-key.pem
Azure Key Vault	az keyvault secret set --vault-name mantis --name ca-key --file ca-key.pem

Hardware Security Module (HSM)

For the highest security, store the CA key in an HSM:

• AWS CloudHSM
• Azure Dedicated HSM
• On-premise PKCS#11 HSM

Verification After Restore

After restoring certificates from backup, verify everything works:

# Verify CA certificate
openssl x509 -in /etc/mantis/certs/ca-cert.pem -noout -text | head -20

# Verify component certificates are signed by the CA
openssl verify -CAfile /etc/mantis/certs/ca-cert.pem /etc/mantis/certs/mandible-cert.pem
openssl verify -CAfile /etc/mantis/certs/ca-cert.pem /etc/mantis/certs/server-cert.pem
openssl verify -CAfile /etc/mantis/certs/ca-cert.pem /etc/mantis/certs/client-cert.pem
openssl verify -CAfile /etc/mantis/certs/ca-cert.pem /etc/mantis/certs/dispatch-cert.pem

# Check certificate expiry dates
for cert in /etc/mantis/certs/*-cert.pem; do
  echo "$cert: $(openssl x509 -in "$cert" -noout -enddate)"
done

# Restart services and verify connectivity.
# /health/grpc exposes the overall status (healthy/degraded/unhealthy) to any
# caller; the breaker internals (circuit_state, failure_count, success_count)
# are only returned to authenticated callers. Anonymous requests get an empty
# circuit_state and zeroed counts, so pass a token to inspect the breaker.
systemctl restart mantis-mandible mantis-thorax mantis-tarsus
curl -s -H "Authorization: Bearer $TOKEN" \
  https://localhost:3000/api/v1/health/grpc | jq

Next Steps

• Key Rotation Runbook — Scheduled certificate rotation
• Database Backup — Back up the Mantis database
• Disaster Recovery — Full system recovery procedures
• Certificate Issues — Troubleshoot certificate problems