Troubleshooting

Diagnose and resolve common deployment issues.

Common Issues

Target Offline

┌─────────────────────────────────────────────────────────────┐
│  ✗ Error: Target Offline                                    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Target: web-prod-03                                        │
│  Status: Offline (last seen 2 hours ago)                    │
│                                                             │
│  Possible causes:                                           │
│  • Tarsus agent not running                                 │
│  • Network connectivity issues                              │
│  • Firewall blocking connection                             │
│  • Server powered off or rebooted                           │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Diagnosis:

1. Check if the server is accessible
2. Verify Tarsus agent is running
3. Check network/firewall rules
4. Review Tarsus logs

Resolution:

# On target machine
systemctl status tarsus
systemctl start tarsus

# Check logs
journalctl -u tarsus -f

Command Timeout

┌─────────────────────────────────────────────────────────────┐
│  ✗ Error: Command Timeout                                   │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Action: deploy-app                                         │
│  Timeout: 300 seconds                                       │
│  Elapsed: 300 seconds                                       │
│                                                             │
│  The command did not complete within the timeout period.    │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Diagnosis:

1. Check what the command was doing
2. Look for network or I/O bottlenecks
3. Verify resource availability

Resolution:

Cause	Solution
Slow network	Increase timeout or optimize transfer
Large file copy	Use streaming or chunked transfer
Hung process	Fix underlying issue or add timeout handling
Resource exhaustion	Free up disk/memory

Permission Denied

┌─────────────────────────────────────────────────────────────┐
│  ✗ Error: Permission Denied                                 │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Command: cp /tmp/app.zip /opt/apps/                        │
│  Exit code: 1                                               │
│  Error: Permission denied: /opt/apps/                       │
│                                                             │
│  The Tarsus agent does not have permission to write to      │
│  the target directory.                                      │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Diagnosis:

1. Check Tarsus user permissions
2. Verify directory ownership
3. Review SELinux/AppArmor policies

Resolution:

# Fix directory permissions
sudo chown -R tarsus:tarsus /opt/apps
sudo chmod 755 /opt/apps

# Or run Tarsus with appropriate user

Connection Refused

┌─────────────────────────────────────────────────────────────┐
│  ✗ Error: Connection Refused                                │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Target: web-prod-01:9342                                  │
│  Error: Connection refused                                  │
│                                                             │
│  Thorax could not connect to the Tarsus agent.              │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Diagnosis:

1. Check Tarsus is running and listening
2. Verify port is correct
3. Check firewall rules
4. Verify TLS certificates

Resolution:

# Check if Tarsus is listening
netstat -tlnp | grep 9342

# Check firewall
iptables -L -n | grep 9342

# Test connectivity from Thorax
nc -zv target-hostname 9342

Certificate Error

┌─────────────────────────────────────────────────────────────┐
│  ✗ Error: Certificate Verification Failed                   │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Target: web-prod-01                                        │
│  Error: certificate has expired                             │
│                                                             │
│  The TLS certificate presented by Tarsus is not valid.      │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Diagnosis:

1. Check certificate expiration
2. Verify certificate chain
3. Confirm CA trust

Resolution:

# Check certificate expiration
openssl x509 -in /etc/mantis/certs/client.crt -noout -dates

# Regenerate a client certificate (dev helper), then re-pre-register it
mantisctl cert generate-client --name web-prod-01
mantisctl cert revoke <old-registration-id> --reason "Certificate renewed"
mantisctl cert pre-register --name web-prod-01 --from-cert ./certs/web-prod-01-cert.pem

Version Not Found

┌─────────────────────────────────────────────────────────────┐
│  ✗ Error: Version Not Found                                 │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Solution: customer-portal                                  │
│  Requested version: 1.3.0                                   │
│  Available versions: 1.0.0, 1.1.0, 1.2.0                    │
│                                                             │
│  The requested version does not exist.                      │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Resolution:

1. Check available versions
2. Create the missing version
3. Use a different version

Variable Not Set

┌─────────────────────────────────────────────────────────────┐
│  ✗ Error: Required Variable Not Set                         │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Variable: database_host                                    │
│  Environment: Production                                    │
│                                                             │
│  This variable is required but has no value for the         │
│  target environment.                                        │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Resolution:

1. Navigate to solution variables
2. Set value for the environment
3. Retry deployment

Diagnostic Tools

Deployment Logs

Access detailed logs:

# View deployment logs
mantisctl deployment logs <deployment-id>

# Filter by target
mantisctl deployment logs <deployment-id> --target <target-id>

# Filter by log level
mantisctl deployment logs <deployment-id> --level error

Target Health Check

Verify target connectivity:

# Pre-flight check for a specific target (validates health + freezes)
mantisctl deployment check --solution-id customer-portal --target web-prod-01

# Pre-flight check for an environment
mantisctl deployment check --solution-id customer-portal --environment-id production

Connection Test

Test the network path to a target’s listen port (default 9342):

# DNS resolution
getent hosts web-prod-01.company.com

# TCP reachability of the Tarsus listen port
nc -vz web-prod-01.company.com 9342

# Inspect the target's certificate over TLS
openssl s_client -connect web-prod-01.company.com:9342 -servername web-prod-01.company.com </dev/null

Variable Resolution

The pre-flight check and the deployment preview API (POST /api/v1/deployments/preview) report the variables that will be applied to a deployment, including which prompted values are still required. Run the pre-flight check above, or call the preview endpoint, to verify variable resolution before deploying.

Debugging Steps

Step 1: Identify the Error

Step 2: Gather Information

Information	Source
Error message	Deployment logs
Target status	Target list
Command output	Log streaming
System state	Target machine

Step 3: Check Common Causes

For connection issues:

• Tarsus agent running
• Network accessible
• Firewall allows traffic
• Certificates valid

For command failures:

• Script syntax correct
• Dependencies installed
• Permissions adequate
• Resources available

For timeout issues:

• Network bandwidth
• Disk I/O performance
• CPU/memory load
• Command complexity

Step 4: Test Fix

# Deploy to a single target first
mantisctl deployment create --solution-id customer-portal \
  --source-version 1.3.0 \
  --target web-prod-01

Step 5: Verify Resolution

Confirm the issue is fixed before deploying to all targets.

Error Reference

Exit Codes

Code	Meaning
0	Success
1	General error
126	Permission denied
127	Command not found
128+N	Killed by signal N
137	Killed (SIGKILL, likely OOM)
143	Terminated (SIGTERM)
255	Exit status out of range

Common Error Messages

Error	Meaning	Solution
ECONNREFUSED	Connection refused	Check Tarsus is running
ETIMEDOUT	Connection timeout	Check network path
ENOTFOUND	DNS lookup failed	Verify hostname
EACCES	Permission denied	Fix file permissions
ENOENT	File not found	Check paths in scripts
ENOMEM	Out of memory	Free memory or increase
ENOSPC	No space left	Free disk space

Recovery Procedures

Failed Deployment Recovery

Rollback After Failure

1. Identify affected targets
2. Determine previous working version
3. Deploy previous version to affected targets

# Rollback specific targets to the previous version
mantisctl deployment create --solution-id customer-portal \
  --source-version 1.2.0 \
  --target web-prod-01 \
  --target web-prod-02

Emergency Recovery

If deployments are consistently failing:

1. Stop - Cancel active deployments
2. Assess - Review system state
3. Stabilize - Rollback if needed
4. Diagnose - Find root cause
5. Fix - Address underlying issue
6. Test - Verify in lower environment
7. Deploy - Retry with fix

Best Practices

1. Monitor Deployments

• Watch for early failures
• Review logs in real-time
• Set up alerts for failures

2. Use Staged Rollouts

Phase 1: 1 target (canary)
Phase 2: 10% of targets
Phase 3: 50% of targets
Phase 4: Remaining targets

3. Test in Lower Environments

Always deploy to development/staging first.

4. Keep Rollback Ready

• Know the previous working version
• Verify rollback works before production
• Document rollback procedure

5. Post-Incident Review

After resolving issues:

1. Document what happened
2. Identify root cause
3. Implement prevention
4. Update procedures

Getting Help

Check Documentation

• Review action and script documentation
• Verify configuration matches examples
• Check for known issues

Contact Support

Provide:

• Deployment ID
• Error messages
• Target details
• Steps to reproduce

Community Resources

• GitHub Issues
• Community forums
• Knowledge base

Next Steps

• Monitoring - Track deployment progress
• Log Streaming - View execution logs
• Cancellation - Stop deployments
• Overview - Return to deployments overview