Connection Problems

Connection Problems

This guide covers network connectivity issues between Mantis components. The typical communication flow is:

Tarsus --[register/heartbeat]--> Mandible --[dispatch]--> Thorax --[execute]--> Tarsus

All inter-component communication uses mTLS, so connectivity problems may be network-level or TLS-level.

Diagnosing Connectivity

Step 1: Identify Which Connection Is Failing

Connection	Protocol	Default Port	Check Command
Client to Mandible (API)	HTTPS	3000	curl https://host:3000/api/v1/health
Mandible to Thorax	gRPC (mTLS)	50051	Check /api/v1/health/grpc
Thorax to Mandible (internal)	gRPC (mTLS)	50052	Check Thorax logs for connection errors
Tarsus to Mandible (register)	gRPC (mTLS)	50052	Check Tarsus logs for registration errors
Tarsus to assigned Thorax	gRPC (mTLS)	50051	Check deployment execution logs (Tarsus dials Thorax; Thorax pushes over that connection in listen mode)
Mandible to PostgreSQL	TCP	5432	psql $DATABASE_URL -c "SELECT 1"
Mandible to RabbitMQ	AMQP/TLS	5671	Check queue health in Mandible logs

Step 2: Test Network Connectivity

# Test basic TCP connectivity
nc -zv thorax-host 50051

# Test with timeout
timeout 5 bash -c "echo > /dev/tcp/thorax-host/50051" && echo "Open" || echo "Closed"

# Check DNS resolution
nslookup thorax-host
dig thorax-host

Step 3: Test TLS Connectivity

# Test TLS handshake
openssl s_client -connect thorax-host:50051 \
  -cert /etc/mantis/certs/mandible.crt \
  -key /etc/mantis/certs/mandible.key \
  -CAfile /etc/mantis/certs/ca.crt

# Check certificate details
openssl s_client -connect thorax-host:50051 < /dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Common Connection Problems

Firewall Blocking Traffic

Symptom: Connection times out rather than being refused.

# Check if port is reachable
nc -zv -w 5 target-host 3000

# Check local firewall rules (Linux)
iptables -L -n | grep -E "(3000|50051|50052)"

# Check firewall status
ufw status        # Ubuntu
firewall-cmd --list-all  # RHEL/CentOS

Fix: Open required ports between components:

From	To	Port	Purpose
Load balancer	Mandible	3000	API traffic
Mandible	Thorax	50051	Orchestration gRPC
Thorax	Mandible	50052	Internal gRPC
Thorax	Tarsus	9342	Command execution (listen mode)
Tarsus	Mandible	50052	Registration/heartbeat (gRPC)
Mandible	PostgreSQL	5432	Database
Mandible	RabbitMQ	5671	Message queue

DNS Resolution Failures

Symptom: Name or service not known or could not resolve hostname in logs.

# Verify DNS resolution
nslookup mandible-host
dig mandible-host +short

# Check /etc/hosts for local overrides
cat /etc/hosts | grep mantis

# In Docker, check service names resolve
docker compose exec thorax nslookup mandible

Fix: Ensure hostnames in configuration match DNS or /etc/hosts entries. In Docker Compose, use service names (e.g., mandible, thorax).

Connection Refused

Symptom: Connection refused error immediately.

This means the port is reachable but nothing is listening.

1. Verify the target service is running
2. Check the service is binding to the expected address (not just 127.0.0.1 when accessed remotely)
3. Confirm the port matches the configuration

# Check what's listening on the expected port
ss -tlnp | grep 3000
# or
netstat -tlnp | grep 3000

gRPC Connection Issues

Symptom: Mandible logs show gRPC connection failed or circuit breaker opens.

# Check gRPC health. The circuit_state and failure_count fields are only
# returned to authenticated callers, so pass a token; an anonymous request
# shows only the overall status with an empty circuit_state.
curl -s -H "Authorization: Bearer $TOKEN" \
  https://mantis.example.com/api/v1/health/grpc | jq

# If circuit breaker is open, check failure count
# The circuit breaker will attempt recovery in half-open state

Common gRPC causes:

• Thorax is not running or crashed
• TLS certificate mismatch between Mandible and Thorax
• Network partition between Mandible and Thorax hosts
• Thorax is overloaded and not responding to health checks

Tarsus Registration Failures

Symptom: Tarsus cannot register with Mandible.

1. Verify Tarsus can reach Mandible’s internal gRPC port (default 50052)
2. Check Tarsus certificate is signed by the trusted CA
3. Ensure the registration endpoint is accessible
4. Review Tarsus logs for specific error messages

# Test gRPC reachability from the Tarsus host (registration uses gRPC, not REST)
nc -zv mandible-host 50052

Docker-Specific Issues

Container Network Isolation

Symptom: Services cannot reach each other in Docker Compose.

# Verify all services are on the same network
docker network inspect mantis_default

# Test connectivity between containers
docker compose exec mandible ping thorax
docker compose exec mandible nc -zv thorax 50051

Port Mapping Conflicts

Symptom: port already allocated when starting containers.

# Find what's using the port
lsof -i :3000
# or
ss -tlnp | grep 3000

# Use alternative port mappings in docker-compose
# ports:
#   - "3001:3000"

Next Steps

• Certificate Issues — TLS-specific troubleshooting
• Queue Issues — RabbitMQ connectivity
• Common Issues — General troubleshooting