Skip to Content

Performance Issues

TL;DR

CMDOP performance issues include slow command execution, file transfer bottlenecks, high latency, and memory problems. Optimize by batching commands into single executions, using cmdop files sync for incremental transfers, compressing files before transfer, and streaming large outputs instead of buffering. Expected latency ranges from 20-50ms (US) to 100-200ms (Asia). Use cmdop exec server "true" to benchmark round-trip time.

Why is command execution slow?

What are the symptoms?

  • Commands take longer than expected
  • Noticeable delay between typing and response
  • Timeout errors on simple commands

How do I diagnose slow commands?

# Measure latency
time cmdop exec server "echo hello"
 
# Check network latency
ping grpc.cmdop.com
 
# Check server load
cmdop exec server "uptime"

How do I fix slow command execution?

Network Latency

# Check your connection
mtr grpc.cmdop.com
 
# Expected latency by region:
# US: 20-50ms
# EU: 50-100ms
# Asia: 100-200ms

Server Location

For self-hosted, deploy Control Plane closer to your machines.

Timeout Configuration

# Increase timeout for slow commands
cmdop exec server "slow-command" --timeout 300

Command Optimization

# Avoid multiple round trips
# Bad: multiple separate commands
cmdop exec server "cd /app"
cmdop exec server "npm install"
cmdop exec server "npm run build"
 
# Good: single command
cmdop exec server "cd /app && npm install && npm run build"

Why are file transfers slow?

What are the symptoms?

  • File uploads/downloads are slow
  • Progress bar moves slowly
  • Transfers timeout

How do I diagnose transfer speed issues?

# Test bandwidth
cmdop exec server "dd if=/dev/zero bs=1M count=100 | pv > /dev/null"
 
# Check file size
ls -lh localfile

How do I improve file transfer speed?

Compression

# Compress before transfer
tar -czf archive.tar.gz directory/
cmdop files put archive.tar.gz server:/tmp/
 
# Decompress on server
cmdop exec server "cd /tmp && tar -xzf archive.tar.gz"

Chunk Size

The SDK uses 10MB chunks by default. For slow connections:

# SDK: adjust chunk size
await client.download.url(
    url="...",
    local_path="file.zip",
    chunk_size=5 * 1024 * 1024,  # 5MB chunks
)

Sync Instead of Full Transfer

# Only transfer changes
cmdop files sync ./local server:/remote

Parallel Transfers

# Transfer multiple files in parallel
cmdop files put ./file1 server:/tmp/file1 &
cmdop files put ./file2 server:/tmp/file2 &
wait

How do I reduce high latency?

How do I diagnose latency issues?

# Measure round-trip time
time cmdop exec server "true"
 
# Network path analysis
traceroute grpc.cmdop.com

How do I fix high latency?

Use Regional Endpoint

For self-hosted:

export CMDOP_SERVER_ADDRESS=us-west.cmdop.company.com:443

Reduce Chattiness

# Batch operations
cmdop exec server "uptime; df -h; free -m"

Keep Sessions Open

# Reuse terminal session instead of exec
cmdop terminal server
# Run multiple commands in same session

How do I fix memory issues?

How do I troubleshoot agent memory usage?

How do I diagnose agent memory problems?

# Check agent memory
ps aux | grep "cmdop connect"
 
# Check system memory
free -m

How do I reduce agent memory usage?

# Restart agent to clear memory
cmdop disconnect
cmdop connect

For persistent issues:

# Limit agent memory (systemd)
# /etc/systemd/system/cmdop-agent.service
[Service]
MemoryMax=256M

How do I fix SDK memory issues?

How do I stream large output instead of buffering?

# Bad: load all output
result = client.terminal.execute("find /")
print(result.output)  # Huge string
 
# Good: stream output
async for event in client.terminal.stream("find /"):
    if event.type == "output":
        process_line(event.text)

How do I properly close sessions to free memory?

# Always close sessions
async with client.terminal.create_session() as session:
    # work...
# Automatically closed
 
# Or explicitly
session = await client.terminal.create_session()
try:
    # work...
finally:
    await session.close()

How do I fix CPU issues?

How do I troubleshoot agent CPU usage?

How do I diagnose high agent CPU?

# Check agent CPU
top -p $(pgrep -f "cmdop connect")
 
# Watch over time
pidstat -p $(pgrep -f "cmdop connect") 1

What causes high CPU and how do I fix it?

High CPU usually indicates:

  1. Many concurrent sessions
  2. Large file transfers
  3. Streaming a lot of output

Mitigations:

  • Limit concurrent sessions
  • Transfer during off-peak
  • Reduce output verbosity

How do I fix control plane CPU issues?

How do I find slow database queries?

-- Check slow queries
SELECT query, calls, mean_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;

What indexes should I add?

-- Common indexes
CREATE INDEX idx_sessions_machine ON sessions(machine_id);
CREATE INDEX idx_audit_timestamp ON audit_logs(timestamp);

How do I fix connection pool exhaustion?

What are the symptoms?

Error: connection pool exhausted

How do I fix connection pool exhaustion?

SDK Configuration

from cmdop import ConnectionConfig
 
config = ConnectionConfig(
    max_connections=50,  # Increase pool
    connection_timeout_seconds=30,
)
 
client = CMDOPClient.remote(api_key="...", config=config)

Close Unused Clients

# Always close clients
client = CMDOPClient.remote(...)
try:
    # work...
finally:
    client.close()

How do I tune timeouts?

How do I configure SDK timeouts?

from cmdop import ConnectionConfig
 
config = ConnectionConfig(
    connect_timeout_seconds=15,   # Connection establishment
    request_timeout_seconds=60,   # Individual requests
    stream_timeout_seconds=0,     # Streaming (0=unlimited)
)

How do I configure CLI timeouts?

# Per-command timeout
cmdop exec server "slow-command" --timeout 300
 
# Global timeout
export CMDOP_REQUEST_TIMEOUT=60

How do I benchmark CMDOP performance?

How do I run a basic benchmark?

# Command execution latency
for i in {1..10}; do
  time cmdop exec server "true" 2>&1 | grep real
done
 
# File transfer speed
dd if=/dev/urandom of=testfile bs=1M count=100
time cmdop files put testfile server:/tmp/
cmdop exec server "rm /tmp/testfile"
rm testfile

How do I benchmark with the SDK?

import time
from cmdop import CMDOPClient
 
client = CMDOPClient.remote(api_key="...")
 
# Measure execution
times = []
for _ in range(100):
    start = time.time()
    client.terminal.execute("true")
    times.append(time.time() - start)
 
print(f"Average: {sum(times)/len(times)*1000:.1f}ms")
print(f"P99: {sorted(times)[98]*1000:.1f}ms")

How do I monitor CMDOP performance?

How do I set up Prometheus metrics?

If using self-hosted with metrics enabled:

# prometheus.yml
scrape_configs:
  - job_name: 'cmdop'
    static_configs:
      - targets: ['cmdop-control-plane:9090']

Key metrics:

  • cmdop_session_latency_seconds
  • cmdop_command_duration_seconds
  • cmdop_file_transfer_bytes

How do I set up a Grafana dashboard?

Import dashboard from: https://grafana.com/grafana/dashboards/cmdop

Last updated on
Overview