Troubleshooting

Common issues

Simulator binary not found

Error: erst-sim not found

Symptom: Error message indicating the simulator binary cannot be located.Causes:

Simulator binary not built
Binary not in expected location
Incorrect path configuration

Solution:

Build the simulator

cd simulator
cargo build --release
cd ..

Verify binary exists

ls -la simulator/target/release/erst-sim

Set environment variable

export ERST_SIM_PATH=/path/to/erst-sim

Or use flag override

erst debug --sim-path /path/to/erst-sim <tx-hash>

Invalid JSON from simulator

Error: Invalid JSON response

Symptom: Malformed JSON response from the Rust simulator.Causes:

Malformed simulation request
XDR encoding issues
Simulator crash or panic

Solution:

Check XDR encoding:

# Verify XDR is valid Base64
echo "AAAA..." | base64 -d > /dev/null

Enable debug mode:
```
erst debug --debug <tx-hash>
```
Check simulator logs: Look for panic messages or stack traces in stderr.

Validate request format: Ensure the request matches the expected schema:

{
  "envelope_xdr": "valid-base64",
  "result_meta_xdr": "valid-base64",
  "ledger_entries": {}
}

Ledger entry decode error

Error: Failed to decode ledger entry

Symptom: Error decoding ledger state snapshot.Causes:

Incorrect ledger state snapshot
Wrong ledger sequence number
Corrupted ledger data

Solution:

Verify transaction block number:

curl https://horizon-testnet.stellar.org/transactions/<tx-hash>

Check ledger sequence: Ensure you’re fetching state from the correct ledger:

# Get transaction ledger
TX_LEDGER=$(curl -s https://horizon-testnet.stellar.org/transactions/<tx-hash> | jq .ledger)

# Fetch ledger entries at that sequence
curl "https://soroban-testnet.stellar.org/" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getLedgerEntries",
    "params": {
      "keys": ["..."],
      "ledgerSeq": '$TX_LEDGER'
    }
  }'

Validate XDR format: Test XDR decoding separately:

stellar xdr decode --type LedgerEntry "AAAA..."

Horizon API timeout

Error: Horizon request timeout

Symptom: Network timeout when fetching transaction data.Causes:

Network latency or connectivity issues
Horizon endpoint down or slow
Rate limiting

Solution:

Check Horizon status:

curl https://horizon-testnet.stellar.org/

Try different network:

erst debug --network futurenet <tx-hash>

Check network connectivity:
```
ping horizon-testnet.stellar.org
```

Verify rate limits: Check response headers for rate limit information:

curl -I https://horizon-testnet.stellar.org/transactions/<tx-hash>

Use custom timeout: Increase request timeout if needed (future feature).

Debugging workflow issues

Transaction not found

Error: Transaction not found on network

Symptom: Transaction hash not found in Horizon API.Causes:

Invalid transaction hash
Transaction on different network
Very recent transaction not yet indexed

Solution:

Verify transaction hash: Check the hash is 64 hexadecimal characters.

Check correct network:

# Try testnet
erst debug --network testnet <tx-hash>

# Try mainnet
erst debug --network mainnet <tx-hash>

Wait for indexing: For very recent transactions, wait 10-30 seconds and retry.

Query directly:

curl https://horizon-testnet.stellar.org/transactions/<tx-hash>

Simulation produces different result

Issue: Simulation result differs from on-chain result

Symptom: Local simulation produces different output than the actual on-chain transaction.Causes:

State snapshot timing mismatch
Protocol version difference
Ledger state inconsistency

Solution:

Verify exact ledger state: Ensure ledger entries are from the same sequence as the transaction.

Check protocol version:

# Get ledger protocol version
curl https://horizon-testnet.stellar.org/ledgers/<ledger-seq>

Compare transaction metadata:

# Extract result meta from Horizon
curl https://horizon-testnet.stellar.org/transactions/<tx-hash> \
  | jq -r .result_meta_xdr

Enable verbose logging:
```
erst debug --debug --verbose <tx-hash>
```

Interactive trace issues

Error: Interactive mode crashes or displays incorrectly

Symptom: Terminal UI crashes, displays incorrectly, or keyboard shortcuts don’t work.Causes:

Terminal compatibility issues
Insufficient terminal size
Color support problems

Solution:

Check terminal size:

# Interactive mode requires minimum 80x24
tput cols  # Should be >= 80
tput lines # Should be >= 24

Test terminal compatibility:

# Try with different TERM setting
TERM=xterm-256color erst debug -i <tx-hash>

Disable colors if needed:
```
NO_COLOR=1 erst debug -i <tx-hash>
```

Use non-interactive mode:

# Fall back to standard output
erst debug <tx-hash>

Performance issues

Slow simulation execution

Issue: Simulation takes too long

Symptom: Simulation execution time is unacceptably slow.Causes:

Large contract state
Complex transaction with many operations
Network latency fetching data
Inefficient contract code

Solution:

Profile the execution:
```
erst debug --profile <tx-hash>
```
Check state fetch time: Enable timing metrics:
```
erst debug --debug <tx-hash>
```
Optimize network requests:
- Use closer Horizon endpoint
- Cache ledger entries locally
- Use JSON-RPC for batch queries

Analyze flamegraph: Generate and review interactive flamegraph:

erst debug --profile --profile-format html <tx-hash>
# Open <tx-hash>.flamegraph.html in browser

High memory usage

Issue: Excessive memory consumption

Symptom: Erst process uses too much memory.Causes:

Large ledger state
Memory leak in simulator
Many concurrent operations

Solution:

Check memory usage:

# Monitor memory during execution
/usr/bin/time -v erst debug <tx-hash>

Limit concurrent operations: Reduce parallelism if using daemon mode.

Check for leaks:

# Run with valgrind (if available)
valgrind --leak-check=full ./erst debug <tx-hash>

Report memory leaks: If persistent, report with memory profile:

go test -memprofile=mem.prof ./...
go tool pprof mem.prof

HSM and signing issues

PKCS#11 module not found

Error: Failed to load PKCS#11 module

Symptom: Cannot load HSM PKCS#11 library.Solution:

# Verify module path
ls -la /usr/lib/libykcs11.so

# Install YubiKey manager
sudo apt-get install yubikey-manager
# or
brew install yubikey-manager

# Set correct path
export ERST_PKCS11_MODULE=/usr/lib/x86_64-linux-gnu/libykcs11.so

HSM not detected

Error: No slots found

Symptom: HSM device not detected.Solution:

# Check if HSM is connected
yubico-piv-tool -a status

# Verify permissions
sudo usermod -a -G plugdev $USER
# Log out and back in

# List available slots
pkcs11-tool --module /usr/lib/libykcs11.so -L

PIN incorrect

Error: Failed to login: 0x1000

Symptom: HSM PIN authentication failed.Solution:

# Verify PIN
yubico-piv-tool -a verify-pin -P 123456

# Reset PIN if needed (requires PUK)
yubico-piv-tool -a change-pin -P 123456 -N 654321

# Unblock PIN with PUK
yubico-piv-tool -a unblock-pin -P 12345678 -N 123456

Key not found in HSM

Error: Private key not found in HSM

Symptom: Specified key not found on HSM.Solution:

# List available keys
yubico-piv-tool -a list

# List objects in token
pkcs11-tool --module /usr/lib/libykcs11.so -O

# Generate key if needed
yubico-piv-tool -a generate -s 9a -o pubkey.pem -A ECCP256

# Verify key label matches
pkcs11-tool --module /usr/lib/libykcs11.so -O --label "MySigningKey"

Metrics and observability issues

Metrics not appearing

Issue: Prometheus metrics not exposed

Symptom: /metrics endpoint returns empty or no metrics.Solution:

Verify daemon is running:
```
curl http://localhost:8080/health
```
Check metrics endpoint:
```
curl http://localhost:8080/metrics
```
Trigger some activity: Metrics appear only after first use:
```
erst debug <tx-hash>
```
Check Prometheus scrape config: Verify target is configured correctly in prometheus.yml.

Traces not appearing in Jaeger

Issue: OpenTelemetry traces not visible

Symptom: No traces appearing in Jaeger UI.Solution:

Verify tracing is enabled:
```
erst debug --tracing <tx-hash>
```
Check OTLP endpoint:
```
curl http://localhost:4318/v1/traces
```
Verify Jaeger is running:
```
docker ps | grep jaeger
```

Test with curl: Send test trace to verify endpoint:

curl -X POST http://localhost:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans": []}'

Check for errors: Enable debug logging:

OTEL_LOG_LEVEL=debug erst debug --tracing <tx-hash>

Validation errors

Schema validation failures

Error: Validation failed for simulation request

Symptom: Request validation errors before simulation.Causes:

Missing required fields
Invalid Base64 encoding
Malformed XDR data

Solution:

Check required fields:

{
  "envelope_xdr": "required",
  "result_meta_xdr": "required",
  "ledger_entries": {}
}

Validate Base64 encoding:

echo "AAAA..." | base64 -d > /dev/null && echo "Valid" || echo "Invalid"

Test XDR decoding:

stellar xdr decode --type TransactionEnvelope "AAAA..."

Enable strict validation: Use validator in strict mode for detailed errors:
```
erst debug --strict-validation <tx-hash>
```

Getting help

If you encounter an issue not covered in this guide:

Check GitHub Issues: Search for similar problems at https://github.com/dotandev/hintents/issues
Enable debug mode: Run with --debug flag to get detailed logs
Collect diagnostics: Include version info, error messages, and reproduction steps
Report the issue: Create a new GitHub issue with full details

Diagnostic information

When reporting issues, include:

# Version information
erst version

# System information
uname -a
go version
cargo --version

# Error output
erst debug --debug <tx-hash> 2>&1 | tee debug.log

# Environment variables
env | grep ERST_

Never share private keys, HSM PINs, or sensitive credentials when reporting issues. Redact any sensitive information from logs and diagnostics.

Get Started

Core Features

Guides

Configuration

Advanced

Troubleshooting

Common issues

Simulator binary not found

Invalid JSON from simulator

Ledger entry decode error

Horizon API timeout

Debugging workflow issues

Transaction not found

Simulation produces different result

Interactive trace issues

Performance issues

Slow simulation execution

High memory usage

HSM and signing issues

PKCS#11 module not found

HSM not detected

PIN incorrect

Key not found in HSM

Metrics and observability issues

Metrics not appearing

Traces not appearing in Jaeger

Validation errors

Schema validation failures

Getting help

Diagnostic information

Get Started

Core Features

Guides

Configuration

Advanced

Documentation Index

​Common issues

​Simulator binary not found

​Invalid JSON from simulator

​Ledger entry decode error

​Horizon API timeout

​Debugging workflow issues

​Transaction not found

​Simulation produces different result

​Interactive trace issues

​Performance issues

​Slow simulation execution

​High memory usage

​HSM and signing issues

​PKCS#11 module not found

​HSM not detected

​PIN incorrect

​Key not found in HSM

​Metrics and observability issues

​Metrics not appearing

​Traces not appearing in Jaeger

​Validation errors

​Schema validation failures

​Getting help

​Diagnostic information

Common issues

Simulator binary not found

Invalid JSON from simulator

Ledger entry decode error

Horizon API timeout

Debugging workflow issues

Transaction not found

Simulation produces different result

Interactive trace issues

Performance issues

Slow simulation execution

High memory usage

HSM and signing issues

PKCS#11 module not found

HSM not detected

PIN incorrect

Key not found in HSM

Metrics and observability issues

Metrics not appearing

Traces not appearing in Jaeger

Validation errors

Schema validation failures

Getting help

Diagnostic information