Monitoring & Logs

Log locations

All services write to the systemd journal. There are no separate log files by default.

# Live tails
journalctl --user -u shiloh-mixer -f
journalctl --user -u shiloh-web-relay -f
journalctl --user -u mixer_web -f

# Combined view
journalctl --user -u shiloh-mixer -u shiloh-web-relay -u mixer_web -f

# Time-bounded lookback
journalctl --user -u shiloh-mixer --since "30 min ago"

# Last N lines
journalctl --user -u shiloh-mixer -n 100

Log verbosity

Set via the RUST_LOG environment variable. Default is info.

Change verbosity temporarily:

systemctl --user edit shiloh-mixer
# Add or update:
# [Service]
# Environment=RUST_LOG=debug
systemctl --user daemon-reload && systemctl --user restart shiloh-mixer

Useful scoped log targets:

# Debug only ingest, info for everything else
RUST_LOG=shiloh_mixer::ingest=debug,info

# Protocol decoder debug
RUST_LOG=shiloh_protocol=debug,info

Log rotation

The systemd journal rotates automatically based on disk use. On most systems the default max journal size is 10% of filesystem or 4 GB. To cap it for a low-disk server:

# In /etc/systemd/journald.conf (or a drop-in)
[Journal]
SystemMaxUse=500M

Then systemctl restart systemd-journald.

Verbose stats output (--verbose)

The broadcaster and relay clients support a --verbose flag that prints per-second stats:

shiloh-broadcaster connect --server HOST:5005 --name studio --channels 2 --verbose
# → [stat] 375 pps  underruns=0  ring_full=0

underruns > 0 means the local JACK buffer is too small or the CPU is overloaded. See Performance Tuning.

ring_full > 0 (visible in mixer logs as ingest_drop) means the mixer’s RT thread is not consuming ingest samples fast enough.

State file health check

The mixer writes state files to ~/mixer-state/ at runtime:

# Watch mtime — should advance every ~100 ms
watch -n 1 'stat -c "%y %n" ~/mixer-state/meters.json'

# Quick check: is meters.json recent?
python3 -c "
import os, time
age = time.time() - os.path.getmtime(os.path.expanduser('~/mixer-state/meters.json'))
print(f'meters.json age: {age:.1f}s', 'OK' if age < 2 else 'STALE')
"

If meters.json is not updating, the mixer’s meters thread is stuck. Restart shiloh-mixer.

Heartbeat watchdog

The mixer includes an internal heartbeat that ticks the meters.json file. You can hook into this from a simple shell watchdog:

# Simple shell watchdog (run in a tmux or as a cron job)
while true; do
  age=$(( $(date +%s) - $(stat -c %Y ~/mixer-state/meters.json) ))
  if [ "$age" -gt 5 ]; then
    echo "$(date): meters.json stale (${age}s), restarting mixer"
    systemctl --user restart shiloh-mixer
  fi
  sleep 3
done

Systemd OnFailure alerting

Systemd can run a unit when another fails. To send a notification (example uses curl to post to a webhook):

Create ~/.config/systemd/user/notify-failure@.service:

[Unit]
Description=Notify on failure of %i

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'curl -s -X POST https://hooks.example.com/alert \
  -H "Content-Type: application/json" \
  -d "{\"text\": \"shiloh: %i failed on $(hostname)\"}"'

Add to shiloh-mixer.service (via drop-in):

systemctl --user edit shiloh-mixer
# Add:
# [Unit]
# OnFailure=notify-failure@shiloh-mixer.service
systemctl --user daemon-reload

Reading sessions.json

~/mixer-state/sessions.json lists all currently connected relay and broadcaster sessions. Useful for scripting:

# Count connected relays
python3 -c "import json; s=json.load(open(os.path.expanduser('~/mixer-state/sessions.json'))); print(len(s))"

# Names of connected clients
jq '.[].name' ~/mixer-state/sessions.json

Checking the web relay’s PeerConnections

shiloh-web-relay logs each browser connection lifecycle:

journalctl --user -u shiloh-web-relay | grep "PC state"
# PC state: connected   (ICE succeeded, audio flowing)
# PC state: failed      (ICE failed, probably NAT/firewall)
# PC state: closed      (browser tab closed)

Count active browser connections:

journalctl --user -u shiloh-web-relay --since "10 min ago" | \
  grep -c "PC state: connected"