Common Recovery Steps

Ordered from least to most disruptive. Work through these in order — stop when the problem is resolved.

Step 1: Check logs

Before touching any services, read the logs. Most problems are self-diagnosing.

# Last 50 lines from the mixer (most issues start here)
journalctl --user -u shiloh-mixer -n 50

# Full recent window
journalctl --user -u shiloh-mixer --since "5 min ago"

# All services together
journalctl --user -u shiloh-mixer -u shiloh-web-relay -u mixer_web --since "5 min ago"

Look for ERROR, WARN, REJECT, or panic lines. Cross-reference with the specific troubleshooting pages:

  • Mixer — JACK, config, permissions, xruns
  • Broadcaster — registration, rejects
  • Relay — no audio, crackling
  • Browser — WHEP, ICE, HTTPS
  • MIDI — device, channel, debounce

Step 2: Restart individual services

Restart only the failing service. The others continue running.

# Mixer only (reconnects broadcasters and relays automatically)
systemctl --user restart shiloh-mixer

# Web relay only (browser listener reconnects)
systemctl --user restart shiloh-web-relay

# Phoenix UI only (LiveView clients reconnect automatically)
systemctl --user restart mixer_web

After restarting the mixer, broadcasters and relays reconnect automatically (with exponential backoff, resetting after a 10-second stable session). Wait ~30 seconds before escalating.

Step 3: Restart JACK / PipeWire

If the mixer is crashing repeatedly or JACK ports are in a bad state (common after a PipeWire upgrade), restart the audio stack.

systemctl --user restart pipewire pipewire-pulse wireplumber
# Wait a few seconds for the graph to settle
systemctl --user restart shiloh-mixer

Check the graph is healthy after restart:

timeout 2 pw-top -b -n 3 | head -10

Look for at least one node with status R and non-zero QUANT/RATE.

VU meters still at 0 after restart?

This is the driverless-graph symptom. A hardware ALSA node must be clocking the graph. Add the WirePlumber keep-awake rule from OPERATIONS.md and restart WirePlumber.

Step 4: Rebuild the desktop-audio null-sink

If PulseAudio apps (Firefox, VLC) are stuck on “Unknown Output” or pavucontrol shows corked sink-inputs, the desktop-audio null-sink is broken.

# Verify the symptom
journalctl --user -u wireplumber --since "2 min ago" | grep "pending linkable"

# Fix: restart the PipeWire stack
systemctl --user restart pipewire pipewire-pulse wireplumber

# If still broken, clear WirePlumber's stream-restore state
cp ~/.local/state/wireplumber/restore-stream{,.bak.$(date +%s)}
: > ~/.local/state/wireplumber/restore-stream
systemctl --user restart wireplumber

Verify the sink is back:

pactl list sinks short | grep desktop-audio

Step 5: Full system restart sequence

Use this as a last resort when individual service restarts haven’t helped. Follow the exact order — dependency ordering matters.

# 1. Stop everything
systemctl --user stop mixer_web shiloh-web-relay shiloh-mixer

# 2. Restart the audio stack
systemctl --user restart wireplumber pipewire-pulse pipewire
sleep 3

# 3. Verify the graph is running
timeout 2 pw-top -b -n 3 | head -5

# 4. Start the mixer first (broadcasters and relays can't connect until it's up)
systemctl --user start shiloh-mixer
sleep 2

# 5. Check it's healthy
systemctl --user is-active shiloh-mixer
journalctl --user -u shiloh-mixer -n 10

# 6. Start the remaining services
systemctl --user start shiloh-web-relay mixer_web

After a full restart, broadcaster and relay clients reconnect within 30 seconds on their own backoff timers. The mixer UI LiveView reconnects automatically when mixer_web is back.

State file cleanup

If ~/mixer-state/ has stale JSON (e.g. from a crashed session), the mixer will overwrite files on next start. No manual cleanup is needed unless you want to reset to defaults:

# Reset persisted relay assignments (the only file actually written by broadcast.rs)
rm -f ~/mixer-state/relay-assignments.json
systemctl --user restart shiloh-mixer

Note: gains.json is not written by the mixer — gains are in-memory only.
Scenes are stored via DETS by mixer_web as scenes.dets, not as JSON by
shiloh-mixer. Meters and sessions JSON are written at runtime and not
persisted across restarts.