Maintaining your radio on air#
Radiotomate itself is quite stable and made to stream continuously, but sometimes the universe doesn't help: machines/hard drives/networks/buildings might break down or catch fire. This section contains some advice to keep the radio running, even if that means "restarting from a new machine".
Checking containers are running#
Radiotomate is made to run in podman containers launched as a system service thanks to podman's integration to systemd, quadlets).
You can check containers are running with podman ps -a.
The typical result should look like:
$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da9ad3bcdbfe localhost/podman-pause:5.4.2-1766335321 14 minutes ago Up 14 minutes 5a96db1d31b2-service
65ba145efb71 localhost/podman-pause:5.4.2-1766335321 14 minutes ago Up 14 minutes 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp c09122c475cd-infra
cb17f5fd53dd docker.io/martinkirch/radiotomate-playout:latest 14 minutes ago Up 14 minutes 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp radiotomate-playout
66c761d7eafc docker.io/martinkirch/radiotomate-webapps:latest 14 minutes ago Up 14 minutes 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp radiotomate-interface
44a5068f61a0 docker.io/martinkirch/radiotomate-webapps:latest 14 minutes ago Up 14 minutes 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp radiotomate-scheduler
6d0260419bfc docker.io/martinkirch/radiotomate-webapps:latest 14 minutes ago Up 14 minutes 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp radiotomate-dropbox
If containers are stopped you can restart them with
systemctl --user restart radiotomate.service
You can also stop them with
systemctl --user stop radiotomate.service
When containers don't start#
After an update, or after a change in the configuration, you may notice that the system
is not starting, or not completely. podman ps -a may report short "status" as follows:
$ podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fed10cbcf33c localhost/podman-pause:5.4.2-1766335321 About a minute ago Up About a minute 5a96db1d31b2-service
f0c33a3e1e7c localhost/podman-pause:5.4.2-1766335321 About a minute ago Up About a minute 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp e88a1371e1a1-infra
d2c2b5154aed docker.io/martinkirch/radiotomate-playout:latest About a minute ago Up 1 second 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp radiotomate-playout
419231c669ff docker.io/martinkirch/radiotomate-webapps:latest About a minute ago Up About a minute 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp radiotomate-interface
7dad7e826c72 docker.io/martinkirch/radiotomate-webapps:latest About a minute ago Up About a minute 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp radiotomate-scheduler
28b5e7824288 docker.io/martinkirch/radiotomate-webapps:latest About a minute ago Up About a minute 0.0.0.0:5811->6811/tcp, 0.0.0.0:6800->6800/tcp radiotomate-dropbox
Above, it seems the radiotomate-playout container is stuck in a restart loop
(the container name is on the rightmost column).
The system keeps restarting the container but it always fails on start.
This is typically caused by a configuration error.
In the configuration file, radiotomate.yaml, check that blocks are aligned correctly,
with spaces instead of stabs, and that they match examples from
the configuration reference.
You may also restore a previous version of the configuration and try again -
always keep a copy of a working configuration before editing.
You may also see more detailed error messages in:
podman logs radiotomate-playout
If something goes wrong but all containers are running, have a look at application's logs instead.
Logs#
Each of the 4 processes behind Radiotomate have its own journal,
a text file where it notes what's happening.
By default these files are all located in the radiotomate_data folder.
Their names are:
playout.log, written by the playout process, the one who actually play sounds. It is a good place to start if you hear something unexpected on air.scheduler.log, written by the scheduler process, the one who pushes sounds to playout on time (for sound carts or stream relays) or when needed (auto-DJ). It also stores the metadata of sounds as they're played and relays it (if enabled).interface.log, written by the Web interface most users will be interacting with.audit.log, written by the Web interface as well, records every action performed by authenticated users: their username, the HTTP verb and the path of each request.dropbox.loglogs what happens to files that appear in the music dropbox folder.
Before filing a bug or asking for help, please look into these logs for errors and copy errors you think relevant in your message.
Ensuring the system stays on#
The very first potential interruption come from the power supply. If you're running Radiotomate on your own hardware, take care to configure it so it automatically turns on when power comes back (this is a classic BIOS option).
You will also want to update your machine regularly, which will imply to reboot it and interrupt the stream for a long minute. We advise you prepare an alternative stream source for such operations.
By default the installer uses quadlets,
so Radiotomate processes are considered as user-level systemd services.
To ensure that they are started at boot time,
radiotomate will loginctl enable-linger the user running radiotomate.
This also ensures that it restarts automatically in case of crash:
if a process is not started it might be blocked by a configuration error.
The start sound#
Radiotomate restarts automatically, but that should not happen regularly. If it restarts often then something is wrong, but it restarts so fast that it might be hard to notice (it happened): that's why it has a start sound.
By default, radiotomate starts by sounding a horn,
from the file radiotomate_data/default_start_sound.wav.
Change it as you like.
If you hear it too often,
and especially when you're not restarting the application yourself,
it is time to look at the logs.
Backing up#
Backing up starts at installation time: we advise you copy-paste each command and action you made, in a separate file. This will ease your next installation. It will be more comfortable to do a long-term install after a test phase. It will be a life saver when your machine completely breaks and you have to reinstall in a hurry.
By default Radiotomate puts all its data into one folder: radiotomate_data.
This is the folder you want to back-up.
Keep a complete copy of this folder on another machine.
If possible, keep a second complete copy in another building.
The minimal and fast way to back-up is to use rsync.
Start a terminal in the parent folder of Radiotomate's data root and invoke it as follows:
rsync -av radiotomate_data /media/hdd2/radio_backup
If you'd like to have the ability to restore your data at a precise point in time without keeping 50 complete copies of your data, you should consider a specialized backup program like restic or borg.
Restoring#
Restoring takes two steps:
- re-install the application (Radiotomate's pod and containers)
- restore your radio's data
Restoring the application requires you to launch the installer with the same parameters you used the previous time.
When this is done, stop the new (empty) radio entirely:
systemctl --user stop radiotomate.service
Then copy your radio's data folder back to the new data root. When this is done, restart the radio:
systemctl --user start radiotomate.service
We advise you try to restore regularly, using any spare machine you can find. This will ensure that your back-up is working correctly, which is important because you don't want to find out what's wrong when a real emergency happens. It will also bring you a test machine, which can be interesting if you want to experiment new configurations, new programs' schedules or a new version of Radiotomate.