Server crash after disk full with error logs from latest git

Hi, this is a fresh LT install from git (v264d24313, ubuntu18 inside an LXC container).

The container crashed 2 times because of disk (30GB) getting filled with playout errors in /var/log/syslog.
The install was running smoothly for a couple of hours and then it crashed (happened the same thing in two different installs).

Here there’s a few logs (I’ll try to share some more later)

May 30 15:22:36 xtrema airtime-playout[95]:     'ends__range': ('{str_current}Z,{str_end}Z'.format(str_current, str_end)),
May 30 15:22:36 xtrema airtime-playout[95]: KeyError: 'str_current'
May 30 15:22:36 xtrema airtime-playout[95]: 2021-05-30 15:22:36,940 [pypofetch] [ERROR]  Unable to fetch schedule
May 30 15:22:36 xtrema airtime-playout[95]: 2021-05-30 15:22:36,940 [pypofetch] [ERROR]  'str_current'
May 30 15:22:36 xtrema airtime-playout[95]: Traceback (most recent call last):
May 30 15:22:36 xtrema airtime-playout[95]:   File "/usr/local/lib/python3.6/dist-packages/airtime_playout-1.0-py3.6.egg/pypo/pypofetch.py", line 440, in manual_schedule_fetch
May 30 15:22:36 xtrema airtime-playout[95]:     self.schedule_data = self.api_client.get_schedule()
May 30 15:22:36 xtrema airtime-playout[95]:   File "/usr/local/lib/python3.6/dist-packages/api_clients-2.0.0-py3.6.egg/api_clients/version2.py", line 51, in get_schedule
May 30 15:22:36 xtrema airtime-playout[95]:     'ends__range': ('{str_current}Z,{str_end}Z'.format(str_current, str_end)),
May 30 15:22:36 xtrema airtime-playout[95]: KeyError: 'str_current'
May 30 15:22:36 xtrema airtime-playout[95]: 2021-05-30 15:22:36,940 [pypofetch] [INFO ]  Loop #5667510
May 30 15:22:36 xtrema airtime-playout[95]: 2021-05-30 15:22:36,941 [pypofetch] [INFO ]  Queue timeout. Fetching schedule manually
May 30 15:22:36 xtrema airtime-playout[95]: 2021-05-30 15:22:36,941 [pypofetch] [ERROR]  Unable to fetch schedule
May 30 15:22:36 xtrema airtime-playout[95]: 2021-05-30 15:22:36,941 [pypofetch] [ERROR]  'str_current'
May 30 15:22:36 xtrema airtime-playout[95]: Traceback (most recent call last):
May 30 15:22:36 xtrema airtime-playout[95]:   File "/usr/local/lib/python3.6/dist-packages/airtime_playout-1.0-py3.6.egg/pypo/pypofetch.py", line 440, in manual_schedule_fetch
May 30 15:22:36 xtrema airtime-playout[95]:     self.schedule_data = self.api_client.get_schedule()
May 30 15:22:36 xtrema airtime-playout[95]:   File "/usr/local/lib/python3.6/dist-packages/api_clients-2.0.0-py3.6.egg/api_clients/version2.py", line 51, in get_schedule
May 30 15:22:36 xtrema airtime-playout[95]:     'ends__range': ('{str_current}Z,{str_end}Z'.format(str_current, str_end)),
May 30 15:22:36 xtrema airtime-playout[95]: KeyError: 'str_current'

Any hints?
thanks!

Could it be an issue with the show being empty? we’ve been using the server mainly for live transmissions so the shows were empty…
I’m testing now adding tracks to the shows even thought we’re using the live input and the log error seem to have dissapeared.
Now I’m seeing the usual:

May 31 11:59:07 xtrema airtime-playout[97]: 2021-05-31 07:59:07,935 [pypofetch] [INFO ]  Handling command: update_schedule
May 31 11:59:07 xtrema airtime-playout[97]: 2021-05-31 07:59:07,950 [pypofetch] [INFO ]  New timeout: 400
May 31 11:59:07 xtrema airtime-playout[97]: 2021-05-31 07:59:07,950 [pypofetch] [INFO ]  Loop #163
May 31 11:59:07 xtrema airtime-liquidsoap[74]: 2021/05/31 07:59:07 [server:3] New client: localhost.
May 31 11:59:07 xtrema airtime-liquidsoap[74]: 2021/05/31 07:59:07 [lang:3] dynamic_source.get_id
May 31 11:59:07 xtrema airtime-liquidsoap[74]: 2021/05/31 07:59:07 [server:3] Client localhost disconnected.
May 31 11:59:07 xtrema airtime-playout[97]: 2021-05-31 07:59:07,950 [pypoliqqueue] [INFO ]  New schedule received: {'2021-05-31-12-03-32': {'id': 4, 'type': 'file', 'metadata': {'id': 4, 'name': '', 'mime': 'audio/mp3', 'ftype': 'audioclip'......

Could it be the empty shows (or even not having shows scheduled at all) the problem? can the log level be reduced so it doesn’t fill up the disk?
thanks

For the time being we are scheduling random tracks even though we are live streaming to see if it helps.

Fixed in

Thanks a lot @paddatrapper!!

1 Like