Output going randomly offline

Hi, we can’t make the output work properly. We have shows scheduled but the Onair light is off. We can see the “now playing” metadata correctly, but the stream metadata shows as “standby” when I play it in an external player and there’s no sound.
When there’s a new track starting to play, the “Onair” light comes on but just for a few seconds.
It seems to work fine with webstreams.
I found smtg that might be related in issue #756 but from what I understood that should be solved.

I’m seeing this issue in a fresh LT vb65ded9fc@ubuntu18 install where I recovered a backup db and tracks from an alpha8 install in ubuntu16 that has been on air for a while.

Here are some logs:

[cross_5844:3] No next track ready yet.
2021/01/31 17:00:09 [lang:3] URL now http://ainelstream.net:8768/ (change: false)
2021/01/31 17:00:09 [lang:3] New track inside HTTP stream
2021/01/31 17:00:09 [lang:3] status: stopped
2021/01/31 17:00:09 [lang:3] need to cross: false
2021/01/31 17:00:09 [lang:3] remaining inf sec before, 0. sec after
2021/01/31 17:00:09 [clock.wallclock_main:2] We must catchup 2.84 seconds!
2021/01/31 17:00:10 [dummy(dot)2:3] Source failed (no more tracks) stopping output…
2021/01/31 17:09:51 [server:3] New client: localhost.

2021-01-31 17:42:51,393 [pypoliqqueue] [INFO ] waiting 0s until next scheduled item
2021-01-31 17:45:31,738 [pypofetch] [INFO ] Queue timeout. Fetching schedule manually
2021-01-31 17:45:31,867 [pypoliquidsoap] [INFO ] Need to add items to Liquidsoap now: {67883}
2021-01-31 17:45:31,867 [pypofetch] [INFO ] Loop #257
2021-01-31 17:46:11,676 [pypoliquidsoap] [WARNI] File /var/tmp/airtime/pypo/cache/scheduler/701.mp3 did not become ready in less than 5 seconds. Skipping…
2021-01-31 17:46:11,676 [pypoliqqueue] [INFO ] waiting 76.323464s until next scheduled item
2021-01-31 17:48:52,145 [pypoliquidsoap] [WARNI] File /var/tmp/airtime/pypo/cache/scheduler/701.mp3 did not become ready in less than 5 seconds. Skipping…

And the part in airtime.conf that gets me a bit confused.

api_client = ‘airtime’
cache_dir = ‘/var/tmp/airtime/pypo/cache/’
file_dir = ‘/var/tmp/airtime/pypo/files/’
tmp_dir = ‘/var/tmp/airtime/pypo/tmp/’
cache_base_dir = ‘/var/tmp/airtime/pypo’
log_base_dir = ‘/var/log/airtime’
pypo_log_dir = ‘/var/log/airtime/pypo’
liquidsoap_log_dir = ‘/var/log/airtime/pypo-liquidsoap’

Thanks for any tips!

I found something more in the logs:

2021-01-31 20:32:38,893 [pypofetch] [INFO ]  Handling command: update_schedule
2021-01-31 20:32:38,895 [pypofetch] [ERROR]  end - start length: 196.0
2021-01-31 20:32:38,896 [pypofetch] [ERROR]  cue_out - cue_in length: 194.901
2021-01-31 20:32:38,896 [pypofetch] [ERROR]  Two lengths are not equal!!!
2021-01-31 20:32:38,899 [pypofetch] [INFO ]  New timeout: 480

I am also having the same problem and i am using ubuntu 18.04 and i install using the cloned git.

any help please.

Did it ever work? or is it happening since you installed it?

What about webstreams? do they work?

Never worked since I installed it.
Webstreams works fine. I don’t know what the problem is.

So I guess we can say is not “randomly” going offline.
Whenever there are tracks involved it fails, but with webstreams works fine.

I’m guessing it could be an ubuntu 18 update that broke something since it was working fine a few weeks ago… I’m not sure if it started happening after an update though, I’m just saying because I’m trying a new install now and I use a base ubuntu lxc container I made a couple of months ago, and when I updated it these are the packages upgraded:

The following NEW packages will be installed:
The following packages will be upgraded:
  apt apt-transport-https apt-utils ca-certificates curl distro-info-data krb5-locales libapt-inst2.0 libapt-pkg5.0 libaudit-common libaudit1 libc-bin
  libc-dev-bin libc6 libc6-dev libcryptsetup12 libcurl3-gnutls libcurl4 libfreetype6 libgssapi-krb5-2 libk5crypto3 libkrb5-3 libkrb5support0 libldap-2.4-2
  libldap-common libnetplan0 libnss-systemd libopenexr22 libp11-kit0 libpam-systemd libperl5.26 libpq5 libproxy1v5 libpulse0 libpulsedsp libsasl2-2
  libsasl2-modules libsasl2-modules-db libssl-dev libssl1.0.0 libssl1.1 libsystemd0 libudev1 libwavpack1 linux-libc-dev locales multiarch-support netplan.io
  nplan openssl perl perl-base perl-modules-5.26 php-pear postgresql-10 postgresql-client-10 pulseaudio pulseaudio-utils python-pip-whl python3-cryptography
  python3-pip python3-xdg rabbitmq-server sudo systemd systemd-sysv tar tzdata udev unzip
70 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

There are a couple of pulseaudio and rabbit packages…
Will tell later how it goes with this new install…

Any hints please? where to read, check, etc?

In case this helps libretime-analyzer takes a quite a while to restart (1 or 2 min). I found this in the logs:

Feb  1 21:14:09  systemd[1]: libretime-analyzer.service: State 'stop-sigterm' timed out. Killing.
Feb  1 21:14:09  systemd[1]: libretime-analyzer.service: Killing process 4557 (airtime_analyze) with signal SIGKILL.
Feb  1 21:14:09  systemd[1]: libretime-analyzer.service: Main process exited, code=killed, status=9/KILL
Feb  1 21:14:09  systemd[1]: libretime-analyzer.service: Failed with result 'timeout'.
Feb  1 21:14:09  systemd[1]: Stopped LibreTime Media Analyzer Service.
Feb  1 21:14:09  systemd[1]: libretime-analyzer.service: Failed to reset devices.list: Operation not permitted
Feb  1 21:14:09  systemd[1]: Started LibreTime Media Analyzer Service.
Feb  1 21:14:10  airtime_analyzer[4822]: 2021-02-01 21:14:10,255 [message_listener] [INFO ]   Listening for messages...

Sorry to bump, I think the proper place for this should be LT’s git… I’ve just opened an issue here.