A few days ago I uninstalled Libretime and did a fresh reinstall from the latest git version (3.0.0-alpha.9-209-gcf75b14e2).
I’ve tried everything that I can think of in Libretime and Icecast settings, plus spent a lot of time playing with apache configs and modifying /etc/hosts, but no matter what I do, when creating a new show I get the same error in pypo.log when adding any content:
2021-07-12 18:59:33,367 [pypofile] [ERROR] <Response [401]>
2021-07-12 18:59:33,370 [pypofile] [ERROR] Could not copy from https://kbfgstream.org/rest/media/9 to /var/tmp/airtime/pypo/cache/scheduler/9.mp3
2021-07-12 18:59:33,371 [pypofile] [ERROR] 401 - Error occurred downloading file
Looking at /var/tmp/airtime/pypo/cache/scheduler shows nothing but zero byte files:
drwxr-xr-x 2 www-data www-data 4096 Jul 12 19:00 .
drwxr-xr-x 3 www-data www-data 4096 Jun 25 18:59 ..
-rw-r--r-- 1 www-data www-data 0 Jul 12 19:00 6.mp3
-rw-r--r-- 1 www-data www-data 0 Jul 12 19:00 9.mp3
In every case the Libretime status page shows all normal, and everything looks good on the streaming page.
Digging a bit deeper, something seems amiss earlier in the game. Here are some logs taken right after rebooting, and before even logging in to the web interface:
/var/log/airtime/pypo/pypo.log (at startup):
2021-07-12 17:25:07,861 [__main__] [INFO ] ###########################################
2021-07-12 17:25:07,863 [__main__] [INFO ] # *** pypo *** #
2021-07-12 17:25:07,863 [__main__] [INFO ] # Liquidsoap Scheduled Playout System #
2021-07-12 17:25:07,863 [__main__] [INFO ] ###########################################
2021-07-12 17:25:07,863 [__main__] [INFO ] Timezone: ('PST', 'PDT')
2021-07-12 17:25:07,863 [__main__] [INFO ] UTC time: 2021-07-13 00:25:07.863760
2021-07-12 17:25:07,992 [version1] [INFO ] Airtime API version found: 1.1
2021-07-12 17:25:07,992 [version1] [INFO ] pypo is only compatible with API version: 1.1
2021-07-12 17:25:08,104 [__main__] [ERROR] [Errno 111] Connection refused
2021-07-12 17:25:08,104 [__main__] [WARNI] Liquidsoap doesn't appear to be running!, Sleeping and trying again
2021-07-12 17:25:09,117 [__main__] [INFO ] Liquidsoap version string found 1.3.3
2021-07-12 17:25:09,119 [pypomessagehandler] [INFO ] Initializing RabbitMQ stuff
2021-07-12 17:25:09,145 [pypofetch] [INFO ] PypoFetch: init complete
2021-07-12 17:25:09,189 [mixins] [WARNI] Broker connection error, trying again in 2.0 seconds: ConnectionRefusedError(111, 'Connection refused').
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/transport.py", line 138, in _connect
host, port, family, socket.SOCK_STREAM, SOL_TCP)
File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/kombu-4.6.10-py3.7.egg/kombu/utils/functional.py", line 344, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/kombu-4.6.10-py3.7.egg/kombu/connection.py", line 870, in _connection_factory
connection = self._establish_connection()
File "/usr/local/lib/python3.7/dist-packages/kombu-4.6.10-py3.7.egg/kombu/connection.py", line 806, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.7/dist-packages/kombu-4.6.10-py3.7.egg/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/connection.py", line 314, in connect
self.transport.connect()
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/transport.py", line 78, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/transport.py", line 149, in _connect
"failed to resolve broker hostname"))
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/transport.py", line 162, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
2021-07-12 17:25:09,192 [pypoliqqueue] [INFO ] waiting indefinitely for schedule
2021-07-12 17:25:09,201 [recorder] [INFO ] RecorderFetch: init complete
2021-07-12 17:25:09,350 [telnetliquidsoap] [INFO ] streams.live_dj_stop
2021-07-12 17:25:09,350 [telnetliquidsoap] [INFO ] streams.master_dj_stop
2021-07-12 17:25:09,350 [telnetliquidsoap] [INFO ] streams.scheduled_play_start
2021-07-12 17:25:09,350 [telnetliquidsoap] [INFO ] b'vars.stream_metadata_type 0\n'
2021-07-12 17:25:09,350 [telnetliquidsoap] [INFO ] b'vars.station_name KBFG 107.3 FM\n'
2021-07-12 17:25:09,351 [telnetliquidsoap] [INFO ] b'vars.default_dj_fade 0.000\n'
2021-07-12 17:25:09,380 [recorder] [INFO ] Started...
2021-07-12 17:25:09,639 [recorder] [INFO ] Parsing recording show schedules...
2021-07-12 17:25:09,639 [recorder] [INFO ] Bootstrap recorder schedule received: {'shows': [], 'is_recording': False, 'server_timezone': 'America/Los_Angeles'}
2021-07-12 17:25:09,639 [recorder] [INFO ] Bootstrap complete: got initial copy of the schedule
2021-07-12 17:25:09,664 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:25:09,664 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:25:09,828 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:25:09,828 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:25:10,015 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:25:10,019 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:25:10,200 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:25:10,201 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:25:10,340 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:25:10,343 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:25:10,344 [pypofetch] [INFO ] Loop #1
2021-07-12 17:25:11,206 [mixins] [WARNI] Broker connection error, trying again in 4.0 seconds: ConnectionRefusedError(111, 'Connection refused').
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/transport.py", line 138, in _connect
host, port, family, socket.SOCK_STREAM, SOL_TCP)
File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/kombu-4.6.10-py3.7.egg/kombu/utils/functional.py", line 344, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/kombu-4.6.10-py3.7.egg/kombu/connection.py", line 870, in _connection_factory
connection = self._establish_connection()
File "/usr/local/lib/python3.7/dist-packages/kombu-4.6.10-py3.7.egg/kombu/connection.py", line 806, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.7/dist-packages/kombu-4.6.10-py3.7.egg/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/connection.py", line 314, in connect
self.transport.connect()
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/transport.py", line 78, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/transport.py", line 149, in _connect
"failed to resolve broker hostname"))
File "/usr/local/lib/python3.7/dist-packages/amqp-2.6.1-py3.7.egg/amqp/transport.py", line 162, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
2021-07-12 17:25:15,217 [mixins] [INFO ] Connected to amqp://airtime:**@127.0.0.1:5672//airtime
pypo.log about 7 minutes later:
2021-07-12 17:31:50,344 [pypofetch] [INFO ] Queue timeout. Fetching schedule manually
2021-07-12 17:31:50,392 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:31:50,393 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:31:50,434 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:31:50,434 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:31:50,470 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:31:50,471 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:31:50,509 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:31:50,509 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:31:50,545 [pypofetch] [ERROR] Unable to fetch schedule
2021-07-12 17:31:50,545 [pypofetch] [ERROR] byte indices must be integers or slices, not str
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airtime_playout-1.0-py3.7.egg/pypo/pypofetch.py", line 460, in manual_schedule_fetch
self.schedule_data = self.api_client.get_schedule()
File "/usr/local/lib/python3.7/dist-packages/api_clients-2.0.0-py3.7.egg/api_clients/version2.py", line 61, in get_schedule
start = isoparse(item["starts"])
TypeError: byte indices must be integers or slices, not str
2021-07-12 17:31:50,546 [pypofetch] [INFO ] Loop #2
/var/log/airtime/zendphp.log (at startup):
2021-07-12T17:25:08-07:00 INFO (6): kbfgstream.org Registered Component: pypo@52.88.234.152
2021-07-12T17:25:09-07:00 INFO (6): kbfgstream.org Registered Component: show-recorder@52.88.234.152
2021-07-12T17:25:10-07:00 INFO (6): kbfgstream.org Notifying RabbitMQ to send message to pypo
2021-07-12T17:25:10-07:00 ERR (3): kbfgstream.org [ErrorController.php:26 - errorAction()] - stream_socket_client(): unable to connect to tcp://127.0.0.1:5672 (Connection refused)
2021-07-12T17:25:10-07:00 ERR (3): kbfgstream.org [ErrorController.php:27 - errorAction()] - #0 [internal function]: PhpAmqpLib\Wire\IO\StreamIO->error_handler(2, 'stream_socket_c...', '/usr/share/airt...', 134, Array)
#1 /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php(134): stream_socket_client('tcp://127.0.0.1...', 111, 'Connection refu...', 3, 4, Resource id #14)
#2 /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(196): PhpAmqpLib\Wire\IO\StreamIO->connect()
#3 /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(180): PhpAmqpLib\Connection\AbstractConnection->connect()
#4 /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AMQPStreamConnection.php(51): PhpAmqpLib\Connection\AbstractConnection->__construct('airtime', 'airtime', '/airtime', false, 'AMQPLAIN', NULL, 'en_US', Object(PhpAmqpLib\Wire\IO\StreamIO), 0)
#5 /usr/share/airtime/php/airtime_mvc/application/models/RabbitMq.php(23): PhpAmqpLib\Connection\AMQPStreamConnection->__construct('127.0.0.1', '5672', 'airtime', 'airtime', '/airtime')
#6 /usr/share/airtime/php/airtime_mvc/application/models/RabbitMq.php(50): Application_Model_RabbitMq::sendMessage('airtime-pypo', 'direct', true, '{"event_type":"...')
#7 /usr/share/airtime/php/airtime_mvc/application/controllers/ApiController.php(1212): Application_Model_RabbitMq::SendMessageToPypo('reset_liquidsoa...', Array)
#8 /usr/share/airtime/php/vendor/zf1s/zend-controller/library/Zend/Controller/Action.php(516): ApiController->rabbitmqDoPushAction()
#9 /usr/share/airtime/php/vendor/zf1s/zend-controller/library/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('rabbitmqDoPushA...')
#10 /usr/share/airtime/php/vendor/zf1s/zend-controller/library/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))
#11 /usr/share/airtime/php/vendor/zf1s/zend-application/library/Zend/Application/Bootstrap/Bootstrap.php(105): Zend_Controller_Front->dispatch()
#12 /usr/share/airtime/php/vendor/zf1s/zend-application/library/Zend/Application.php(384): Zend_Application_Bootstrap_Bootstrap->run()
#13 /usr/share/airtime/php/airtime_mvc/application/airtime-boot.php(84): Zend_Application->run()
#14 /usr/share/airtime/php/airtime_mvc/public/index.php(68): require_once('/usr/share/airt...')
#15 {main}
2021-07-12T17:25:10-07:00 ERR (3): kbfgstream.org [ErrorController.php:54 - errorAction()] - An internal application error has occurred.: ErrorException: stream_socket_client(): unable to connect to tcp://127.0.0.1:5672 (Connection refused) in /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php:134
Stack trace:
#0 [internal function]: PhpAmqpLib\Wire\IO\StreamIO->error_handler(2, 'stream_socket_c...', '/usr/share/airt...', 134, Array)
#1 /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php(134): stream_socket_client('tcp://127.0.0.1...', 111, 'Connection refu...', 3, 4, Resource id #14)
#2 /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(196): PhpAmqpLib\Wire\IO\StreamIO->connect()
#3 /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(180): PhpAmqpLib\Connection\AbstractConnection->connect()
#4 /usr/share/airtime/php/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AMQPStreamConnection.php(51): PhpAmqpLib\Connection\AbstractConnection->__construct('airtime', 'airtime', '/airtime', false, 'AMQPLAIN', NULL, 'en_US', Object(PhpAmqpLib\Wire\IO\StreamIO), 0)
#5 /usr/share/airtime/php/airtime_mvc/application/models/RabbitMq.php(23): PhpAmqpLib\Connection\AMQPStreamConnection->__construct('127.0.0.1', '5672', 'airtime', 'airtime', '/airtime')
#6 /usr/share/airtime/php/airtime_mvc/application/models/RabbitMq.php(50): Application_Model_RabbitMq::sendMessage('airtime-pypo', 'direct', true, '{"event_type":"...')
#7 /usr/share/airtime/php/airtime_mvc/application/controllers/ApiController.php(1212): Application_Model_RabbitMq::SendMessageToPypo('reset_liquidsoa...', Array)
#8 /usr/share/airtime/php/vendor/zf1s/zend-controller/library/Zend/Controller/Action.php(516): ApiController->rabbitmqDoPushAction()
#9 /usr/share/airtime/php/vendor/zf1s/zend-controller/library/Zend/Controller/Dispatcher/Standard.php(308): Zend_Controller_Action->dispatch('ra
I don’t know what is going on here, but I can’t think of anything else to do. Strange that last summer I did a Libretime install on an AWS server with Debian Buster and had absolutely no problems.
Has anyone done a successful install of an ssl enabled Libretime in the past couple of weeks? I’m not sure if I’m hitting a bug, or just missing something obvious.