Disk space getting filled with cache

Get Help
1

Describe the problem

Disk space gets filled recurrently with temporary files and causes the server to stop playout.
I’m not completely sure what kind of temp files are causing this but if I reboot the vps the free space goes back to normal.

To reproduce
This is happening in only one instance, I’m trying to find out if the user is doing anything particular regarding the way of using / scheduling the platform.

Relevant log output or error messages

I’m getting lots of

libretime_playout.player.file:copy_file:47 - copying file 1316 to cache /var/lib/libretime/playout/scheduler/…

==> /var/log/libretime/playout.log <==
2024-01-24 18:05:45,992 | INFO | libretime_playout.player.file:copy_file:47 - copying file 859 to cache /var/lib/libretime/playout/scheduler/859.m4a
2024-01-24 18:05:59,278 | INFO | libretime_playout.player.file:copy_file:47 - copying file 335 to cache /var/lib/libretime/playout/scheduler/335.mp3
2024-01-24 18:05:59,535 | INFO | libretime_playout.player.file:copy_file:47 - copying file 1093 to cache /var/lib/libretime/playout/scheduler/1093.mp3
2024-01-24 18:05:59,902 | INFO | libretime_playout.player.file:copy_file:47 - copying file 1089 to cache /var/lib/libretime/playout/scheduler/1089.mp3
2024-01-24 18:06:00,209 | INFO | libretime_playout.player.file:copy_file:47 - copying file 1090 to cache /var/lib/libretime/playout/scheduler/1090.mp3
2024-01-24 18:06:00,591 | INFO | libretime_playout.player.file:copy_file:47 - copying file 1080 to cache /var/lib/libretime/playout/scheduler/1080.mp3
2024-01-24 18:06:01,042 | INFO | libretime_playout.player.file:copy_file:47 - copying file 849 to cache /var/lib/libretime/playout/scheduler/849.m4a

==> /var/log/libretime/liquidsoap.log <==
2024/01/24 18:06:04 [server:3] New client: 127.0.0.1.
2024/01/24 18:06:04 [server:3] Client 127.0.0.1 disconnected.

==> /var/log/libretime/liquidsoap.log <==
2024/01/24 18:08:04 [cue_cut_9615:3] Cueing out…
2024/01/24 18:08:04 [switch:blank+schedule:3] Switch to map_metadata:offline with forgetful transition.
2024/01/24 18:08:04 [lang:3] transition called…
2024/01/24 18:08:04 [decoder:3] Method “FFMPEG” accepted “/var/lib/libretime/playout/scheduler/1400.mp3”.
2024/01/24 18:08:04 [dummy(dot)2:3] Source failed (no more tracks) stopping output…
2024/01/24 18:08:04 [dummy:3] Source failed (no more tracks) stopping output…
2024/01/24 18:08:04 [s3:3] Prepared “/var/lib/libretime/playout/scheduler/1400.mp3” (RID 6).
2024/01/24 18:08:04 [switch_9646:3] Switch to insert_metadata_9631.
2024/01/24 18:08:04 [ffmpeg:3] [mp3float @ 0x7f7e1c070e80] Could not update timestamps for skipped samples.
2024/01/24 18:08:04 [lang:3] timeout --signal=KILL 45 libretime-playout-notify media ‘8482’ &
2024/01/24 18:08:04 [lang:3] Using message format 0
2024/01/24 18:08:04 [lang:3] Using message format 0
2024/01/24 18:08:04 [switch:blank+schedule:3] Switch to map_metadata:schedule with transition.
2024/01/24 18:08:04 [lang:3] transition called…
2024-01-24 18:08:05,688 | INFO | libretime_playout.notify.main:media:72 - Sending currently playing media id ‘8482’

LibreTime version
We were experiencing the issue in a fresh v3.0.2, so I upgraded to v4.0.0
and it’s still happening.

Installation method and OS / Environment
Script installer. Debian Bullseye. LXC with LVM disk volume.

2

We’re running LibreTime v3.0 Alpha from 2020. We’ve found our /tmp folder filling with files belonging to celery:celery that are the automatically downloaded podcast files. To mitigate the problem I’ve created a script /etc/cron.daily/clean-tmp that does

find /tmp -user celery -mtime +2 -delete >> ${logfile} 2>&1

In addition, there are celery files in /tmp of podcast episodes that download properly, but aren’t added to the track list. There appears to be nothing wrong with these files (valid MP3s with metadata), so I rename them with a .mp3 extension and submit them with the Upload screen in the WebUI.

To help identify these files I run the output of exiftool through an awk script, then compare the output to the track list where “Owner = admin”

exiftool * | getlibretime.awk

where getlibretime.awk is:

#! /usr/bin/gawk -f

# Program	: getlibretimetmp.awk
# Purpose	: Find the files in /tmp that have not been processed by LibreTime
# Author	: Bob Jonkman
# Date		: 25 May 2022
# Modified	: 15 Dec 2023 Bob Jonkman
#		  - Cleared variables, case a pattern doesn't exist for an entry
#                 - Add index number to datetime, in case of duplicate datetimes

# Usage		: exiftool /tmp/tmp* | gawk -f getlibretimetmp

BEGIN		{ FS = ": "	# input Field Separator
		}

/File Name/	{
			FileName = $2
		}

/File Inode/	{
                     ## Add index number in case of duplicate datetimes (fancy leading zeros)
			i++
 			FileInode[i] = $2 "(" substr("000" i, length(i)+1, 3) ")"
			FileNameArr[FileInode[i]] = FileName
			FileName = ""
		}

/Album/		{
			Album[FileInode[i]] = $2
		}

/Title/		{
			Title[FileInode[i]] = $2
		}

END		{	NumFiles = asort(FileInode, SortedDates)
			for (i=1 ; i <= NumFiles ; i++)
			{
				print (SortedDates[i] , FileNameArr[SortedDates[i]] , "\"" Album[SortedDates[i]] "\",\"" Title[SortedDates[i]] "\"" )
			}
		}


# EOF: getlibretimetmp.awk

I haven’t reported this as a bug (yet) since our version is so old and I want to see the behaviour on an up-to-date version first.

–Bob.

(edit: fix typo)
(2nd edit: Bob figures out code blocks)