Automatic File System Import, Watched Folders & more

The first option would be great for our use as well. I thought it was already available, but after playing with it for a while, I determined that I didn’t have quite enough fortitude to fight with the PHP scripts that I thought would do it.

Yeah there were a few shell scripts that accomplished the importing of files but it was outside of the scope of the libretime install to make them work. This PR once fixed will add the service that basically makes a folder that imports files copied into it. It is working I just need to finish up adding an option to make the folder user-defined during install and change the documentation.

Just yesterday our collective discussed how we plan to migrate over to using libretime.

For 16 years the station has always kept our music library organized using folders in the operating system organized by artist and album. This has been far from ideal for various reasons. To my knowledge the automation software we’ve used, like Zara radio and Radit, scan these folders without moving files, which gives a certain security for non-tech folks who have seen buggy automation software come and go, but at least we have a backup of the music library.

My understanding from reading the documentation is that imported files are moved into sub-folders according their metadata. Is this correct?

Metadata in our music library has always been sketchy, especially because we try to give priority to local artists. We’re now in a “cleaning house” stage of selecting what we DO want from the years of accumulated music, and then improving metadata in those files with tools like Puddletag. We’ll need to be well organized and in agreement as a group to ensure that people don’t import files without metadata.

No technology can substitute for strong group consensus, but since this thread asks about needs, features that would help out collective radios include

  1. It sure would be great if the media folder could be a “source of truth” for libretime, ie. if files are added, moved, or deleted from that folder then those changes would be reflected in the libretime interface. We’re currently thinking we’ll network mount our main music folder as you mentioned via sshfs. If however the answer to my file organization question above is yes, then we can teach that workflow in our upcoming trainings.
  2. It would also be great to either alert a user that the files they’re importing don’t contain metadata, or even to include an optional step in the import process where users could review and add metadata.

muchas gracias!!!
-ryan

Hey Ryan,
That part of the documentation is out of date. There is no “watched folder” functionality currently for LibreTime, before I put together this latest PR there was no easy way to import music from a filesystem at all, other than a few shell scripts. You basically had to manually upload everything through the web interface.

What I put together currently only creates a folder where anything you copy into it will be uploaded to LibreTime and deleted from the filesystem. I tried to implement the watched folder setup through modifying the code a previous contributor had put together but ran into numerous issues where it wouldn’t playback files and basically found that it wasn’t very compatible with the web API. When LibreTime was still the Airtime SaaS-dev branch (aka Airtime.pro) they refactored the code so that it could work with Amazon S3 so that they could put everyones data in a single cloud repository and not have to worry about it. But they left the filesystem as an option and that is what we have been using.

What currently happens to a file as it is imported is that it is moved into /srv/airtime/stor/imported/#/artist/track.file and the # corresponds to the user ID of the person uploading it. Everything will be put under user ID /1/ at the moment when uploaded through the PR I put together. The metadata is what is used to display and search for tracks in LibreTime, so it is pretty important that it be consistent. I believe that when you edit the track title etc it also changes the metadata on the file. But I’m not totally sure.

So I haven’t finished coding my PR. But let me provide some comments on your suggestions.

For the first 1 - I understand what you are proposing here but I’m not sure how easily it could be implemented as the watched folder functionality doesn’t currently exist. Specifically LibreTime stores all of the metadata for its files in the Postgres SQL database, and it creates this through analyzing the file upon import. LibreTime would need to rescan all of the files and see if any of them had changed and then update the database if it detects a change. If a file is deleted or moved it isn’t as straight forward either. So that is an issue.

So this is a challenge for stations using LibreTime and that is how to maintain large databases of music. Another issue with the whole remote filesystem is the source of truth plan would be what happens if someone uploads a file directly to the LibreTime web UI. The way watched folders originally worked is the files normally uploaded would be added to the local filesystem and the watched folders are separate and so nothing is ever added to them but instead just stored on the local filesystem.

  1. The web interface currently doesn’t do any review alert and I’m not totally sure how to best implement this with any kind of filesystem import. Figuring out a way to do this seems like a good idea though.

So one additional feature I was thinking of adding to my code was the ability to scan a folder and import everything in it. This could be used with a remote filesystem but the question becomes how to determine if a file was imported already. There is no duplicate detection currently in LibreTime and so I think this would be essential before we scan a folder and import everything while leaving the files intact. Also my thought is that the files would be copied to LibreTime’s local filesystem rather than it attempting to play files from a sshfs or other network mount. This would help ensure that they are locally available for play even if there is an issue with the network or whatever. But it also makes the duplicate detection aspect of it even more important.

So I’m not sure how this would work AS IS with your current setup but when I get back to working on the PR I’ll definitely try to keep in mind the feedback you gave. I wonder how automated filesystem importing by scanning an existing folder would work for you all. I don’t think there is easy way to keep your existing filesystem as the sole source of truth if you are going to have people using the web UI to uploaded as well. I do think this is an interesting problem to try to solve, and the filesystem/folders created by LibreTime are kind of a mess in comparison to those created by humans with attention to detail.

  1. you rock.

  2. if we need to just tell everybody not to touch the folder of music, that’s fine. we’ll be doing trainings anyway, so that can just be part of it. we could even set folder permissions to allow access to libretime but prevent the default server desktop user from making changes without sudo. The important thing is that tracks are organized within sub-folders by artist. that means we can make a backup that’s still usable, organize-able.

more soon, gotta run now to go cook dinner for my hungry 3yo and 5yo, somethign i’m sure you’re totally unable to understand :wink: i also wanna ask you about how best to contribute documentation in spanish. thank you take care!!!

Hello to all.
I am working to organize our Music Data Base for import in Libretime. We ripped about 39’000 titles since 19 years. Our handicap is that we didn’t used Metadata Tags and all the files are stored in one folder without sub-folders. I want to change this mistake. I have to tag all files. MusicBarainzPicard is very useful for this. After beginning the import I saw that it would be nice to have BPM values in tags to use all advantages of intelligent Playlists . MusicBrainzPicard has plugins for BPM analyzis, but it does not work. I found Piston BPM detector to make BPM tags, but this tool is made for Windows and I have to use it in wine. This solution is not so clean like I wished.
Maybe somebody can upgrade the Libretime import process.
It exist command line tools to analyze BPM value and write this tag into the audio file.
My question is now, will it be possible to make that the importing process in Libretime will do this automatically, when no BPM value exists or when it is = 0 ?
I am a newcomer in Libretime and all what I have tried was very, very nice. Bravo…
I want to promote Libretime for new local non commercialcommunity radio stations.
Thank you for all this nice work.

I have worked on the airtime_analyzer code and it is currently written in python 2.7 - if there is a CLI tool that works under Linux it could probably be added to the import process.

Hi Robb,

thank you for your quick relpy.

I will make some tests with BPM tools in command line. Take a look if
ogg, mp3, flac files can be analyzed and tags be written by this tools.

http://manpages.ubuntu.com/manpages/bionic/man1/bpm.1.html

First I will try this one: http://www.pogo.org.uk/~mark/bpm-tools/

It is very old, but maybe there is no better tool available

I asked Mark Hills for news about his tool and if we can use it,

Now I am waiting for his answer.

Best regards, Klaus

I found more info’s about BPM analyzing and tag.
It seams that BPM tools are working fine for detection of BPM but
overwrite genre and album tags.

This page shows maybe more information.


maybe tomorrow I get some time to make some tests.

Cool, thanks for the update.

1 Like

Hello robbt.
I installed bpm tools in my ubuntu 18.04 PC and made first test.
Bpm tools detected BPM of indicated “audio-file” but didn’t write the BPM tag in to the files.
I got the return that it needs sox.
So I installed sox from official bionic archive.
https://launchpad.net/ubuntu/bionic/amd64/libsox-fmt-base
But this doesn’t work.
It wasn’t enough.
When I tried to tag my audio mp3 file I got the return that mp3 couldn’t be treated.
The sudo apt-get install libsox-fmt-mp3 was the solution.
Now I can analyze and write with simple command line the BPM value in the mp3 file without changing the other existing tags. Maybe we need to install libsox-fmt-XXX for other .xxx formats too. I think it will be useful to add this function in the audio file import process. This means too that the installation of Libretime should add this tools and their dependencies too and maybe add a bottom to choice this function in Libretime.

Hi, coming back to BPM tools. I think if BPM will be part of analyzing during importing audio files we can increase Libretime. I made some experimentation and found following way.
I installed bpm tools in my ubuntu 18.04 PC and made first test.
Bpm tools detected BPM of indicated “audio-file” but didn’t write the BPM tag in to the files.
I got the return that it needs sox.
So I installed sox from official bionic archive.


But this doesn’t work.
It wasn’t enough.
When I tried to tag my audio mp3 file I got the return that mp3 couldn’t be treated.
The sudo apt-get install libsox-fmt-mp3 was the solution.
Now I can analyze and write with simple command line the BPM value in the mp3 file without changing the other existing tags. Maybe we need to install libsox-fmt-XXX for other .xxx formats too. I think it will be useful to add this function in the audio file import process. This means too that the installation of Libretime should add this tools and their dependencies too and maybe add a bottom to choice this function in Libretime.

Hey Klaus,

There’s a PR for the watch folders, and BPM calculation is implemented: https://github.com/TeHikuMedia/libretime/blob/140fc94d3a93fbd632a7e725c889d570a751277b/python_apps/libretime_watch/libretime_watch/metadata.py#L113. Sorry this PR hasn’t been merged yet but it’s nearly ready…

Yep,

if this can work also in Libretime Master, it will be a useful step to complete Libretime.

Today I have to do BPM tag in terminal command line with: bpm-tag “*.mp3”

robbt A possibility i would like is to get a warning when libretime imports an mp3 file without artist_name. Is this a possibility in a later phase?

Just following up about the mp3 import question. This is a good idea, I don’t think anyone would be opposed to it but I also don’t know if it will be remembered without a feature request on github. Nobody is working on that part of LibreTime too much at the moment but thanks for sharing an idea for a possible upgrade.

When we upload a pre-recorded show (Track Type=PRS) we put the show name in the “Album” field (and show host in the “Creator” field). It would be good to have Serge’s warning on different fields, depending on “Track Type”.

F’rinstance, we absolutely need “Genre” on all MUS track types, but not on the PRS track types.

I’ve created a feature request in Github: https://github.com/LibreTime/libretime/issues/1129

–Bob.

So i’m going to re activate this topic with a Solution that just works.

It is not pretty but i’m happy with it, it just does it’s job.
Based on the FTP Upload hook i wrote this PHP Script:

#!/usr/bin/php
<?php
$uploaddir = "/path/to/uploads/folder/"; //Path to folder where files will be uploaded, preferably full path ! (must have / at the end)
$api_key = "YOUR API KEY GOES HERE"; //LibreTime API Key (needed for import)

$dir = scandir($uploaddir);
array_splice($dir, 0, 2);

foreach($dir as $file)
{
 echo "Uploading " . $uploaddir . $file . "\n";
 shell_exec("curl -fL --max-time 30 http://127.0.0.1:80/rest/media -u " . $api_key . ": -X POST -F file=@" . $uploaddir . $file);
}
?>


Just set the 2 needed variables and run it, does what it should.
I have it run via a cronjob (crontab -e Settings: * 1 * * * /path/to/import.php) every hour.

Works like intended and i don’t have to use the Upload via Web UI anymore.

Maybe someone will be happy with it, i’m fairly certain it is a “bad” way of doing it, i could see how to upload the file with PHP Curl but i’m lazy so i did not :crazy_face:

Greetings from Germany :slight_smile:

1 Like

I was having trouble with your php script importing files with spaces or special characters (only tested with single quotes at the moment). I rewrote your script in Python:

import os
from pathlib import Path
import requests
from requests.auth import HTTPAuthBasic


APIKEY = "YOUR_API_KEY"
IMPORTDIR = "/path/to/import/directory"


def import_audio():
    localhost = "http://127.0.0.1:80/rest/media"
    api_key = HTTPBasicAuth(APIKEY, "")

    for files in os.listdir(IMPORTDIR):
        data = {"name": files}
        local_file = {"file": open(os.path.join(IMPORTDIR, files), 'rb')}
        response = request.post(localhost, auth=api_key, data=data, files=local_file)
        print(response.text)


if __name__ == "__main__":
    import_audio()

One thing I’d like this script to do is assign track_type upon import. I tried a few different things, but none worked. I’ll update if I figure it out.

So, I don’t like putting my API key, database username and password in plaintext hardcoded in the script. But this gets me where I wanted to be. I’m confident now, that if I scraped the source for getting config settings from /etc/airtime/airtime.conf I’d be able to make this script into a watchdog-like service.

#!/usr/bin/python3
import os
import requests

import psycopg2

from pathlib import Path
from requests.auth import HTTPBasicAuth


APIKEY = "YOUR_API_KEY" # Should be parsed from config
IMPORTDIR = Path('~/Music').expanduser() # Should be cli argument?
TRACK_ID = 'desired_track_id' # Certainly needs to be a cli argument.


def import_music() -> list:
    # localhost could also be cli argument as this part of the script could be ran from outside the server
    localhost = "http://127.0.0.1:80/rest/media"
    api_key = HTTPBasicAuth(APIKEY, "")

    track_ids = []

    for files in os.listdir(IMPORTDIR):
        data = {"name": files}
        local_file = {"file": open(os.path.join(IMPORTDIR, files), 'rb')}
        response = requests.post(localhost, auth=api_key, data=data, files=local_file)
        resp_data = response.json()
        track_ids.append(data['id'])

    return track_ids


def alter_track_type(track_ids: list, track_type:str):
    psql = psycopg2.connect("dbname='DATABASE' user='USERNAME' host='localhost' password='PASSWORD'")
    cursor = psql.cursor()
    for track_id in track_ids:
        cursor.execute(f"update cc_files set track_type = \'{track_type}\' where id = {track_id}")

    psql.commit()
    cursor.close()
    psql.close()


if __name__ == "__main__":
    alter_track_type(import_music(), TRACK_ID)

While I was writing this, I was worried that I would have to be strict about tagging track_type as there was a point where I thought about just update cc_files set track_type = 'whatever' where track_type = '';. But I was pretty stoked when I realized that that requests.post returns a json, that can be parsed.

So now, in theory, you could setup a directory that ingests files according to rules. Like for instance:

Directory /music_imports/500/
This is a smb file share or whatever…
the cli or cronjob might look like: audio-ingest /music_imports/500 500
these files take the track_type 500…

I think you might be able to also query libretime to see that it has ingest audio properly with like:

def query_libretime(track_id):
    track_url = f"http://127.0.0.1:80/rest/media/{track_id}"
    ingest = requests.get(track_url, auth=api_key)
    data = ingest.json()
    if data['md5'] is not 'null':
        print('Audio has been ingested')

then you can delete the original… maybe rename the file during ingestion… like:

Prince - Raspberry Beret.mp3 # Original file
Prince - Raspberry Beret.mp3.INGEST # has been uploaded, but has not yet been hashed
Prince - Raspberry Beret.mp3.ACCEPTED # has been properly imported and will next be deleted
-- no file # has been deleted since it has been properly imported by libretime...

And in case it has been rejected, no deletion…

Prince - Raspberry Beret.mp3.REJECTED # file remains in the directory and is ignored by further scans

An entry is then made to the log as to why the file has been rejected… But I’m not sure if hash is still compiled regardless of whether or not the file has been properly imported. This will be tested later…

*Note, I’m a little blurry-eyed, so there may be mistakes in my code…