Bulk import times out and fails

Hi all,
I’ve just installed my first Libretime instance, and I’m looking forward to giving it a spin.

I am facing the following issue while trying to import my library:

Describe the problem
Bulk import runs for about a minute, then fails. When restarted, it skips the file it previously failed on due to having already recorded the md5 hash, continues and always crashes after about a minute.

To reproduce
Steps to reproduce the behavior.

  1. Run the following command: ‘sudo -u libretime libretime-api bulk_import --path /home/caturria/library’
  2. Wait 30 - 60 seconds.
  3. See traceback and runtime error.

Expected behavior
Approx. 2000 tracks successfully uploaded.

Relevant log output or error messages

From terminal:

  File "/usr/local/lib/python3.9/dist-packages/django/core/management/__init__.py", line 413, in execute                
    self.fetch_command(subcommand).run_from_argv(self.argv)                                                             
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/base.py", line 354, in run_from_argv              
    self.execute(*args, **cmd_options)                                                                                  
  File "/usr/local/lib/python3.9/dist-packages/django/core/management/base.py", line 398, in execute                    
    output = self.handle(*args, **options)                                                                              
  File "/usr/local/lib/python3.9/dist-packages/libretime_api/storage/management/commands/bulk_import.py", line 66, in ha
ndle                                                                                                                    
    importer.import_dir(Path(path).resolve(), library, allowed_extensions)                                              
  File "/usr/local/lib/python3.9/dist-packages/libretime_api/storage/management/commands/bulk_import.py", line 160, in i
mport_dir                                                                                                               
    self._walk_dir(path, library, allowed_extensions)                                                                   
  File "/usr/local/lib/python3.9/dist-packages/libretime_api/storage/management/commands/bulk_import.py", line 136, in _
walk_dir                                                                                                                
    self._walk_dir(sub_path, library, allowed_extensions)                                                               
  File "/usr/local/lib/python3.9/dist-packages/libretime_api/storage/management/commands/bulk_import.py", line 136, in _
walk_dir                                                                                                                
    self._walk_dir(sub_path, library, allowed_extensions)                                                               
  File "/usr/local/lib/python3.9/dist-packages/libretime_api/storage/management/commands/bulk_import.py", line 142, in _
walk_dir                                                                                                                
    self._handle_file(sub_path.resolve(), library)                                                                      
  File "/usr/local/lib/python3.9/dist-packages/libretime_api/storage/management/commands/bulk_import.py", line 120, in _
handle_file                                                                                                             
    self._upload_file(filepath, library)                                                                                
  File "/usr/local/lib/python3.9/dist-packages/libretime_api/storage/management/commands/bulk_import.py", line 102, in _
upload_file                                                                                                             
    raise RuntimeError(f"could not upload {filepath}") from exception                                                   
RuntimeError: could not upload /home/caturria/library/Elton John/The Last Song.mp3                           

The last few lines in /var/log/libbretime/analyzer.log are as follows:

2022-12-29 12:30:56.259 | INFO     | libretime_analyzer.status_reporter:send_http_request:106 - HTTP request sent successfully.
2022-12-29 12:30:58.926 | INFO     | libretime_analyzer.message_listener:msg_received_callback:101 -  - Received 'b'{"file_id":1069,"tmp_file_path":"\\/srv\\/libretime\\/organize\\/1069-Goodbye Yellow Brick Road.mp3","import_directory":"\\/srv\\/libretime\\/\\/imported\\/1","original_filename":"Goodbye Yellow Brick Road.mp3"}'' on routing_key ''
2022-12-29 12:30:59.010 | INFO     | libretime_analyzer.status_reporter:send_http_request:106 - HTTP request sent successfully.
2022-12-29 12:31:01.706 | INFO     | libretime_analyzer.message_listener:msg_received_callback:101 -  - Received 'b'{"file_id":1070,"tmp_file_path":"\\/srv\\/libretime\\/organize\\/1070-Cry To Heaven.mp3","import_directory":"\\/srv\\/libretime\\/\\/imported\\/1","original_filename":"Cry To Heaven.mp3"}'' on routing_key ''
2022-12-29 12:31:01.785 | INFO     | libretime_analyzer.status_reporter:send_http_request:106 - HTTP request sent successfully.
2022-12-29 12:31:05.020 | INFO     | libretime_analyzer.status_reporter:send_http_request:106 - HTTP request sent successfully.

Also worth noting: in earlier attempts I was receiving logged messages pertaining to execution timeouts. So I went and changed PHP’s max_execution_time from 30 to 3600 in both php.ini and in Libretime’s php-fpm configuration; same behaviour persists, minus warnings about execution timeout. Unfortunately those original logs were overwritten.

LibreTime version
The version of LibreTime your are running. You can find the LibreTime version on the Settings > Status page.

LibreTime version: (e.g. 3.0.0-beta.1 or main-c9bc3a12987e74e0980fc8adcc4702d9f8ad4249)
3.0.1

Installation method and OS / Environment
The method used to install and the operating system / environment your installation is running on.

Operating system: (e.g. Ubuntu Focal or Debian Bullseye)
Debian Bullseye

Method: (e.g. Installer script or Docker compose)
Installer script

Update:
I can eventually get content uploaded, but the bulk-upload script has to be restarted after every 1 - 50 tracks and failed tracks have to be cleared.

Hi,

Could you provide the legacy logs ? I suspect that the importer failed during an HTTP request, so the error should be available in the legacy logs. The client should print more details about the error, my bad.

Jo

Hi,

Legacy.log is empty.
Kind regards,

Jordan.