API is unreachable behind nginx

Hi! Before I start - many thanks for the amazing work on this project.

Describe the problem
I’m setting up LibreTime for an existing web radio, and I’m having trouble exposing the API behind nginx.

A few considerations:

  • LibreTime must run on the same host as the radio itself (which is a very simple nodejs express site, also running in a Docker container)
  • The radio’s UI should be accessible at https://example.com; LibreTime should be accessible at https://libretime.example.com

At first glance, this already works nicely: I pass a custom default.conf to the nginx container, which redirects traffic to the relevant Docker container based on the server name. So far so good. Looking at the logs below however, I see that everything isn’t running so smoothly. What I read from the logs is:

  • The playout container executes self.services.version_url()["api_version"] which sends a request to LIBRETIME_GENERAL_PUBLIC_URL from the container’s environment, which is set to http://nginx here
  • The nginx container receives the request and forwards it to example.com
  • The server returns a 404 because example.com/api/version/api_key/[REDACTED] doesn’t exist; however libretime.example.com/api/version/api_key/[REDACTED] does exist

Same thing for liquidsoap, and again https://example.com/api/v2/info doesn’t exist, but https://libretime.example.com/api/v2/info does.

The fix is pretty obvious: somehow I need to get nginx to redirect all requests to the API to libretime.example.com instead of example.com. In practice though, I’m not sure how to do that and I’d appreciate any hints. I can’t seem to attach files to this post, so I’m pasting my nginx default.conf below.

A more general thought also: I find it odd that LibreTime components would connect to the API through the public URL (i.e. nginx). Since this traffic is internal, I would expect the components to connect to the API container directly. I tried setting LIBRETIME_GENERAL_PUBLIC_URL to http://api and http://api:9001 to see if that would work - no luck. Am I missing something?

nginx default.conf

server {
  server_name example.com;
  listen      80;
  return      301 https://$server_name$request_uri;
}
server{
  server_name  example.com;

  location / {
    proxy_pass http://web;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  listen              [::]:443 ssl ipv6only=on; # managed by Certbot
  listen              443 ssl; # managed by Certbot
  ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; # managed by Certbot
  include             /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam         /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
  server_name libretime.example.com;
  listen      80;
  return      301 https://$server_name$request_uri;
}
server {
  server_name libretime.example.com;

  listen              443 ssl; # managed by Certbot
  ssl_certificate     /etc/letsencrypt/live/libretime.example.com/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/libretime.example.com/privkey.pem; # managed by Certbot
  include             /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam         /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  root                 /var/www/html/public;
  index                index.php index.html index.htm;
  client_max_body_size 512M;
  client_body_timeout  300s;

  location ~ \.php$ {
    fastcgi_buffers 64 4K;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;

    #try_files $uri =404;
    try_files $fastcgi_script_name =404;

    include fastcgi_params;

    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    set $path_info $fastcgi_path_info;
    fastcgi_param PATH_INFO $path_info;
    include fastcgi_params;

    fastcgi_index index.php;
    fastcgi_pass legacy:9000;
  }

  location / {
    try_files $uri $uri/ /index.php$is_args$args;
  }

  location ~ ^/api/(v2|browser) {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_redirect off;
    proxy_pass http://api:9001;
  }
}

Expected behavior
All LibreTime components should be able to access the API.

Relevant log output or error messages
playout

nginx_1            | 192.168.144.11 - - [15/Oct/2022:11:05:14 +0000] "nginx" "GET /api/version/api_key/[REDACTED] HTTP/1.1" 301 169 "-" "python-requests/2.28.1" 0.000 <"-" >"
nginx_1            | 192.168.144.1 - - [15/Oct/2022:11:05:14 +0000] "example.com" "GET /api/version/api_key/[REDACTED] HTTP/1.1" 404 174 "-" "python-requests/2.28.1" 0.002 <"-" >"
playout_1          | ERROR:root:GET https://example.com/api/version/api_key/[REDACTED] request failed '404':
playout_1          | Payload: None
playout_1          | Response:  <!DOCTYPE html>
playout_1          | <html lang="en">
playout_1          |     <head>
playout_1          |         <meta charset="UTF-8">
playout_1          |     </head>
playout_1          |     <body>
playout_1          |     	<h1>404</h1>
playout_1          |         <p>There's nothing here.</p>
playout_1          |     </body>
playout_1          | </html>
playout_1          |
playout_1          | ERROR:root:404 Client Error: Not Found for url: https://example.com/api/version/api_key/[REDACTED]
playout_1          | Traceback (most recent call last):
playout_1          |   File "/usr/local/lib/python3.10/site-packages/libretime_api_client/v1.py", line 90, in __get_api_version
playout_1          |     return self.services.version_url()["api_version"]
playout_1          |   File "/usr/local/lib/python3.10/site-packages/libretime_api_client/_utils.py", line 108, in __call__
playout_1          |     res.raise_for_status()
playout_1          |   File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
playout_1          |     raise HTTPError(http_error_msg, response=self)
playout_1          | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://example.com/api/version/api_key/[REDACTED]

liquidsoap

nginx_1            | 192.168.144.6 - - [15/Oct/2022:11:18:42 +0000] "nginx" "GET /api/v2/info HTTP/1.1" 301 169 "-" "python-requests/2.28.1" 0.000 <"-" >"
nginx_1            | 192.168.144.1 - - [15/Oct/2022:11:18:42 +0000] "example.com" "GET /api/v2/info HTTP/1.1" 404 174 "-" "python-requests/2.28.1" 0.008 <"-" >"
liquidsoap_1       | 2022-10-15 11:18:42.328 | ERROR    | libretime_api_client._client:_request:78 - 404 Client Error: Not Found for url: https://example.com/api/v2/info
liquidsoap_1       | Traceback (most recent call last):
liquidsoap_1       |   File "/usr/local/bin/libretime-liquidsoap", line 8, in <module>
liquidsoap_1       |     sys.exit(cli())
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
liquidsoap_1       |     return self.main(*args, **kwargs)
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1053, in main
liquidsoap_1       |     rv = self.invoke(ctx)
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
liquidsoap_1       |     return ctx.invoke(self.callback, **ctx.params)
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 754, in invoke
liquidsoap_1       |     return __callback(*args, **kwargs)
liquidsoap_1       |   File "/src/libretime_playout/liquidsoap/main.py", line 37, in cli
liquidsoap_1       |     info = Info(**api_client.get_info().json())
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/libretime_api_client/v2.py", line 12, in get_info
liquidsoap_1       |     return self._request("GET", "/api/v2/info", **kwargs)
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/libretime_api_client/_client.py", line 79, in _request
liquidsoap_1       |     raise exception
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/libretime_api_client/_client.py", line 74, in _request
liquidsoap_1       |     response.raise_for_status()
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
liquidsoap_1       |     raise HTTPError(http_error_msg, response=self)
liquidsoap_1       | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://example.com/api/v2/info

LibreTime version
LibreTime version: 3.0.0-beta.2

Installation method and OS / Environment
Operating system: Debian
Method: Docker compose

Cheers,
Oliver

Hello Oliver,
thanks for this detailed description, I really appreciate the effort.

So we consider the nginx container as internal as we expected people to work on the port exposed by this container. Though your setup should be supported, it seems like we didn’t considered that people might want to reuse nginx to serve there own files.

You are probably hitting this 404, because we call the API without a proper Host (https://example.com), and nginx doesn’t know how to dispatch the request to the appropriate back end.

playout → call http://nginx → nginx → go to default serve since we don’t know http://nginxhttp://example.com → no /api/version in example.com 404

I don’t think it is a good idea to bypass the nginx server for internal communication though, the playout service might want to download really large files or put some load on the api, so having the nginx server in between will always help. Having a centralized place for internal traffic logs is also a good idea.

I will have to think a bit on how to properly handle this, but for now you could try to add the http://nginx host as one of the libretime domain in the nginx config.

Putting the libretime server section at the top of the nginx config file can also be a workaround, but that’s not really clean.

You could also remove the LIBRETIME_GENERAL_PUBLIC_URL env vars from the docker-compose file, so that playout calls https://libretime.example.or directly. Communication would then take a bigger route, but I think in your case its ok as everything will be handled inside the nginx container (unless you have yet another reverse proxy between maybe your public IP and the your setup ?)

Hope this gives you enough hints to find a solution until we document a proper way to handle this.

I’ll try to reproduce your setup, and see if an idea comes to me while doing.

Cheers,
Jo

Yup, I reproduced this, and adding the nginx serve_name fixed it:

# [...]

server {
  server_name libretime.example.com nginx;

  # [...]
}

Hi Jo,

Wow. Thanks for the super quick and detailed reply.

I’ll confess that I considered that for a while :smiley:

Setting the server_name in the nginx conf results in a bunch of other errors for me:

liquidsoap_1       | Traceback (most recent call last):
liquidsoap_1       |   File "/usr/local/bin/libretime-liquidsoap", line 8, in <module>
liquidsoap_1       |     sys.exit(cli())
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
liquidsoap_1       |     return self.main(*args, **kwargs)
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1053, in main
liquidsoap_1       |     rv = self.invoke(ctx)
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
liquidsoap_1       |     return ctx.invoke(self.callback, **ctx.params)
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 754, in invoke
liquidsoap_1       |     return __callback(*args, **kwargs)
liquidsoap_1       |   File "/src/libretime_playout/liquidsoap/main.py", line 38, in cli
liquidsoap_1       |     preferences = StreamPreferences(**api_client.get_stream_preferences().json())
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/libretime_api_client/v2.py", line 39, in get_stream_preferences
liquidsoap_1       |     return self._request("GET", "/api/v2/stream/preferences", **kwargs)
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/libretime_api_client/_client.py", line 79, in _request
liquidsoap_1       |     raise exception
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/libretime_api_client/_client.py", line 74, in _request
liquidsoap_1       |     response.raise_for_status()
liquidsoap_1       |   File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
liquidsoap_1       |     raise HTTPError(http_error_msg, response=self)
liquidsoap_1       | requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://libretime.example.com/api/v2/stream/preferences
prod_liquidsoap_1 exited with code 1
playout_1          | Traceback (most recent call last):
playout_1          |   File "/usr/local/bin/libretime-playout", line 8, in <module>
playout_1          |     sys.exit(cli())
playout_1          |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
playout_1          |     return self.main(*args, **kwargs)
playout_1          |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1053, in main
playout_1          |     rv = self.invoke(ctx)
playout_1          |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
playout_1          |     return ctx.invoke(self.callback, **ctx.params)
playout_1          |   File "/usr/local/lib/python3.10/site-packages/click/core.py", line 754, in invoke
playout_1          |     return __callback(*args, **kwargs)
playout_1          |   File "/src/libretime_playout/main.py", line 84, in cli
playout_1          |     liq_version = liq_client.wait_for_version()
playout_1          |   File "/src/libretime_playout/liquidsoap/client/_client.py", line 52, in wait_for_version
playout_1          |     version = self.version()
playout_1          |   File "/src/libretime_playout/liquidsoap/client/_client.py", line 45, in version
playout_1          |     with self.conn:
playout_1          |   File "/src/libretime_playout/liquidsoap/client/_connection.py", line 53, in __enter__
playout_1          |     self.connect()
playout_1          |   File "/src/libretime_playout/liquidsoap/client/_connection.py", line 74, in connect
playout_1          |     self._sock = create_connection(
playout_1          |   File "/usr/local/lib/python3.10/socket.py", line 824, in create_connection
playout_1          |     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
playout_1          |   File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
playout_1          |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
playout_1          | socket.gaierror: [Errno -2] Name or service not known
prod_playout_1 exited with code 1

However I made it work by removing the LIBRETIME_GENERAL_PUBLIC_URL env from all containers. At this point I just want to get it to work, so I’ll stick with this solution for now.

Well, one step closer… now this :slight_smile:

liquidsoap_1       | 2022/10/15 14:37:36 [LibreTime!:3] Connecting mount main for source@icecast...
liquidsoap_1       | 2022/10/15 14:37:36 [LibreTime!:2] Connection failed: could not read data from host: Connection reset by peer in read()
liquidsoap_1       | 2022/10/15 14:37:36 [lang:3] timeout --signal=KILL 45 libretime-playout-notify stream '1' '1665844644.1' --error='could not read data from host: Connection reset by peer in read()' &
liquidsoap_1       | 2022/10/15 14:37:36 [LibreTime!:3] Will try again in 5.00 sec.
liquidsoap_1       | 2022-10-15 14:37:37.161 | INFO     | libretime_playout.notify.main:stream:92 - Sending output stream '1' status 'could not read data from host: Connection reset by peer in read()'

I’ll see if I can figure it out by myself and start a new thread if not.

Thanks again for the help! Enjoy your weekend.
Oliver

You could also try to set an alias in the docker-compose stack and remove the LIBRETIME_GENERAL_PUBLIC_URL env var for the rest of the containers:

  nginx:
    image: nginx
    ports:
      - 8080:80
    depends_on:
      - legacy
    volumes:
      - libretime_assets:/var/www/html:ro
      - ${NGINX_CONFIG_FILEPATH:-./nginx.conf}:/etc/nginx/conf.d/default.conf:ro
    networks:
      default:
        aliases:
          - libretime.example.com

So i guess this has to do with subdomains. We also use subdomains https://www. goes to our own website whereas https://lt. (as example) is redirected to the libretime application. At the moment we are setting up a testserver to make the transition from Libretime Alpha (with Apache) to Libretime Beta as smooth as possible. We are testing different scenario’s. Are there certain points we need to pay extra attention to?
Kind regards,
Serge