Does anyone know of a good way to avoid port number clashes with ephemeral ports on Linux? I forward local ports such as 33070 into containers (e.g. 33070 -> 3306) so I can connect to services running with them from the host. Occasionally though I get conflicts because another piece of software has chosen that port - usually for a HTTP connection.
Is there a range I can use / avoid?
@pwaring you could add another loopback IP, e.g. 127.0.0.2, or use another Private Net IP. You could even add dockerhost as a name instead of localhost.
@pwaring You can use `lsof` to check if the port you want to use is already in use in a script, e.g. test a random port > 1024 until you find an open one.
`lsof -nP -iTCP -sTCP:LISTEN`
Though come to think of it, that would not work because I think you would not actively use the port, so other apps would still try to get a socket on the port.
@pwaring I believe that some software permits port 0 to be specified, meaning "you find an available ephemeral port for me". I'd expect any such software to report the port number on startup, which you could read from the container's log stream.
@bencardoen I don't think that would work as Docker requires the port to be specified in its docker-compose.yml file. Docker will always use the same port, so I want to make sure that it doesn't clash with an ephemeral port.
@pwaring Alternatively, if the problem isn't that the containerised service uses a contended port; but instead it's that your choice of port to map to the service's standard port has a clash, then this section from the Docker docs seems to have you somewhat covered: https://docs.docker.com/get-started/docker-concepts/running-containers/publishing-ports/#publishing-to-ephemeral-ports
@pwaring echo "49152 65535" > /proc/sys/net/ipv4/ip_local_port_range
Now outgoing connections will be made from the specified range, and the others will (probably) be OK for you to use?
@marek Thanks, that's what I'm looking for - I'm not going to change the list but now I know what the list *is* I can make sure I choose a port which isn't in that range.
@jonathanmatthews It's the static port forwarding I have in Docker Compose that's the issue - sometimes that clashes with an ephemeral port. I don't want Docker to use an ephemeral port because it will change every time and I'll have to update all the software that assumes it (e.g. DBeaver which I use for connecting to MySQL).
So it turns out that the problem with ephemeral ports is that there is a standard which species the range. FreeBSD and Windows follow that RFC but Linux does not (or at least not the distribution I'm using).
cat /proc/sys/net/ipv4/ip_local_port_range
32768 60999
RFC 6335:
the System Ports, also known as the Well Known Ports, from 0-1023
the User Ports, also known as the Registered Ports, from 1024-49151
the Dynamic Ports, also known as the private or Ephemeral Ports, from 49152-65535
What I need is to avoid:
System Ports: These require root access to bind to anyway
User Ports: Any that have been assigned by IANA (e.g. 3306 for MySQL)
Dynamic Ports: These can be used at any time by an application so can't be relied on (chance of a conflict is small but not zero)
If Docker uses an ephemeral port, you can use something like Traefik to route to that ephemeral port with no management effort.
- https://doc.traefik.io/traefik/providers/docker/#port-detection
Hard coding ports get tricky when config files need to be changed by end users due to port conflicts.
@inawhilecrocodile @jonathanmatthews I'm not sure if that solves the problem - I always want to connect to a given local port for a given container. I do not want that port to change because then I have to change my config everywhere else (I have 20+ Docker projects).
If Traefik routes to an ephemeral port then isn't that just shifting the problem of a port clash from Docker to Traefik (as well as adding another layer of complexity)?
- replies
- 1
- announces
- 0
- likes
- 0
Traefik is acting as a reverse proxy. Its listening on a port for incoming requests and it routes those to downstream containers based on whatever routing configuration you set up.
You don't have to think about container ports. You think about container names/labels/what hostname or path needs to go to where.
Maybe the following link will give you a better idea:
It is just a different way of doing things.
@jmtd Thanks, Podman is on my list of things to look at but I haven't had chance yet (also I'm wary of diverting too far from the setups of other developers on the same project as it's hard enough when we have a mix of Windows, macOS and Linux...)
@pwaring @bencardoen doesn't docker assign a random host port of one is not specified in the mapping configuration? So if an extra lookup post-start isn't a deal-breaker them that's probably the simplest solution here
@pwaring @bencardoen Yup. The docs even refer to this as ephemeral ports https://docs.docker.com/get-started/docker-concepts/running-containers/publishing-ports/#publishing-to-ephemeral-ports
@nCrazed @bencardoen It does assign a random host port if I don't specify one, but I don't want that behaviour because I have applications configured to connect to that port. If it's random every time I have to reconfigure all my other software (e.g. DBeaver).
@pwaring @nCrazed This cloudflare post has some interesting relevant concepts, e.g. (but have not tested) seems like you can sysctl the ephemeral port range. https://blog.cloudflare.com/how-to-stop-running-out-of-ephemeral-ports-and-start-to-love-long-lived-connections/
Other options could be to use cgroups/ulimit to restrict port use inside the container? I tested none of these, so ymmv, but what if you can fence off a range of ports that the kernel can't assign to user space, and therefore would be guaranteed yours at creation to fwd?