It's common to spin up a server in a test so that you can do full end-to-end requests of it. It's a very important sort of test, to make sure things work all together. Most of the work I do is in complex web backends, and there's so much risk of not having all the request processing and middleware and setup exactly the same in a mock test... you must do at least some end-to-end tests or you're making a gamble that's going to bite you.

And this is great, but you quickly run into a problem: port collisions! This can happen when you run multiple tests at once and all of them start a separate server, and whoops, two have picked the same port. Or it can happen if something else running on your development machine happens to be running on the port you chose. It's annoying when it happens, too, because it's often hard to reproduce.

So... how do we fix that? You read the title^[1], so you know where we're going, but let's go there together.

Can we pick a random port?

There are a few potential solutions to this. Perhaps the most obvious is binding to a port you choose randomly. This will work a lot of the time, but it's going to be flaky. You can drive down the probability of collision, but it's going to happen sometimes. Side note, I think the only thing worse than a test that fails 10% of the time is one that fails 1% of the time. It's not flaky enough to drive urgency for anyone to fix it, but it's flaky enough that in a team context, you will run into this on a daily basis. Ask me how I know.

How often you get a collision depends on a lot of factors. How many times do you bind a port in the range? How many other services might bind something in that range? How likely are two things to run concurrently?

As a simple example, let's say we pick a random port in the range 9000-9999, and you have 4 concurrent tests that will overlap. If you uniformly sample from this range, then you will have a 1/1000 chance of a collision from the second test, a 2/1000 chance from the third, and a 3/1000 chance from the fourth. Our probability of having no collision is (999/1000 * 998/1000 * 997/1000) = 0.994. That means that we have a 0.6% chance of a collision. This isn't horrible, but it's not great!

Incremental ports?

We could also have each test increment the port it picks by 1. I've done this before, and it avoids one set of problems from collisions, but it makes a new problem. Now you're sweeping across the entire range starting from the first port. If you have anything else running on your system that binds in that range, you'll run into a collision!

And if you run your entire test suite in parallel, you're much more likely to have a problem now, since they all start at the same port.

Sooooo ephemeral ports from the kernel?

The problem we've had all along is that we don't have full information. If we know the system state and all the currently open ports, then binding to one that's not in use is an easy problem. And you know who knows all that info? The kernel does.

And it turns out, this is something we can ask the kernel for. We can just say "please give me a nice unused port" and it will! There's a range of ports that the kernel uses for this. It varies by system, but it's not usually very relevant what the particular range is. On my system, I can find the range by checking /proc/sys/net/ipv4/ip_local_port_range. My ephemeral port range is from 32768 to 60999. I'm curious why the range stops there instead of going all the way up, so that's a future investigation.

To get an ephemeral port on Linux systems, you bind or listen on port 0. Then the kernel will hand you back a port in the ephemeral range. And you know that it's available, since the kernel is keeping track. It's possible to have an issue here if the full range of ports has been exhausted but, you know what, if you hit that limit, you probably have other problems^[2].

The only thing is that if you've bound to an unknown port, how do you send requests to it? We can get the port we've bound to by another syscall, getsockname(2). This lets us find out what address a socket is bound to, and then we can do something with that information. For tests, that means that you'll need to find a way to communicate this port from the listener to the requester. If they're in the same process, I like to do this by either injecting in the listener or returning the address. If you're doing something like postgres or redis on an ephemeral port, then you'd probably have to find the port from its output, which is tedious but doable.

Here's an example from a web app I'm working on. This is how a simple test looks. We launch the web server, binding to port 0, and get the address back. Then we can send requests to that address!

#[tokio::test]
pub async fn gets_200_healthcheck() {
    let config = test_config().unwrap();

    let (addr, handle) = launch_webserver("localhost", 0, app(&config))
        .await
        .expect("server should launch");

    let url = format!("http://{addr}/_health");
    let resp = reqwest::get(url).await.expect("should get response");
    assert_eq!(resp.status().as_u16(), 200);

    abort_and_wait(handle).await;
}

And inside launch_webserver, the relevant two lines are:

let listener = TcpListener::bind(format!("{host}:{port}")).await?;
let addr = listener.local_addr()?;

...where port = 0 in our case.

That's all we have to do, and we'll get a much more reliable test setup.

1. I think suspenseful titles can be fun, improve storytelling, and drive attention. But sometimes you really need a clear, honest, spoiler of a title. Giving away the answer is great when you're giving information that people might want to quickly internalize. ↩

2. If you do run into this, I'm very curious to hear about the circumstances. It's the kind of problem that I'd love to look at and work on. It's kind of messy, and you know that there's something very interesting that led to it being this way. ↩