Erlang for Procedural Programmers, Part III

The C10k Problem is about handling ten thousand connections at the same time. Let’s use the Erlang from Part I and Part II to write a TCP echo server that can handle ten thousand clients, in this case on a Raspberry Pi. I’ll assume you’re familiar with socket programming in C, but if you’re not then you ought to check out the excellent Beej’s Guide to Network Programming.

Let’s start with the skeleton of a module that provides a start function with two arguments:

-module(xcho).
-export([start/2]).

start(Port, ListenerCount) -> ok.

In C we’d use assertions and/or conditionals to validate arguments, but since this is Erlang we’ll use when statements:

-module(xcho).
-export([start/2]).

start(Port, ListenerCount) when Port > 0, Port < 65536, ListenerCount > 0 -> 
    ok.

Now that we have valid inputs, let’s look into Erlang’s gen_tcp module - we’ll use the listen/2 function to listen for connections. It’s declared as such:

listen(Port, Options) -> {ok, ListenSocket} | {error, Reason}

Meaning it takes a port and port options and returns a tuple of either the atom ok and a valid listening socket or the atom error and a reason. The call to listen will look like this:

{ ok, LSocket } = gen_tcp:listen(Port, [binary, {packet, 0}, {active, false}])

and since atoms can only assign to themselves, the Erlang VM will throw an error if gen_tcp:listen/2 returns anything but a tuple starting with the ok atom - we don’t need error checking or validation in later parts since they will never be reached if an error occurs.

The way the app will work is to start ListenerCount processes to accept connections, and each of these processes runs an infinite loop where it waits for a connection to accept and once one arrives it spawns another process to handle that connection and goes back to listening. Let’s write the code from the bottom-up, starting with the function that handles a client connection:

connection(Socket) ->
    loop(Socket),
    ok = gen_tcp:close(Socket).

loop(Socket) -> 
    case gen_tcp:recv(Socket, 0) of 
        { ok, X } -> gen_tcp:send(Socket, X), loop(Socket);
        { error, closed } -> ok;
        _ -> error
    end.

Here connection/1 calls loop/1 and then closes the socket. loop/1 keeps echoing back whatever it receives until the connection is closed or an error occurs. loop/1 works by calling gen_tcp:recv/2 then pattern matching the result: if it receives an input it sends it back using gen_tcp:send/2 then it calls itself - this doesn’t lead to infinite recursion exhausting stack space, as Erlang has well-defined behavior for tail-recursive functions like this. In this case the name of the function loop/1 is descriptive of what it effectively does: it’s a loop, even though it’s one that masquarades as a function - a very common idiom in Erlang.

The listen function that runs on ListenerCount threads is mostly straightforward:

listen(LSocket) -> 
    { ok, Socket } = gen_tcp:accept(LSocket),
    Pid = spawn(fun () -> connection(Socket) end),
    ok = gen_tcp:controlling_process(Socket, Pid),
    listen(LSocket).

The part that’s specific to Erlang is the call to gen_tcp:controlling_process/2, which transfers the ownership of the socket to the spawned process.

Putting it all together, the start/2 function should be straightforward:

start(Port, ListenerCount) when Port > 0, Port < 65536, ListenerCount > 0 -> 
    { ok, LSocket } = gen_tcp:listen(Port, [binary, {packet, 0}, {active, false}]),
    lists:foreach(fun (_) -> spawn(fun () -> listen(LSocket) end) end, lists:seq(1, ListenerCount-1)),
    listen(LSocket),
    ok = gen_tcp:close(LSocket).

It uses foreach to spawn the listener processes, and acts as a listener itself. Here’s the final, complete program:

-module(xcho).
-export([start/2]).

start(Port, ListenerCount) when Port > 0, Port < 65536, ListenerCount > 0 -> 
    { ok, LSocket } = gen_tcp:listen(Port, [binary, {packet, 0}, {active, false}]),
    lists:foreach(fun (_) -> spawn(fun () -> listen(LSocket) end) end, lists:seq(1, ListenerCount-1)),
    listen(LSocket),
    ok = gen_tcp:close(LSocket).

listen(LSocket) -> 
    { ok, Socket } = gen_tcp:accept(LSocket),
    Pid = spawn(fun () -> connection(Socket) end),
    ok = gen_tcp:controlling_process(Socket, Pid),
    listen(LSocket).

connection(Socket) ->
    loop(Socket),
    ok = gen_tcp:close(Socket).

loop(Socket) -> 
    case gen_tcp:recv(Socket, 0) of 
        { ok, X } -> gen_tcp:send(Socket, X), loop(Socket);
        { error, closed } -> ok;
        _ -> error
    end.

Running it on a Raspberry Pi 3B+ running Raspbian with a desktop environment, I get ~9200 clients before connections start to drop. There are a few things that can be considered to cross that final gap of ~800 users: running headless / Raspbian server, adding active cooling (I had a passive heatsink on my Pi), and exploring HiPE - natively compiling Erlang to the underlying architecture - but in my case I just cooled down the Pi in a freezer for a short while and it worked so I stopped there.

The code is pretty concise, and it gets you a lot of robust concurrency in 25 lines of code. There’s still a lot to learn about Erlang - I didn’t even cover interprocess message passing and multi-node setups, but this series should give procedural programmers enough info to get the ball rolling. Joe Armstrong’s Thesis is an excellent read, very well written from both a technical and an editorial standpoint, and it explains the rationale for a lot of the decisions, and is a great guide for building reliable software-based systems. And when software-based systems are reliable everybody wins.

Sherief, FYI

Erlang for Procedural Programmers, Part III