Firefox WebRTC GMP H.264 heap overflow via short EncodedImage (CVE-2026-2757)

Author

Joshua Rogers

Date Published

Firefox CVE-2026-2757

AISLE's autonomous security analyzer discovered a buffer overflow vulnerability in Mozilla's WebRTC that affected both Firefox and Thunderbird, the two major consumers of the WebRTC library. This marks AISLE's fourth high severity vulnerability finding in Firefox, with CVE-2025-13016, CVE-2025-14321, and CVE-2025-14331 previously being assigned for other findings. Our analyzer also found high severity vulnerabilities in Chrome, as well as Safari's WebKit.

Today's vulnerability, CVE-2026-2757, provides an attacker-controlled out-of-bounds read/write primitive. It was discovered in Firefox's WebRTC H.264 decoding path when using the Gecko Media Plugin (GMP) decoder. Specifically, WebrtcGmpVideoDecoder::Decode_g() could be reached with undersized EncodedImage buffers (1-3 bytes), causing a 4-byte header write past the end of an allocation and a size underflow that turns a copy into a massive out-of-bounds read/write in the GMP process. Mozilla fixed the issue and shipped the patch in Firefox 148.

This blog sets out the technical details on the finding.

Background: WebRTC, H.264, and the GMP decoder path

Firefox's WebRTC stack supports multiple decoding backends for video codecs like H.264. In most common configurations, H.264 decoding is handled by platform decoders or in-process pipelines. Firefox also has a fallback path that routes H.264 decode through Gecko Media Plugins (GMPs), historically used for components like OpenH264.

When WebRTC receives an encoded H.264 frame, it is represented as a webrtc::EncodedImage and passed into the decoder implementation. In the GMP path, Firefox wraps the encoded bytes into a GMPVideoEncodedFrame and sends it across the GMP boundary for decoding in the plugin process. That packaging step is where the boundary condition bug lived.

Root cause

The vulnerable logic was in dom/media/webrtc/libwebrtcglue/WebrtcGmpVideoCodec.cpp, in the handoff from the WebRTC EncodedImage to a GMPVideoEncodedFrame.

At a high level, the code:

  1. Only rejected zero-length inputs in WebrtcGmpVideoDecoder::Decode().
  2. Allocated a GMPVideoEncodedFrame sized exactly to aDecodeData->mImage.size().
  3. Unconditionally wrote a 4-byte header into the frame buffer.
  4. Copied the remaining payload as size - 4 bytes.

The critical bug is that steps (3) and (4) assume size >= 4, but step (1) only enforced size != 0. For 1-3 byte frames:

  • The 4-byte header write overruns the allocation.
  • The size - 4 computation underflows (as an unsigned size), yielding a huge length that turns memcpy into an out-of-bounds read/write operation.

Here’s the (simplified) code path:

C/C++
1// dom/media/webrtc/libwebrtcglue/WebrtcGmpVideoCodec.cpp
2int32_t WebrtcGmpVideoDecoder::Decode(const webrtc::EncodedImage& aInputImage,
3                                      bool aMissingFrames,
4                                      int64_t aRenderTimeMs) {
5  ...
6  if (!aInputImage.size()) {
7    return WEBRTC_VIDEO_CODEC_ERROR;
8  }
9  ...
10}
11

12void WebrtcGmpVideoDecoder::Decode_g(UniquePtr<GMPDecodeData>&& aDecodeData) {
13  ...
14  err = frame->CreateEmptyFrame(aDecodeData->mImage.size());
15  ...
16  *(reinterpret_cast<uint32_t*>(frame->Buffer())) = frame->Size();
17  memcpy(frame->Buffer() + 4, aDecodeData->mImage.data() + 4,
18         frame->Size() - 4);
19  ...
20}

Triggering the bug

The condition is simple: get the GMP H.264 decoder path to attempt to decode an EncodedImage smaller than 4 bytes.

A practical way to do that in the browser is to use WebRTC Encoded Transforms to modify frames immediately before decode. In our PoC that we sent Mozilla, we set up a local WebRTC loopback, negotiated H.264, attached an RTCRtpScriptTransform on the receiver, and truncated the incoming encoded frames down to 1-3 bytes before they reach WebrtcGmpVideoDecoder::Decode_g(). We also configure Firefox to use the GMP H.264 decoder path for WebRTC by setting media.navigator.mediadatadecoder_h264_enabled=false (forces the GMP decoder path), and media.navigator.mediadatadecoder_vpx_enabled=false (avoids codec fallback). We also enable a deterministic media source and ScriptTransforms with media.navigator.streams.fake=true media.peerconnection.scripttransform.enabled=true.

For the time being, we leave out the full JavaScript PoC, to allow users to update to a patched version of Firefox.

Read/Write behavior

This bug yields memory safety primitives inside the GMP process:

READ:

  • The source pointer advances by 4 (mImage.data() + 4), which is out-of-bounds for undersized inputs. Combined with the underflowed copy size, this becomes an out-of-bounds read.

WRITE:

  • The 4-byte header store overflows for size < 4.
  • The destination memcpy range becomes effectively unbounded due to the underflowed length, producing out-of-bounds writes as well.

Fix

The fix is to make the size assumption explicit and non-bypassable by validating the buffer length before doing any size - 4 arithmetic, and to use checked arithmetic to avoid implicit conversion and underflow pitfalls.

Mozilla's landed patch uses checked arithmetic (CheckedInt<uint32_t>) so that undersized frames and invalid sizes are rejected safely, rather than reaching the header write and memcpy. The fix landed as commit c5b22051c757.

Timeline

  • 2025-11-21 Reported by Igor Morgenstern of the AISLE Research Team.
  • 2026-02-24 Fixed release shipped in Firefox 148.

Increasing Coverage with AISLE

CVE-2026-2757 is a representative example of just one of the endless bug classes in which AISLE's analyzer shows its true skill: boundary-condition violations at trust boundaries and format adapters, where small inputs slip through a permissive front-door check and reach code that assumes a minimum size.

If you're interested in learning about more vulnerabilities found with AISLEs analyzer, check out our other posts for more findings, or reach us at [email protected].