CVE-2025-11187: OpenSSL PKCS#12 Stack Overflow & DoS

Author

Joshua Rogers

Date Published

OpenSSL CVE-2025-11187

It would be difficult to overstate OpenSSL's role in modern internet security. In fact, if you're reading this over HTTPS, it's more than likely that OpenSSL or a derivative handled the encryption one way or another.

The AISLE Research Team recently analyzed OpenSSL with our autonomous system and discovered all 12 security vulnerabilities in the January 2026 coordinated release. In this article, we'll walk through a moderate severity vulnerability, CVE-2025-11187, which can trigger a stack-based buffer overflow as well as a NULL or invalid pointer dereference during MAC verification.

CVE-2025-11187: Improper Validation of PBMAC1 Parameters in PKCS#12 MAC Verification

The vulnerability lives in crypto/pkcs12/p12_mutl.c, in the PBMAC1_PBKDF2_HMAC() function. When OpenSSL verifies a PKCS#12 file that uses PBMAC1 for its MAC, it reads parameters from the file and uses them without validation. Two of those parameters can trigger memory corruption:

  1. The keylength parameter can cause a stack-based buffer overflow
  2. The salt parameter can cause a NULL or invalid pointer dereference

Both paths can crash the application, and the buffer overflow may enable code execution depending on platform mitigations.

Affected versions: OpenSSL 3.4, 3.5, and 3.6. Earlier versions (3.3, 3.0, 1.1.1, 1.0.2) are not affected because they don't support PBMAC1.

Background: PKCS#12 and PBMAC1

To understand how CVE-2025-11187 can be exploited, you need a bit of background on how PKCS#12 files work. PKCS#12 is built on ASN.1, a famously complicated notation which, despite the best efforts of security engineers to kill its usage, lives on in real world applications today. It packages private keys, certificates, and related data into a single password-protected container with encryption and integrity checks.

PBMAC1 is a password-based MAC scheme that derives a proper MAC key from a password using a KDF such as PBKDF2 and then applies a MAC (typically HMAC), and it is commonly represented as an ASN.1 AlgorithmIdentifier with parameters describing the KDF and MAC.

The vulnerability arises during verification of PBMAC1-protected PKCS#12 files, where a maliciously crafted file can manipulate the parsing logic.

The parameters involved are:

  • Salt: Random data mixed with the password during key derivation
  • Iteration count: Number of PBKDF2 iterations
  • Key length: The length of the derived key

Root Cause Analysis

The vulnerability is reachable through OpenSSL's PKCS#12 parsing APIs. When an application calls PKCS12_parse() to extract keys and certificates from a .p12 file, or PKCS12_verify_mac() to check the file's integrity, OpenSSL verifies the MAC before proceeding. For PBMAC1-protected files, this triggers the following call chain:

C
1PKCS12\_parse() or PKCS12\_verify\_mac()
2  → pkcs12\_gen\_mac()
3

4    → PBMAC1\_PBKDF2\_HMAC()  ← vulnerable function

The vulnerable function derives a MAC key from a password using parameters read directly from the PKCS#12 file. Here's the problematic code path, simplified for readability:

C
1static int PBMAC1_PBKDF2_HMAC(OSSL_LIB_CTX *ctx, const char *propq,
2    const char *pass, int passlen,
3    const X509_ALGOR *macalg, unsigned char *key)
4{
5    PBKDF2PARAM *pbkdf2_param = NULL;
6    int keylen = 0;
7    const ASN1_OCTET_STRING *pbkdf2_salt = NULL;
8    pbkdf2_param = PBMAC1_get1_pbkdf2_param(macalg);
9    
10    // ... digest algorithm lookup ...
11    // VULNERABLE: salt used without type validation
12    pbkdf2_salt = pbkdf2_param->salt->value.octet_string;
13    // VULNERABLE: keylength used without bounds checking
14    keylen = ASN1_INTEGER_get(pbkdf2_param->keylength);
15

16    // Key derivation writes 'keylen' bytes to 'key' buffer
17    PKCS5_PBKDF2_HMAC(pass, passlen, pbkdf2_salt->data, pbkdf2_salt->length,
18

19                      iter, kdf_md, keylen, key);
20

21}

The key buffer is allocated on the stack in the calling function:

C
1static int pkcs12_gen_mac(PKCS12 *p12, const char *pass, int passlen,
2    unsigned char *mac, unsigned int *maclen, ...)
3{
4    unsigned char key[EVP_MAX_MD_SIZE], *salt;  // 64-byte stack buffer
5    // ...
6}

EVP_MAX_MD_SIZE is 64 bytes.

Attack Path 1: Stack-Based Buffer Overflow (CWE-787)

If the keylength field in the PKCS#12 file is set to a value larger than 64, PBKDF2 writes beyond the buffer boundary. The overflow length is controlled by the attacker.

For instance, a file with keylength = 500 causes PBKDF2 to write 500 bytes into a 64-byte buffer, corrupting 436 bytes of adjacent stack memory. This can overwrite return addresses, saved registers, or other local variables.

Attack Path 2: NULL/Invalid Pointer Dereference (CWE-476)

The PBKDF2 salt field in ASN.1 is defined as a CHOICE type:

C
1salt CHOICE {
2    specified    OCTET STRING,
3    otherSource  AlgorithmIdentifier
4}

The code assumed salt was always an OCTET STRING and accessed salt->value.octet_string directly. If an attacker provides a malformed salt with a different ASN.1 type, the dereference reads garbage or NULL, causing a crash.

The Fix

At its root, CVE-2025-11187 is a straightforward input validation bug where the code trusted parameters from a file format that attackers can craft. After our autonomous analyzer found the vulnerability, it recommended a fix that was accepted by the OpenSSL team. The fix adds two simple checks to prevent exploitation:

C
1/* Validate salt is an OCTET STRING choice */
2if (pbkdf2_param->salt == NULL
3    || pbkdf2_param->salt->type != V_ASN1_OCTET_STRING) {
4

5    ERR_raise(ERR_LIB_PKCS12, PKCS12_R_PARSE_ERROR);
6    goto err;
7}
8

9pbkdf2_salt = pbkdf2_param->salt->value.octet_string;
10

11/* RFC 9579 specifies missing key length as invalid */
12if (pbkdf2_param->keylength != NULL)
13    keylen = ASN1_INTEGER_get(pbkdf2_param->keylength);
14

15if (keylen <= 0 || keylen > EVP_MAX_MD_SIZE) {
16

17    ERR_raise(ERR_LIB_PKCS12, PKCS12_R_PARSE_ERROR);
18    goto err;
19}

Now, the salt is checked to confirm it's actually an OCTET STRING before dereferencing. Also, the key length is bounded to EVP_MAX_MD_SIZE (64 bytes), matching the buffer size.

Note that the fix enforces RFC 9579's requirement that keyLength must be present and valid.

Why Severity is "Moderate"

Despite being a stack overflow with attacker-controlled length, this CVE received a CVSS score of 6.1 (Medium). Here’s why:

  1. Limited attack surface. PKCS#12 files store private keys, which applications rarely accept from untrusted sources. Instead, you'd typically import a .p12 file that you exported yourself or received from your IT department.
  2. User interaction required. Because the victim must open or process the malicious file, this isn't a remote, unauthenticated attack. An attacker would have to socially engineer someone to import a crafted certificate bundle.
  3. Platform mitigations. Modern systems deploy stack canaries, ASLR, and DEP. While these don't make exploitation impossible, you'd have to bypass these protections to do so reliably.
  4. FIPS modules unaffected. PKCS#12 processing happens outside the FIPS module boundary in OpenSSL 3.4, 3.5, and 3.6, so organizations running FIPS-validated configs aren't exposed through that path.

Conclusion

Though the attack surface was limited, OpenSSL's ubiquity means that any stack overflow with attacker-controlled length in its codebase is a serious vulnerability. If your application processes PKCS#12 files, make sure you update to OpenSSL 3.4.1, 3.5.1, or 3.6.1.

To read more about the discoveries we've made using our autonomous analyzer, check out these resources:

And if you have questions about AISLE's research, contact us at [email protected]

Our appreciation goes to Tomáš Mráz, Matt Caswell, Neil Horman, and the OpenSSL team for their collaboration throughout this process. This vulnerability was discovered by Stanislav Fort and Petr Šimeček of the AISLE Research Team. Hamza of Metadust discovered it independently 33 days later.