Make EVP_PKEY_asn1_new() stricter with its input

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from openssl#6880)

(cherry picked from commit 38eca7f)

Author: levitte

CHANGES

@@ -9,6 +9,11 @@
 
  Changes between 1.1.0h and 1.1.0i [xx XXX xxxx]
 
+  *) Make EVP_PKEY_asn1_new() a bit stricter about its input.  A NULL pem_str
+     parameter is no longer accepted, as it leads to a corrupt table.  NULL
+     pem_str is reserved for alias entries only.
+     [Richard Levitte]
+
   *) Revert blinding in ECDSA sign and instead make problematic addition
      length-invariant. Switch even to fixed-length Montgomery multiplication.
      [Andy Polyakov]

crypto/asn1/ameth_lib.c

@@ -255,6 +255,18 @@ EVP_PKEY_ASN1_METHOD *EVP_PKEY_asn1_new(int id, int flags,
             goto err;
     }
 
+    /*
+     * One of the following must be true:
+     *
+     * pem_str == NULL AND ASN1_PKEY_ALIAS is set
+     * pem_str != NULL AND ASN1_PKEY_ALIAS is clear
+     *
+     * Anything else is an error and may lead to a corrupt ASN1 method table
+     */
+    if (!((pem_str == NULL && (flags & ASN1_PKEY_ALIAS) != 0)
+          || (pem_str != NULL && (flags & ASN1_PKEY_ALIAS) == 0)))
+        goto err;
+
     if (pem_str) {
         ameth->pem_str = OPENSSL_strdup(pem_str);
         if (!ameth->pem_str)