Stop raising ERR_R_MALLOC_FAILURE in most places

Since OPENSSL_malloc() and friends report ERR_R_MALLOC_FAILURE, and
at least handle the file name and line number they are called from,
there's no need to report ERR_R_MALLOC_FAILURE where they are called
directly, or when SSLfatal() and RLAYERfatal() is used, the reason
`ERR_R_MALLOC_FAILURE` is changed to `ERR_R_CRYPTO_LIB`.

There were a number of places where `ERR_R_MALLOC_FAILURE` was reported
even though it was a function from a different sub-system that was
called.  Those places are changed to report ERR_R_{lib}_LIB, where
{lib} is the name of that sub-system.
Some of them are tricky to get right, as we have a lot of functions
that belong in the ASN1 sub-system, and all the `sk_` calls or from
the CRYPTO sub-system.

Some extra adaptation was necessary where there were custom OPENSSL_malloc()
wrappers, and some bugs are fixed alongside these changes.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19301)

1 files changed, 5 insertions, 5 deletions

diff --git a/crypto/x509/x509_obj.c b/crypto/x509/x509_obj.c
index 12c6d6f78b..2af7203b01 100644
--- a/crypto/x509/x509_obj.c
+++ b/crypto/x509/x509_obj.c
@@ -41,9 +41,9 @@ char *X509_NAME_oneline(const X509_NAME *a, char *buf, int len)
 
     if (buf == NULL) {
         if ((b = BUF_MEM_new()) == NULL)
-            goto err;
+            goto buferr;
         if (!BUF_MEM_grow(b, 200))
-            goto err;
+            goto buferr;
         b->data[0] = '\0';
         len = 200;
     } else if (len == 0) {
@@ -124,7 +124,7 @@ char *X509_NAME_oneline(const X509_NAME *a, char *buf, int len)
         }
         if (b != NULL) {
             if (!BUF_MEM_grow(b, l + 1))
-                goto err;
+                goto buferr;
             p = &(b->data[lold]);
         } else if (l > len) {
             break;
@@ -179,8 +179,8 @@ char *X509_NAME_oneline(const X509_NAME *a, char *buf, int len)
     if (i == 0)
         *p = '\0';
     return p;
- err:
-    ERR_raise(ERR_LIB_X509, ERR_R_MALLOC_FAILURE);
+ buferr:
+    ERR_raise(ERR_LIB_X509, ERR_R_BUF_LIB);
  end:
     BUF_MEM_free(b);
     return NULL;