Avoid an infinite loop in BN_GF2m_mod_inv

If p is set to 1 when calling BN_GF2m_mod_inv then an infinite loop will
result. Calling this function set 1 when applications call this directly
is a non-sensical value - so this would be considered a bug in the caller.

It does not seem possible to cause OpenSSL internal callers of
BN_GF2m_mod_inv to call it with a value of 1.

So, for the above reasons, this is not considered a security issue.
Reported by Bing Shi.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Todd Short <todd.short@me.com>
(Merged from https://github.com/openssl/openssl/pull/22960)

1 files changed, 7 insertions, 1 deletions

diff --git a/crypto/bn/bn_gf2m.c b/crypto/bn/bn_gf2m.c
index 83e1f11e18..444c5ca7a3 100644
--- a/crypto/bn/bn_gf2m.c
+++ b/crypto/bn/bn_gf2m.c
@@ -730,14 +730,20 @@ int BN_GF2m_mod_inv(BIGNUM *r, const BIGNUM *a, const BIGNUM *p, BN_CTX *ctx)
 {
     BIGNUM *b = NULL;
     int ret = 0;
+    int numbits;
 
     BN_CTX_start(ctx);
     if ((b = BN_CTX_get(ctx)) == NULL)
         goto err;
 
+    /* Fail on a non-sensical input p value */
+    numbits = BN_num_bits(p);
+    if (numbits <= 1)
+        goto err;
+
     /* generate blinding value */
     do {
-        if (!BN_priv_rand_ex(b, BN_num_bits(p) - 1,
+        if (!BN_priv_rand_ex(b, numbits - 1,
                              BN_RAND_TOP_ANY, BN_RAND_BOTTOM_ANY, 0, ctx))
             goto err;
     } while (BN_is_zero(b));