Extend X509 cert checks and error reporting in v3_{purp,crld}.c and x509_{set,vfy}.c

add various checks for malformedness to static check_chain_extensions() in x509_vfc.c
improve error reporting of X509v3_cache_extensions() in v3_purp.c
add error reporting to x509_init_sig_info() in x509_set.c
improve static setup_dp() and related functions in v3_purp.c and v3_crld.c
add test case for non-conforming cert from https://tools.ietf.org/html/rfc8410#section-10.2

Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/12478)

2 files changed, 41 insertions, 0 deletions

diff --git a/doc/internal/man3/x509v3_cache_extensions.pod b/doc/internal/man3/x509v3_cache_extensions.pod
new file mode 100644
index 0000000000..3fb7609daa
--- /dev/null
+++ b/doc/internal/man3/x509v3_cache_extensions.pod
@@ -0,0 +1,40 @@
+=pod
+
+=head1 NAME
+
+x509v3_cache_extensions
+- cache info on various X.509v3 extensions and further derived certificate data
+
+=head1 SYNOPSIS
+
+ #include <openssl/x509v3.h>
+
+ int x509v3_cache_extensions(X509 *x, OPENSSL_CTX *libctx, const char *propq);
+
+=head1 DESCRIPTION
+
+This function processes any X509v3 extensions present in an X509 object I<x>
+and caches the result of that processing as well as further derived info,
+for instance whether the certificate is self-issued or has version X.509v1.
+It computes the SHA1 digest of the certificate using the default library context
+and property query string and stores the result in x->sha1_hash.
+It sets B<X509_SIG_INFO_VALID> in x->flags if x->siginf was filled successfully,
+which may not be possible if a referenced algorithm is unknown or not available.
+Many OpenSSL functions that use an X509 object call this function implicitly.
+
+=head1 RETURN VALUES
+
+This function returns 0 if the extensions or other portions of the certificate
+are invalid or an error occurred.
+Otherwise it returns 1.
+
+=head1 COPYRIGHT
+
+Copyright 2020 The OpenSSL Project Authors. All Rights Reserved.
+
+Licensed under the Apache License 2.0 (the "License").  You may not use
+this file except in compliance with the License.  You can obtain a copy
+in the file LICENSE in the source distribution or at
+L<https://www.openssl.org/source/license.html>.
+
+=cut
diff --git a/doc/man1/openssl.pod b/doc/man1/openssl.pod
index 1f344217a2..3ae273b5bf 100644
--- a/doc/man1/openssl.pod
+++ b/doc/man1/openssl.pod
@@ -906,6 +906,7 @@ a verification time, the check is not suppressed.
 =item B<-x509_strict>
 
 This disables non-compliant workarounds for broken certificates.
+Thus errors are thrown on certificates not compliant with RFC 5280.
 
 =item B<-ignore_critical>