Add support for compressed certificates (RFC8879)

* Compressed Certificate extension (server/client)
* Server certificates (send/receive)
* Client certificate (send/receive)

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18186)

7 files changed, 247 insertions, 7 deletions

diff --git a/doc/build.info b/doc/build.info
index 8884b0dd5f..71c7f3aa40 100644
--- a/doc/build.info
+++ b/doc/build.info
@@ -2167,6 +2167,10 @@ DEPEND[html/man3/SSL_CTX_set0_CA_list.html]=man3/SSL_CTX_set0_CA_list.pod
 GENERATE[html/man3/SSL_CTX_set0_CA_list.html]=man3/SSL_CTX_set0_CA_list.pod
 DEPEND[man/man3/SSL_CTX_set0_CA_list.3]=man3/SSL_CTX_set0_CA_list.pod
 GENERATE[man/man3/SSL_CTX_set0_CA_list.3]=man3/SSL_CTX_set0_CA_list.pod
+DEPEND[html/man3/SSL_CTX_set1_cert_comp_preference.html]=man3/SSL_CTX_set1_cert_comp_preference.pod
+GENERATE[html/man3/SSL_CTX_set1_cert_comp_preference.html]=man3/SSL_CTX_set1_cert_comp_preference.pod
+DEPEND[man/man3/SSL_CTX_set1_cert_comp_preference.3]=man3/SSL_CTX_set1_cert_comp_preference.pod
+GENERATE[man/man3/SSL_CTX_set1_cert_comp_preference.3]=man3/SSL_CTX_set1_cert_comp_preference.pod
 DEPEND[html/man3/SSL_CTX_set1_curves.html]=man3/SSL_CTX_set1_curves.pod
 GENERATE[html/man3/SSL_CTX_set1_curves.html]=man3/SSL_CTX_set1_curves.pod
 DEPEND[man/man3/SSL_CTX_set1_curves.3]=man3/SSL_CTX_set1_curves.pod
@@ -3313,6 +3317,7 @@ html/man3/SSL_CTX_sess_set_cache_size.html \
 html/man3/SSL_CTX_sess_set_get_cb.html \
 html/man3/SSL_CTX_sessions.html \
 html/man3/SSL_CTX_set0_CA_list.html \
+html/man3/SSL_CTX_set1_cert_comp_preference.html \
 html/man3/SSL_CTX_set1_curves.html \
 html/man3/SSL_CTX_set1_sigalgs.html \
 html/man3/SSL_CTX_set1_verify_cert_store.html \
@@ -3918,6 +3923,7 @@ man/man3/SSL_CTX_sess_set_cache_size.3 \
 man/man3/SSL_CTX_sess_set_get_cb.3 \
 man/man3/SSL_CTX_sessions.3 \
 man/man3/SSL_CTX_set0_CA_list.3 \
+man/man3/SSL_CTX_set1_cert_comp_preference.3 \
 man/man3/SSL_CTX_set1_curves.3 \
 man/man3/SSL_CTX_set1_sigalgs.3 \
 man/man3/SSL_CTX_set1_verify_cert_store.3 \
diff --git a/doc/man1/openssl-s_client.pod.in b/doc/man1/openssl-s_client.pod.in
index 5f6f74c51d..86b9aff91e 100644
--- a/doc/man1/openssl-s_client.pod.in
+++ b/doc/man1/openssl-s_client.pod.in
@@ -83,6 +83,8 @@ B<openssl> B<s_client>
 [B<-read_buf>]
 [B<-ignore_unexpected_eof>]
 [B<-bugs>]
+[B<-no_tx_cert_comp>]
+[B<-no_rx_cert_comp>]
 [B<-comp>]
 [B<-no_comp>]
 [B<-brief>]
@@ -601,6 +603,14 @@ For more information on shutting down a connection, see L<SSL_shutdown(3)>.
 There are several known bugs in SSL and TLS implementations. Adding this
 option enables various workarounds.
 
+=item B<-no_tx_cert_comp>
+
+Disables support for sending TLSv1.3 compressed certificates.
+
+=item B<-no_rx_cert_comp>
+
+Disables support for receiving TLSv1.3 compressed certificate.
+
 =item B<-comp>
 
 Enables support for SSL/TLS compression.
@@ -930,7 +940,9 @@ The B<-certform> option has become obsolete in OpenSSL 3.0.0 and has no effect.
 
 The B<-engine> option was deprecated in OpenSSL 3.0.
 
-The -tfo option was added in OpenSSL 3.2.
+
+The B<-tfo>, B<-no_tx_cert_comp>, and B<-no_rx_cert_comp> options were added
+in OpenSSL 3.2.
 
 =head1 COPYRIGHT
 
diff --git a/doc/man1/openssl-s_server.pod.in b/doc/man1/openssl-s_server.pod.in
index 8fa041c2fe..94f3b4b46c 100644
--- a/doc/man1/openssl-s_server.pod.in
+++ b/doc/man1/openssl-s_server.pod.in
@@ -92,6 +92,8 @@ B<openssl> B<s_server>
 [B<-naccept> I<+int>]
 [B<-read_buf> I<+int>]
 [B<-bugs>]
+[B<-no_tx_cert_comp>]
+[B<-no_rx_cert_comp>]
 [B<-no_comp>]
 [B<-comp>]
 [B<-no_ticket>]
@@ -139,6 +141,7 @@ B<openssl> B<s_server>
 [B<-no_anti_replay>]
 [B<-num_tickets>]
 [B<-tfo>]
+[B<-cert_comp>]
 {- $OpenSSL::safe::opt_name_synopsis -}
 {- $OpenSSL::safe::opt_version_synopsis -}
 {- $OpenSSL::safe::opt_v_synopsis -}
@@ -604,6 +607,14 @@ further information).
 There are several known bugs in SSL and TLS implementations. Adding this
 option enables various workarounds.
 
+=item B<-no_tx_cert_comp>
+
+Disables support for sending TLSv1.3 compressed certificates.
+
+=item B<-no_rx_cert_comp>
+
+Disables support for receiving TLSv1.3 compressed certificates.
+
 =item B<-no_comp>
 
 Disable negotiation of TLS compression.
@@ -820,6 +831,9 @@ data that was sent will be rejected.
 
 Enable acceptance of TCP Fast Open (RFC7413) connections.
 
+=item B<-cert_comp>
+
+Pre-compresses certificates (RFC8879) that will be sent during the handshake.
 
 {- $OpenSSL::safe::opt_name_item -}
 
@@ -947,7 +961,8 @@ The
 The B<-srpvfile>, B<-srpuserseed>, and B<-engine>
 option were deprecated in OpenSSL 3.0.
 
-The -tfo option was added in OpenSSL 3.2.
+The B<-tfo>, B<-no_tx_cert_comp>, and B<-no_rx_cert_comp> options were added
+in OpenSSL 3.2.
 
 =head1 COPYRIGHT
 
diff --git a/doc/man3/COMP_CTX_new.pod b/doc/man3/COMP_CTX_new.pod
index 1fe1d691df..7e1c8c4a83 100644
--- a/doc/man3/COMP_CTX_new.pod
+++ b/doc/man3/COMP_CTX_new.pod
@@ -98,7 +98,7 @@ COMP_zstd_oneshot() returns a B<COMP_METHOD> for one-shot Zstandard compression.
 BIO_f_zlib(), BIO_f_brotli() BIO_f_zstd() each return a B<BIO_METHOD> that may be used to
 create a B<BIO> via B<BIO_new(3)> to read and write compressed files or streams.
 The functions are only available if the corresponding algorithm is compiled into
-the OpenSSL library.
+the OpenSSL library. NULL may be returned if the algorithm fails to load dynamically.
 
 =head1 NOTES
 
@@ -123,11 +123,12 @@ L<SSL_set_options(3)> functions.
 
 Compression is also used to support certificate compression as described
 in RFC8879 L<https://datatracker.ietf.org/doc/html/rfc8879>.
-It may be disabled via the SSL_OP_NO_CERTIFICATE_COMPRESSION option of
-the L<SSL_CTX_set_options(3)> or L<SSL_set_options(3)> functions.
+It may be disabled via the SSL_OP_NO_TX_CERTIFICATE_COMPRESSION and
+SSL_OP_NO_RX_CERTIFICATE_COMPRESSION options of the
+L<SSL_CTX_set_options(3)> or L<SSL_set_options(3)> functions.
 
 COMP_zlib(), COMP_brotli() and COMP_zstd() are stream-based compression methods.
-Internal state (including compression dictionary) is maintained between calls. 
+Internal state (including compression dictionary) is maintained between calls.
 If an error is returned, the stream is corrupted, and should be closed.
 
 COMP_brotli_oneshot() and COMP_zstd_oneshot() are not stream-based. These
@@ -152,7 +153,8 @@ bytes stored in the output buffer I<out>. This may be 0. On failure,
 COMP_get_name() returns a B<const char *> that must not be freed
 on success, or NULL on failure.
 
-BIO_f_zlib(), BIO_f_brotli() and BIO_f_zstd() return a B<BIO_METHOD>.
+BIO_f_zlib(), BIO_f_brotli() and BIO_f_zstd() return NULL on error, and
+a B<BIO_METHOD> on success.
 
 =head1 SEE ALSO
 
diff --git a/doc/man3/SSL_CONF_cmd.pod b/doc/man3/SSL_CONF_cmd.pod
index 400bd223c6..c20df37e3b 100644
--- a/doc/man3/SSL_CONF_cmd.pod
+++ b/doc/man3/SSL_CONF_cmd.pod
@@ -162,6 +162,24 @@ This is a synonym for the B<-groups> command.
 This sets the temporary curve used for ephemeral ECDH modes. Only used
 by servers.
 
+=item B<-tx_cert_comp>
+
+Enables support for sending TLSv1.3 compressed certificates.
+
+=item B<-no_tx_cert_comp>
+
+Disables support for sending TLSv1.3 compressed certificates.
+
+=item B<-rx_cert_comp>
+
+Enables support for receiving TLSv1.3 compressed certificates.
+
+=item B<-no_rx_cert_comp>
+
+Disables support for receiving TLSv1.3 compressed certificates.
+
+=item B<-comp>
+
 The B<groups> argument is a curve name or the special value B<auto> which
 picks an appropriate curve based on client and server preferences. The
 curve can be either the B<NIST> name (e.g. B<P-256>) or an OpenSSL OID name
@@ -535,6 +553,14 @@ B<SSL_OP_ENABLE_KTLS>.
 B<StrictCertCheck>: Enable strict certificate checking. Equivalent to
 setting B<SSL_CERT_FLAG_TLS_STRICT> with SSL_CTX_set_cert_flags().
 
+B<TxCertificateCompression>: support sending compressed certificates, enabled by
+default. Inverse of B<SSL_OP_NO_TX_CERTIFICATE_COMPRESSION>: that is,
+B<-TxCertificateCompression> is the same as setting B<SSL_OP_NO_TX_CERTIFICATE_COMPRESSION>.
+
+B<RxCertificateCompression>: support receiving compressed certificates, enabled by
+default. Inverse of B<SSL_OP_NO_RX_CERTIFICATE_COMPRESSION>: that is,
+B<-RxCertificateCompression> is the same as setting B<SSL_OP_NO_RX_CERTIFICATE_COMPRESSION>.
+
 =item B<VerifyMode>
 
 The B<value> argument is a comma separated list of flags to set.
@@ -736,6 +762,9 @@ B<AllowNoDHEKEX> and B<PrioritizeChaCha> were added in OpenSSL 1.1.1.
 The B<UnsafeLegacyServerConnect> option is no longer set by default from
 OpenSSL 3.0.
 
+The B<TxCertificateCompression> and B<RxCertificateCompression> options were
+added in OpenSSL 3.2.
+
 =head1 COPYRIGHT
 
 Copyright 2012-2022 The OpenSSL Project Authors. All Rights Reserved.
diff --git a/doc/man3/SSL_CTX_set1_cert_comp_preference.pod b/doc/man3/SSL_CTX_set1_cert_comp_preference.pod
new file mode 100644
index 0000000000..2fe35873f7
--- /dev/null
+++ b/doc/man3/SSL_CTX_set1_cert_comp_preference.pod
@@ -0,0 +1,160 @@
+=pod
+
+=head1 NAME
+
+SSL_CTX_set1_cert_comp_preference,
+SSL_set1_cert_comp_preference,
+SSL_CTX_compress_certs,
+SSL_compress_certs,
+SSL_CTX_get1_compressed_cert,
+SSL_get1_compressed_cert,
+SSL_CTX_set1_compressed_cert,
+SSL_set1_compressed_cert - Certificate compression functions
+
+=head1 SYNOPSIS
+
+ #include <openssl/ssl.h>
+
+ int SSL_CTX_set1_cert_comp_preference(SSL_CTX *ctx, int *algs, size_t len);
+ int SSL_set1_cert_comp_preference(SSL *ssl, int *algs, size_t len);
+
+ int SSL_CTX_compress_certs(SSL_CTX *ctx, int alg);
+ int SSL_compress_certs(SSL *ssl, int alg);
+
+ size_t SSL_CTX_get1_compressed_cert(SSL_CTX *ctx, int alg, unsigned char **data,
+                                     size_t *orig_len);
+ size_t SSL_get1_compressed_cert(SSL *ssl, int alg, unsigned char **data,
+                                 size_t *orig_len);
+
+ int SSL_CTX_set1_compressed_cert(SSL_CTX *ctx, int alg,
+                                  unsigned char *comp_data,
+                                  size_t comp_length, size_t orig_length);
+ int SSL_set1_compressed_cert(SSL *ssl, int alg, unsigned char *comp_data,
+                              size_t comp_length, size_t orig_length);
+
+
+=head1 DESCRIPTION
+
+These functions control the certificate compression feature. Certificate
+compression is only available for TLSv1.3 as defined in RFC8879.
+
+SSL_CTX_set1_cert_comp_preference() and SSL_set1_cert_comp_preference() are used
+to specify the preferred compression algorithms. The B<algs> argument is an array
+of algorithms, and B<length> is number of elements in the B<algs> array. Only
+those algorithms enabled in the library will be accepted in B<algs>, unknown
+algorithms in B<algs> are ignored. On an error, the preference order is left
+unmodified.
+
+The following compression algorithms (B<alg> arguments) may be used:
+
+=over 4
+
+=item * TLSEXT_comp_cert_brotli
+
+=item * TLSEXT_comp_cert_zlib
+
+=item * TLSEXT_comp_cert_zstd
+
+=back
+
+The above is also the default preference order. If a preference order is not
+specified, then the default preference order is sent to the peer and the
+received peer's preference order will be used when compressing a certificate.
+Otherwise, the configured preference order is sent to the peer and is used
+to filter the peer's preference order.
+
+SSL_CTX_compress_certs() and SSL_compress_certs() are used to pre-compress all
+the configured certificates on an SSL_CTX/SSL object with algorithm B<alg>. If
+B<alg> is 0, then the certificates are compressed with the algorithms specified
+in the preference list. Calling these functions on a client SSL_CTX/SSL object
+will result in an error, as only server certificates may be pre-compressed.
+
+SSL_CTX_get1_compressed_cert() and SSL_get1_compressed_cert() are used to get
+the pre-compressed certificate most recently set that may be stored for later
+use. Calling these functions on a client SSL_CTX/SSL object will result in an
+error, as only server certificates may be pre-compressed. The B<data> and
+B<orig_len> arguments are required.
+
+The compressed certificate data may be passed to SSL_CTX_set1_compressed_cert()
+or SSL_set1_compressed_cert() to provide a pre-compressed version of the
+most recently set certificate. This pre-compressed certificate can only be used
+by a server.
+
+=head1 NOTES
+
+Each side of the connection sends their compression algorithm preference list
+to their peer indicating compressed certificate support. The received preference
+list is filtered by the configured preference list (i.e. the intersection is
+saved). As the default list includes all the enabled algorithms, not specifying
+a preference will allow any enabled algorithm by the peer. The filtered peer's
+preference order is used to determine what algorithm to use when sending a
+compressed certificate.
+
+Only server certificates may be pre-compressed. Calling any of these functions
+(except SSL_CTX_set1_cert_comp_preference()/SSL_set1_cert_comp_preference())
+on a client SSL_CTX/SSL object will return an error. Client certificates are
+compressed on-demand as unique context data from the server is compressed along
+with the certificate.
+
+For SSL_CTX_set1_cert_comp_preference() and SSL_set1_cert_comp_preference()
+the B<len> argument is the size of the B<algs> argument in bytes.
+
+The compressed certificate returned by SSL_CTX_get1_compressed_cert() and
+SSL_get1_compressed_cert() is the last certificate set on the SSL_CTX/SSL object.
+The certificate is copied by the function and the caller must free B<*data> via
+OPENSSL_free().
+
+The compressed certificate data set by SSL_CTX_set1_compressed_cert() and
+SSL_set1_compressed_cert() is copied into the SSL_CTX/SSL object.
+
+SSL_CTX_compress_certs() and SSL_compress_certs() return an error under the
+following conditions:
+
+=over 4
+
+=item * If no certificates have been configured.
+
+=item * If the specified algorithm B<alg> is not enabled.
+
+=item * If B<alg> is 0 and no compression algorithms are enabled.
+
+=back
+
+Sending compressed certificates may be disabled on a connection via the
+SSL_OP_NO_TX_CERTIFICATE_COMPRESSION option. Receiving compressed certificates
+may be disabled on a connection via the SSL_OP_NO_RX_CERTIFICATE_COMPRESSION
+option.
+
+=head1 RETURN VALUES
+
+SSL_CTX_set1_cert_comp_preference(),
+SSL_set1_cert_comp_preference(),
+SSL_CTX_compress_certs(),
+SSL_compress_certs(),
+SSL_CTX_set1_compressed_cert(), and
+SSL_set1_compressed_cert()
+return 1 for success and 0 on error.
+
+SSL_CTX_get1_compressed_cert() and
+SSL_get1_compressed_cert()
+return the length of the allocated memory on success and 0 on error.
+
+=head1 SEE ALSO
+
+L<SSL_CTX_set_options(3)>,
+L<SSL_CTX_use_certificate(3)>
+
+=head1 HISTORY
+
+These functions were added in OpenSSL 3.2.
+
+=head1 COPYRIGHT
+
+Copyright 2022 The OpenSSL Project Authors. All Rights Reserved.
+
+Licensed under the Apache License 2.0 (the "License").  You may not use
+this file except in compliance with the License.  You can obtain a copy
+in the file LICENSE in the source distribution or at
+L<https://www.openssl.org/source/license.html>.
+
+=cut
diff --git a/doc/man3/SSL_CTX_set_options.pod b/doc/man3/SSL_CTX_set_options.pod
index 08522522cd..1da057adb8 100644
--- a/doc/man3/SSL_CTX_set_options.pod
+++ b/doc/man3/SSL_CTX_set_options.pod
@@ -214,6 +214,22 @@ functionality is not required. Those applications can turn this feature off by
 setting this option. This is a server-side opton only. It is ignored by
 clients.
 
+=item SSL_OP_NO_TX_CERTIFICATE_COMPRESSION
+
+Normally clients and servers will transparently attempt to negotiate the
+RFC8879 certificate compression option on TLSv1.3 connections.
+
+If this option is set, the certificate compression extension is ignored
+upon receipt and compressed certificates will not be sent to the peer.
+
+=item SSL_OP_NO_RX_CERTIFICATE_COMPRESSION
+
+Normally clients and servers will transparently attempt to negotiate the
+RFC8879 certificate compression option on TLSv1.3 connections.
+
+If this option is set, the certificate compression extension will not be sent
+and compressed certificates will not be accepted from the peer.
+
 =item SSL_OP_NO_COMPRESSION
 
 Do not use compression even if it is supported. This option is set by default.