Fix SSL_CTX_set1_groups documentation on preference orders

The documentation currently describes SSL_CTX_set1_groups as a
preference order, but this does not match the typical interpretation of
"preference order" in OpenSSL and TLS. Typically, an application can
order more secure options ahead of less secure ones and pick up TLS's
usual downgrade protection guarantees.

TLS 1.3 servers need to balance an additional consideration: some
options will perform worse than others due to key share prediction. The
prototypical selection procedure is to first select the set of more
secure options, then select the most performant among those.

OpenSSL follows this procedure, but it *unconditionally* treats all
configured curves as equivalent security. Per discussion on GitHub,
OpenSSL's position is that this is an intended behavior.

While not supported by built-in providers, OpenSSL now documents that
external providers can extend the group list and CHANGES.md explicitly
cites post-quantum as a use case. With post-quantum providers, it's
unlikely that application developers actually wanted options to be
equivalent security. To avoid security vulnerabilities arising from
mismatched expectations, update the documentation to clarify the server
behavior.

Per the OTC decision in
https://github.com/openssl/openssl/issues/22203#issuecomment-1744465829,
this documentation fix should be backported to stable branches.

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23776)

1 files changed, 41 insertions, 13 deletions

diff --git a/doc/man3/SSL_CTX_set1_curves.pod b/doc/man3/SSL_CTX_set1_curves.pod
index 8e521f755e..1ebabfb481 100644
--- a/doc/man3/SSL_CTX_set1_curves.pod
+++ b/doc/man3/SSL_CTX_set1_curves.pod
@@ -43,22 +43,46 @@ When setting such groups applications should use the "list" form of these
 functions (i.e. SSL_CTX_set1_groups_list() and SSL_set1_groups_list).
 
 SSL_CTX_set1_groups() sets the supported groups for B<ctx> to B<glistlen>
-groups in the array B<glist>. The array consist of all NIDs of groups in
-preference order. For a TLS client the groups are used directly in the
-supported groups extension. For a TLS server the groups are used to
-determine the set of shared groups. Currently supported groups for
-B<TLSv1.3> are B<NID_X9_62_prime256v1>, B<NID_secp384r1>, B<NID_secp521r1>,
-B<NID_X25519>, B<NID_X448>, B<NID_brainpoolP256r1tls13>,
-B<NID_brainpoolP384r1tls13>, B<NID_brainpoolP512r1tls13>, B<NID_ffdhe2048>,
-B<NID_ffdhe3072>, B<NID_ffdhe4096>, B<NID_ffdhe6144> and B<NID_ffdhe8192>.
+groups in the array B<glist>. The array consist of all NIDs of supported groups.
+Currently supported groups for B<TLSv1.3> are B<NID_X9_62_prime256v1>,
+B<NID_secp384r1>, B<NID_secp521r1>, B<NID_X25519>, B<NID_X448>,
+B<NID_brainpoolP256r1tls13>, B<NID_brainpoolP384r1tls13>,
+B<NID_brainpoolP512r1tls13>, B<NID_ffdhe2048>, B<NID_ffdhe3072>,
+B<NID_ffdhe4096>, B<NID_ffdhe6144> and B<NID_ffdhe8192>.
+OpenSSL will use this array in different ways depending on TLS role and version:
+
+=over 4
+
+=item For a TLS client, the groups are used directly in the supported groups
+extension. The extension's preference order, to be evaluated by the server, is
+determined by the order of the elements in the array.
+
+=item For a TLS 1.2 server, the groups determine the selected group. If
+B<SSL_OP_CIPHER_SERVER_PREFERENCE> is set, the order of the elements in the
+array determines the selected group. Otherwise, the order is ignored and the
+client's order determines the selection.
+
+=item For a TLS 1.3 server, the groups determine the selected group, but
+selection is more complex. A TLS 1.3 client sends both a group list as well as a
+predicted subset of groups. Choosing a group outside the predicted subset incurs
+an extra roundtrip. However, in some situations, the most preferred group may
+not be predicted. OpenSSL considers all supported groups to be comparable in
+security and prioritizes avoiding roundtrips above either client or server
+preference order. If an application uses an external provider to extend OpenSSL
+with, e.g., a post-quantum algorithm, this behavior may allow a network attacker
+to downgrade connections to a weaker algorithm.
+
+=back
 
 SSL_CTX_set1_groups_list() sets the supported groups for B<ctx> to
 string B<list>. The string is a colon separated list of group names, for example
-"P-521:P-384:P-256:X25519:ffdhe2048". Currently supported groups for B<TLSv1.3>
-are B<P-256>, B<P-384>, B<P-521>, B<X25519>, B<X448>, B<brainpoolP256r1tls13>,
-B<brainpoolP384r1tls13>, B<brainpoolP512r1tls13>, B<ffdhe2048>, B<ffdhe3072>,
-B<ffdhe4096>, B<ffdhe6144> and B<ffdhe8192>. Support for other groups may be
-added by external providers. If a group name is preceded with the C<?>
+"P-521:P-384:P-256:X25519:ffdhe2048". The groups are used as in
+SSL_CTX_set1_groups(), described above. Currently supported groups for
+B<TLSv1.3> are B<P-256>, B<P-384>, B<P-521>, B<X25519>, B<X448>,
+B<brainpoolP256r1tls13>, B<brainpoolP384r1tls13>, B<brainpoolP512r1tls13>,
+B<ffdhe2048>, B<ffdhe3072>, B<ffdhe4096>, B<ffdhe6144> and B<ffdhe8192>. Support
+for other groups may be added by external providers, however note the discussion
+on TLS 1.3 selection criteria above. If a group name is preceded with the C<?>
 character, it will be ignored if an implementation is missing.
 
 SSL_set1_groups() and SSL_set1_groups_list() are similar except they set
@@ -146,6 +170,10 @@ was added in OpenSSL 3.0.0.
 Support for ignoring unknown groups in SSL_CTX_set1_groups_list() and
 SSL_set1_groups_list() was added in OpenSSL 3.3.
 
+Earlier versions of this document described the list as a preference order.
+However, OpenSSL's behavior as a TLS 1.3 server is to consider I<all>
+supported groups as comparable in security.
+
 =head1 COPYRIGHT
 
 Copyright 2013-2024 The OpenSSL Project Authors. All Rights Reserved.