optimize host_group_vars and vars plugin loading (#79945)

* Improve host_group_vars efficiency:

* normalize the basedir with `os.path.realpath()` once and cache it
* cache missing paths/files
* reduce the calls to `isinstance`

Add a couple more general improvements in vars/plugins.py get_vars_from_path():

* call `PluginLoader.all()` once for vars plugins and reload specific
  plugins subsequently
* don't reload legacy/builtin vars plugins that are not enabled

Add a test for host_group_vars and legacy plugin loading

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

* Add a new is_stateless attribute to the vars plugin baseclass

update integration tests to be quieter and use the same test pattern

Fix deprecation and adjust test that didn't catch the issue (deprecation only occured when the value was False)

move realpath cache to host_group_vars (do not smuggle call state as instance data)

refactor under a single 'if cache:' statement

Call os.path.isdir instead of always calling os.path.exists first. Just call os.path.exists to differentiate between missing and non-directory.

remove call to super(VarsModule, self).get_vars()

use the entity name as the cache key instead of variable location

Remove isinstance checks and use a class attribute just in case any plugins are subclassing Host/Group

Replace startswith by checking index 0 of the name instead, since host/group names are required

* rename is_stateless to cache_instance to make it more clear what it does

* add plugin instance cache using the path to plugin loader

reduce loading stage option if a new instance isn't created

don't require a known subdir on PluginLoader instantiation for backwards
compatibility

rename attribute again

contain reading from/initializing cached instances to a plugin loader method

* Deprecate v2 vars plugins

* Refactor to use the cache in existing plugin loader methods

Rename the attribute again

Refactor host_group_vars with requested changes

Make changelog a bugfixes fragment

Add a deprecation fragment for v2 vars plugins.

Add type hints

* unbreak group_vars

* Apply suggestions from code review

* misc tweaks

* always cache instance by both requested and resolved FQ name
* add lru_cache to stage calculation to avoid repeated config consultation

* handle KeyError from missing stage option

---------

Co-authored-by: Matt Davis <mrd@redhat.com>

3 files changed, 39 insertions, 5 deletions

diff --git a/test/integration/targets/old_style_vars_plugins/deprecation_warning/v2_vars_plugin.py b/test/integration/targets/old_style_vars_plugins/deprecation_warning/v2_vars_plugin.py
new file mode 100644
index 0000000000..f342b698a0
--- /dev/null
+++ b/test/integration/targets/old_style_vars_plugins/deprecation_warning/v2_vars_plugin.py
@@ -0,0 +1,6 @@
+class VarsModule:
+    def get_host_vars(self, entity):
+        return {}
+
+    def get_group_vars(self, entity):
+        return {}
diff --git a/test/integration/targets/old_style_vars_plugins/deprecation_warning/vars.py b/test/integration/targets/old_style_vars_plugins/deprecation_warning/vars.py
index d5c9a422dc..f554be04fb 100644
--- a/test/integration/targets/old_style_vars_plugins/deprecation_warning/vars.py
+++ b/test/integration/targets/old_style_vars_plugins/deprecation_warning/vars.py
@@ -2,7 +2,7 @@ from ansible.plugins.vars import BaseVarsPlugin
 
 
 class VarsModule(BaseVarsPlugin):
-    REQUIRES_WHITELIST = False
+    REQUIRES_WHITELIST = True
 
     def get_vars(self, loader, path, entities):
         return {}
diff --git a/test/integration/targets/old_style_vars_plugins/runme.sh b/test/integration/targets/old_style_vars_plugins/runme.sh
index 4cd1916819..71275b8ac4 100755
--- a/test/integration/targets/old_style_vars_plugins/runme.sh
+++ b/test/integration/targets/old_style_vars_plugins/runme.sh
@@ -12,9 +12,37 @@ export ANSIBLE_VARS_PLUGINS=./vars_plugins
 export ANSIBLE_VARS_ENABLED=require_enabled
 [ "$(ansible-inventory -i localhost, --list --yaml all "$@" | grep -c 'require_enabled')" = "1" ]
 
-# Test the deprecated class attribute
+# Test deprecated features
 export ANSIBLE_VARS_PLUGINS=./deprecation_warning
-WARNING="The VarsModule class variable 'REQUIRES_WHITELIST' is deprecated. Use 'REQUIRES_ENABLED' instead."
+WARNING_1="The VarsModule class variable 'REQUIRES_WHITELIST' is deprecated. Use 'REQUIRES_ENABLED' instead."
+WARNING_2="The vars plugin v2_vars_plugin .* is relying on the deprecated entrypoints 'get_host_vars' and 'get_group_vars'"
 ANSIBLE_DEPRECATION_WARNINGS=True ANSIBLE_NOCOLOR=True ANSIBLE_FORCE_COLOR=False \
-	ansible-inventory -i localhost, --list all 2> err.txt
-ansible localhost -m debug -a "msg={{ lookup('file', 'err.txt') | regex_replace('\n', '') }}" | grep "$WARNING"
+        ansible-inventory -i localhost, --list all "$@" 2> err.txt
+for WARNING in "$WARNING_1" "$WARNING_2"; do
+	ansible localhost -m debug -a "msg={{ lookup('file', 'err.txt') | regex_replace('\n', '') }}" | grep "$WARNING"
+done
+
+# Test how many times vars plugins are loaded for a simple play containing a task
+# host_group_vars is stateless, so we can load it once and reuse it, every other vars plugin should be instantiated before it runs
+cat << EOF > "test_task_vars.yml"
+---
+- hosts: localhost
+  connection: local
+  gather_facts: no
+  tasks:
+  - debug:
+EOF
+
+# hide the debug noise by dumping to a file
+trap 'rm -rf -- "out.txt"' EXIT
+
+ANSIBLE_DEBUG=True ansible-playbook test_task_vars.yml > out.txt
+[ "$(grep -c "Loading VarsModule 'host_group_vars'" out.txt)" -eq 1 ]
+[ "$(grep -c "Loading VarsModule 'require_enabled'" out.txt)" -gt 50 ]
+[ "$(grep -c "Loading VarsModule 'auto_enabled'" out.txt)" -gt 50 ]
+
+export ANSIBLE_VARS_ENABLED=ansible.builtin.host_group_vars
+ANSIBLE_DEBUG=True ansible-playbook test_task_vars.yml > out.txt
+[ "$(grep -c "Loading VarsModule 'host_group_vars'" out.txt)" -eq 1 ]
+[ "$(grep -c "Loading VarsModule 'require_enabled'" out.txt)" -lt 3 ]
+[ "$(grep -c "Loading VarsModule 'auto_enabled'" out.txt)" -gt 50 ]