add docs for job events

* Focus on the ordering of Job Event creation. Important to know when
thinking through different Ansible execution strategies.

1 files changed, 100 insertions, 0 deletions

diff --git a/docs/job_events.md b/docs/job_events.md
new file mode 100644
index 0000000000..60f5ad7afd
--- /dev/null
+++ b/docs/job_events.md
@@ -0,0 +1,100 @@
+## Ansible Callback and Job Events
+There is no concept of a job event in Ansible. Job Events are json structures, created when Ansible calls the Tower callback plugin hooks (i.e. `v2_playbook_on_task_start`, `v2_runner_on_ok`, etc.). The Job Event data structures contain data from the parameters of the callback hooks plus unique ids that reference other Job Events. There is usually a 1-1 relationship between a Job Event and an Ansible callback plugin function call.
+
+## Job Event Relationships
+The Job Event relationship is strictly hierarchical. In the example details below, each Job Event bullet is related to the previous Job Event to form a hierarchy.
+
+* There is always 1 and only 1 `v2_playbook_on_start` event and it is the first event.
+* `v2_playbook_on_play_start` is generated once per-play in the playbook. 2 such events would be generated from the playbook example below.
+* The `v2_playbook_on_task_start` function is called once for each task under the default execution strategy. Other execution strategies (i.e. free or serial) can result in the `v2_playbook_on_task_start` function being multiple times, one for each host. Tower only creates a Job Event for the **first** `v2_playbook_on_task_start` call. Subsequent calls for the same task do **not** result in Job Events being created.
+* `v2_runner_on_[ok, failed, skipped, unreachable, retry, item_on_ok, item_on_failed, item_on_skipped]` One `v2_runner_on_...` Job Event will be created for each `v2_playbook_on_task_start` event.
+
+## Example
+Below is an example inventory and playbook outline along with an example of Job Events generated and their hierarchical relationship.
+```
+# inventory
+[tower]
+hostA
+hostB
+
+[postgres]
+hostC
+```
+
+```
+# main.yml
+---
+- hosts: all
+  name: preflight
+  tasks:
+    - name: check_space_requirements
+      ...
+    - name: check_ram
+      ...
+    - name: check_umask
+      ...
+
+- hosts: all
+  name: install
+  tasks:
+    - name: install_tower
+      ...
+      when: inventory_hostname in ['A', 'B']
+    - name: install_postgres
+      ...
+      when: inventory_hostname == 'C'
+```
+
+Below is a visualization of how Job Events are related to form a hierarchy given a run of the playbook above.
+```
+`-- playbook_on_start
+    |-- playbook_on_play_start-preflight
+    |   |-- playbook_on_task_start-check_space_requirements
+    |   |   |-- runner_on_ok_hostA
+    |   |   |-- runner_on_ok_hostB
+    |   |   `-- runner_on_ok_hostC
+    |   |-- playbook_on_task_start-check_ram
+    |   |   |-- runner_on_ok_hostA
+    |   |   |-- runner_on_ok_hostB
+    |   |   `-- runner_on_ok_hostC
+    |   `-- playbook_on_task_start-check_umask
+    |       |-- runner_on_ok_hostA
+    |       |-- runner_on_ok_hostB
+    |       `-- runner_on_ok_hostC
+    `-- playbook_on_play_start-install
+        |-- playbook_on_task_start-install_tower
+        |   |-- runner_on_ok_hostA
+        |   `-- runner_on_ok_hostB
+        `-- playbook_on_task_start-install_postgres
+            `-- runner_on_ok_hostC
+```
+
+## Job Event Creation Patterns
+Ansible execution strategy heavily influences the creation order of Job Events. The above examples of Job Events creation an hierarchy are also the order in which they are created when the Ansible default execution strategy is used. When other strategies like free and serial are used, the order in which Job Events are created is slightly different. Let's take the previous example playbook and Job Events and show the order in which the Job Events may be created when the free strategy is used. Notice how `runner_on_*` Job Events can be created **after** a `playbook_on_task_start` for the next task runs. This is not the case for the default Ansible execution strategy. Under the default Ansible execution strategy, all `runner_on_*` Job Events will be created before the next `playbook_on_task_start` is generated. 
+
+```
+playbook_on_start
+playbook_on_play_start-preflight
+  playbook_on_task_start-check_space_requirements
+    runner_on_ok_hostA (check_space_requirements)
+  playbook_on_task_start-check_ram
+    runner_on_ok_hostA (check_ram)
+    runner_on_ok_hostC (check_space_requirements)
+    runner_on_ok_hostC (check_ram)
+  playbook_on_task_start-check_umask
+    runner_on_ok_hostB (check_ram)
+    runner_on_ok_hostC (check_umask)
+    runner_on_ok_hostA (check_umask)
+    runner_on_ok_hostB (check_space_requirements)
+    runner_on_ok_hostB (check_umask)
+playbook_on_play_start-install
+  playbook_on_task_start-install_tower
+    runner_on_ok_hostB (install_tower)
+  playbook_on_task_start-install_postgres
+    runner_on_ok_hostC (install_postgres)
+    runner_on_ok_hostA (install_tower)
+```
+
+## Code References
+* More comprehensive list of Job Events and the hierarchy they form https://github.com/ansible/awx/blob/devel/awx/main/models/jobs.py#L870
+* Exhaustive list of Job Events in Tower https://github.com/ansible/awx/blob/devel/awx/main/models/jobs.py#L900