General upgrade of dependencies

Update licenses for new versions after dependency upgrades

pin pycurl to version that does not break on install

implement new workflow for py2/3 requirements management

require twisted tls extras, resolve service-identity version

Upgrade celery to resolve importlib DeprecationWarning

use flags to resolve the unsafe and cache problems

5 files changed, 122 insertions, 95 deletions

diff --git a/requirements/README.md b/requirements/README.md
index 0d3c241b30..f9d31c93eb 100644
--- a/requirements/README.md
+++ b/requirements/README.md
@@ -1,23 +1,46 @@
-The requirements.txt and requirements_ansible.txt files are generated from requirements.in and requirements_ansible.in, respectively, using `pip-tools` `pip-compile`. The following commands should do this if ran inside the tower_tools container.
+The requirements.txt and requirements_ansible.txt files are generated from requirements.in and requirements_ansible.in, respectively, using `pip-tools` `pip-compile`. The following commands should do this if ran inside the tools_awx container.
 
-NOTE: before running `pip-compile`, please copy-paste contents in `requirements/requirements_git.txt` to the top of `requirements/requirements.in` and prepend each copied line with `-e `. Later after `requirements.txt` is generated, don't forget to remove all `git+https://github.com...`-like lines from both `requirements.txt` and `requirements.in` (repeat for `requirements_ansible_git.txt` and `requirements_ansible.in`)
-
-At the end of `requirements/requirements.in`, pip and setuptools need to have their versions pinned.
+Run these commands from the root of the awx repo. This will produce python 3 requirements files.
 
 ```
-python3 -m venvpip install pip-tools /buildit
+python3 -m venv /buildit
 source /buildit/bin/activate
 pip install pip-tools
 pip install pip --upgrade
 
-pip-compile requirements/requirements.in > requirements/requirements.txt
-pip-compile requirements/requirements_ansible.in > requirements/requirements_ansible.txt
+pip-compile -U -r --allow-unsafe --output-file requirements/requirements.txt requirements/requirements.in
+pip-compile -U -r --allow-unsafe --output-file requirements/requirements_ansible_py3.txt requirements/requirements_ansible.in
 ```
 
-## Known Issues
+The Ansible venv requirements file needs to start with the python 2 version
+as a base. Then we can run the tool again to get the python 3 version.
+Consult the output of the `diff` command and add a conditional switch in those cases.
+
+```
+virtualenv -p python2 /buildit_py2
+source /buildit_py2/bin/activate
+pip install pip-tools
+pip install pip --upgrade
+
+pip-compile -U -r --allow-unsafe --output-file requirements/requirements_ansible.txt requirements/requirements_ansible.in
+diff requirements/requirements_ansible_py3.txt requirements/requirements_ansible.txt
+rm requirements/requirements_ansible_py3.txt
+```
 
-* Remove the `-e` from packages of the form `-e git+https://github.com...` in the generated `.txt`. Failure to do so will result in a "bad" RPM and DEB due to the `pip install` laying down a symbolic link with an absolute path from the virtualenv to the git repository that will differ from when the RPM and DEB are build to when the RPM and DEB are installed on a machine. By removing the `-e` the symbolic egg link will not be created and all is well.
+Python 3 exceptions should be added to relevant `requirements_ansible.txt` lines
+after version numbers with the syntax of `; python_version < '3'`.
+
+## Licenses and Source Files
+
+If any library has a change to its license with the upgrade, then the license for that library
+inside of `docs/licenses` needs to be updated.
+
+For libraries that have source distribution requirements (LGPL as an example),
+a tarball of the library is kept along with the license.
+To download the PyPI tarball, you can run this command:
+
+```
+pip download <pypi library name> -d docs/licenses/ --no-binary :all: --no-deps
+```
 
-* As of `pip-tools` `1.8.1` `pip-compile` does not resolve packages specified using a git url. Thus, dependencies for things like `dm.xmlsec.binding` do not get resolved and output to `requirements.txt`. This means that:
-  * can't use `pip install --no-deps` because other deps WILL be sucked in
-  * all dependencies are NOT captured in our `.txt` files. This means you can't rely on the `.txt` when gathering licenses.
+Make sure to delete the old tarball if it is an upgrade.
diff --git a/requirements/requirements.in b/requirements/requirements.in
index 0667c8dcc2..05853a486f 100644
--- a/requirements/requirements.in
+++ b/requirements/requirements.in
@@ -5,8 +5,9 @@ asgiref==1.1.2
 azure-keyvault==1.1.0
 boto==2.47.0
 channels==1.1.8
-celery==4.2.1
+celery==4.3.0
 daphne==1.3.0   # Last before backwards-incompatible channels 2 upgrade
+twisted[tls]>=17.1   # from daphne, see https://github.com/django/daphne/pull/257
 defusedxml==0.5.0  # py36 support https://github.com/tiran/defusedxml/pull/4
 Django==1.11.20
 django-auth-ldap==1.7.0
@@ -43,7 +44,6 @@ social-auth-core==3.0.0
 social-auth-app-django==2.1.0
 requests==2.21.0
 requests-futures==0.9.7
-service-identity==17.0.0
 slackclient==1.1.2
 tacacs_plus==1.0
 twilio==6.10.4
diff --git a/requirements/requirements.txt b/requirements/requirements.txt
index cf2bb212a1..10254944b3 100644
--- a/requirements/requirements.txt
+++ b/requirements/requirements.txt
@@ -2,31 +2,31 @@
 # This file is autogenerated by pip-compile
 # To update, run:
 #
-#    pip-compile requirements/requirements.in
+#    pip-compile -U -r --allow-unsafe --output-file requirements/requirements.txt requirements/requirements.in
 #
 adal==1.2.1               # via msrestazure
-amqp==2.3.2               # via kombu
+amqp==2.4.2               # via kombu
 ansible-runner==1.3.4
 appdirs==1.4.2
 argparse==1.4.0           # via uwsgitop
 asgi-amqp==1.1.3
 asgiref==1.1.2
 asn1crypto==0.24.0        # via cryptography
-attrs==18.2.0             # via automat, service-identity, twisted
-autobahn==19.2.1          # via daphne
+attrs==19.1.0             # via automat, service-identity, twisted
+autobahn==19.3.3          # via daphne
 automat==0.7.0            # via twisted
-azure-common==1.1.18      # via azure-keyvault
+azure-common==1.1.19      # via azure-keyvault
 azure-keyvault==1.1.0
 azure-nspkg==3.0.2        # via azure-keyvault
-billiard==3.5.0.5         # via celery
+billiard==3.6.0.0         # via celery
 boto==2.47.0
-celery==4.2.1
-certifi==2018.11.29       # via msrest, requests
-cffi==1.12.1              # via cryptography
+celery==4.3.0
+certifi==2019.3.9         # via msrest, requests
+cffi==1.12.3              # via cryptography
 channels==1.1.8
 chardet==3.0.4            # via requests
 constantly==15.1.0        # via twisted
-cryptography==2.6.1       # via adal, azure-keyvault, pyopenssl
+cryptography==2.6.1       # via adal, azure-keyvault, pyopenssl, service-identity
 daphne==1.3.0
 defusedxml==0.5.0
 django-auth-ldap==1.7.0
@@ -44,79 +44,81 @@ django-taggit==0.22.2
 django==1.11.20
 djangorestframework-yaml==1.0.3
 djangorestframework==3.7.7
+docutils==0.14            # via python-daemon
 future==0.16.0            # via django-radius
-hyperlink==18.0.0         # via twisted
-idna==2.8                 # via hyperlink, requests
+hyperlink==19.0.0         # via twisted
+idna==2.8                 # via hyperlink, requests, twisted
 incremental==17.5.0       # via twisted
 inflect==2.1.0            # via jaraco.itertools
 irc==16.2
 isodate==0.6.0            # via msrest, python3-saml
 jaraco.classes==2.0       # via jaraco.collections
-jaraco.collections==2.0   # via irc, jaraco.text
+jaraco.collections==2.0   # via irc
 jaraco.functools==2.0     # via irc, jaraco.text, tempora
-jaraco.itertools==4.4.1   # via irc
+jaraco.itertools==4.4.2   # via irc
 jaraco.logging==2.0       # via irc
 jaraco.stream==2.0        # via irc
-jaraco.text==2.0          # via irc, jaraco.collections
+jaraco.text==3.0          # via irc, jaraco.collections
 jinja2==2.10.1
 jsonpickle==1.1           # via asgi-amqp
 jsonschema==2.6.0
-kombu==4.2.1              # via asgi-amqp, celery
-lxml==4.3.1               # via xmlsec
+kombu==4.5.0              # via asgi-amqp, celery
+lockfile==0.12.2          # via python-daemon
+lxml==4.3.3               # via xmlsec
 markdown==2.6.11
-markupsafe==1.1.0         # via jinja2
-more-itertools==6.0.0     # via irc, jaraco.functools, jaraco.itertools
+markupsafe==1.1.1         # via jinja2
+more-itertools==7.0.0     # via irc, jaraco.functools, jaraco.itertools
 msgpack-python==0.5.6     # via asgi-amqp
-msrest==0.6.4             # via azure-keyvault, msrestazure
+msrest==0.6.6             # via azure-keyvault, msrestazure
 msrestazure==0.6.0        # via azure-keyvault
 netaddr==0.7.19           # via pyrad
-oauthlib==2.0.6           # via django-oauth-toolkit, requests-oauthlib, social-auth-core
+oauthlib==3.0.1           # via django-oauth-toolkit, requests-oauthlib, social-auth-core
 ordereddict==1.1
 pexpect==4.6.0
-pkgconfig==1.4.0          # via xmlsec
+pkgconfig==1.5.1          # via xmlsec
 prometheus_client==0.6.0
 psutil==5.4.3
 psycopg2==2.7.3.2
 ptyprocess==0.6.0         # via pexpect
-pyasn1-modules==0.2.4     # via python-ldap, service-identity
+pyasn1-modules==0.2.5     # via python-ldap, service-identity
 pyasn1==0.4.5             # via pyasn1-modules, python-ldap, service-identity
 pycparser==2.19           # via cffi
 pygerduty==0.37.0
 pyhamcrest==1.9.0         # via twisted
 pyjwt==1.7.1              # via adal, social-auth-core, twilio
-pyopenssl==19.0.0         # via service-identity
+pyopenssl==19.0.0         # via twisted
 pyparsing==2.2.0
 pyrad==2.1                # via django-radius
 pysocks==1.6.8            # via twilio
-python-daemon==2.2.0      # via ansible-runner
+python-daemon==2.2.0
 python-dateutil==2.7.2
-python-ldap==3.1.0        # via django-auth-ldap
+python-ldap==3.2.0        # via django-auth-ldap
 python-memcached==1.59
 python-radius==1.0
 python3-openid==3.1.0     # via social-auth-core
 python3-saml==1.4.0
-pytz==2018.9              # via celery, django, irc, tempora, twilio
-pyyaml==5.1               # via djangorestframework-yaml
+pytz==2019.1              # via celery, django, irc, tempora, twilio
+pyyaml==5.1               # via ansible-runner, djangorestframework-yaml
 requests-futures==0.9.7
 requests-oauthlib==1.2.0  # via msrest, social-auth-core
-requests[security]==2.21.0
-service-identity==17.0.0
+requests==2.21.0
+service-identity==18.1.0  # via twisted
 simplejson==3.16.0        # via uwsgitop
-six==1.12.0               # via asgi-amqp, asgiref, autobahn, automat, cryptography, django-extensions, irc, isodate, jaraco.classes, jaraco.collections, jaraco.itertools, jaraco.logging, jaraco.stream, pygerduty, pyhamcrest, pyopenssl, pyrad, python-dateutil, python-memcached, slackclient, social-auth-app-django, social-auth-core, tacacs-plus, tempora, twilio, txaio, websocket-client
+six==1.12.0               # via ansible-runner, asgi-amqp, asgiref, autobahn, automat, cryptography, django-extensions, irc, isodate, jaraco.classes, jaraco.collections, jaraco.itertools, jaraco.logging, jaraco.stream, pygerduty, pyhamcrest, pyopenssl, pyrad, python-dateutil, python-memcached, slackclient, social-auth-app-django, social-auth-core, tacacs-plus, tempora, twilio, txaio, websocket-client
 slackclient==1.1.2
 social-auth-app-django==2.1.0
 social-auth-core==3.0.0
 tacacs_plus==1.0
-tempora==1.14             # via irc, jaraco.logging
+tempora==1.14.1           # via irc, jaraco.logging
 twilio==6.10.4
-twisted==18.9.0           # via daphne
+twisted[tls]==19.2.0
 txaio==18.8.1             # via autobahn
 typing==3.6.6             # via django-extensions
 urllib3==1.24.3           # via requests
 uwsgi==2.0.17
 uwsgitop==0.10.0
-vine==1.2.0               # via amqp
-websocket-client==0.47.0  # via slackclient
+vine==1.3.0               # via amqp, celery
+websocket-client==0.56.0  # via slackclient
 xmlsec==1.3.3             # via python3-saml
 zope.interface==4.6.0     # via twisted
 
diff --git a/requirements/requirements_ansible.in b/requirements/requirements_ansible.in
index df9b482fe9..d3fbd871dd 100644
--- a/requirements/requirements_ansible.in
+++ b/requirements/requirements_ansible.in
@@ -46,6 +46,7 @@ ncclient==0.6.3
 netaddr
 # oVirt/RHV
 ovirt-engine-sdk-python==4.2.4    # minimum set inside Ansible facts module requirements
+pycurl==7.43.0.1    # higher versions will not install without SSL backend specified
 # AWX usage
 pexpect==4.6.0    # same as AWX requirement
 psutil==5.4.3    # same as AWX requirement
diff --git a/requirements/requirements_ansible.txt b/requirements/requirements_ansible.txt
index c9e97d4b2c..c6f8b313a1 100644
--- a/requirements/requirements_ansible.txt
+++ b/requirements/requirements_ansible.txt
@@ -2,12 +2,12 @@
 # This file is autogenerated by pip-compile
 # To update, run:
 #
-#    pip-compile --output-file requirements/requirements_ansible.txt requirements/requirements_ansible.in
+#    pip-compile -U -r --allow-unsafe --output-file requirements/requirements_ansible.txt requirements/requirements_ansible.in
 #
 adal==1.2.1               # via msrestazure
 appdirs==1.4.3            # via openstacksdk
-applicationinsights==0.11.1  # via azure-cli-core
-argcomplete==1.9.4        # via azure-cli-core, knack
+applicationinsights==0.11.9  # via azure-cli-core
+argcomplete==1.9.5        # via azure-cli-core, knack
 asn1crypto==0.24.0        # via cryptography
 azure-cli-core==2.0.35
 azure-cli-nspkg==3.0.2
@@ -42,82 +42,83 @@ azure-mgmt-web==0.41.0
 azure-nspkg==2.0.0
 azure-storage==0.35.1
 backports.ssl-match-hostname==3.5.0.1
-bcrypt==3.1.4             # via paramiko
+bcrypt==3.1.6             # via paramiko
 boto3==1.6.2
 boto==2.47.0
-botocore==1.9.3           # via boto3, s3transfer
-cachetools==3.0.0         # via google-auth
-certifi==2018.1.18        # via msrest, requests
-cffi==1.11.5              # via bcrypt, cryptography, pynacl
+botocore==1.9.23          # via boto3, s3transfer
+cachetools==3.1.0         # via google-auth
+certifi==2019.3.9         # via msrest, requests
+cffi==1.12.3              # via bcrypt, cryptography, pynacl
 chardet==3.0.4            # via requests
-colorama==0.3.9           # via azure-cli-core, knack
-cryptography==2.6.1       # via adal, azure-keyvault, azure-storage, paramiko, pyopenssl, requests-kerberos, requests-ntlm, secretstorage
-decorator==4.2.1          # via openstacksdk
-deprecation==2.0          # via openstacksdk
+colorama==0.4.1           # via azure-cli-core, knack
+cryptography==2.6.1       # via adal, azure-keyvault, azure-storage, openstacksdk, paramiko, pyopenssl, requests-kerberos, requests-ntlm
+decorator==4.4.0          # via dogpile.cache, openstacksdk
 docutils==0.14            # via botocore
-dogpile.cache==0.6.5      # via openstacksdk
+dogpile.cache==0.7.1      # via openstacksdk
 enum34==1.1.6; python_version < '3' # via cryptography, knack, msrest, ovirt-engine-sdk-python
 futures==3.2.0; python_version < '3' # via openstacksdk, s3transfer
 google-auth==1.6.2
-humanfriendly==4.8        # via azure-cli-core
-idna==2.6                 # via cryptography, requests
-ipaddress==1.0.19         # via cryptography, openstacksdk
+humanfriendly==4.18       # via azure-cli-core
+idna==2.8                 # via requests
+ipaddress==1.0.22; python_version < '3' # via cryptography, openstacksdk
 iso8601==0.1.12           # via keystoneauth1, openstacksdk
 isodate==0.6.0            # via msrest
 jinja2==2.10.1
-jmespath==0.9.3           # via azure-cli-core, boto3, botocore, knack, openstacksdk
-jsonpatch==1.21           # via openstacksdk
+jmespath==0.9.4           # via azure-cli-core, boto3, botocore, knack, openstacksdk
+jsonpatch==1.23           # via openstacksdk
 jsonpointer==2.0          # via jsonpatch
-keystoneauth1==3.11.2     # via openstacksdk, os-client-config
+keystoneauth1==3.14.0     # via openstacksdk
 knack==0.3.3              # via azure-cli-core
-lxml==4.1.1               # via ncclient, pyvmomi
-monotonic==1.4            # via humanfriendly
+lxml==4.3.3               # via ncclient, pyvmomi
+markupsafe==1.1.1         # via jinja2
+monotonic==1.5; python_version < '3' # via humanfriendly
 msrest==0.6.1
 msrestazure==0.5.0
-munch==2.2.0              # via openstacksdk
+munch==2.3.2              # via openstacksdk
 ncclient==0.6.3
 netaddr==0.7.19
-netifaces==0.10.6         # via openstacksdk
-ntlm-auth==1.0.6          # via requests-credssp, requests-ntlm
-oauthlib==2.0.6           # via requests-oauthlib
+netifaces==0.10.9         # via openstacksdk
+ntlm-auth==1.3.0          # via requests-credssp, requests-ntlm
+oauthlib==3.0.1           # via requests-oauthlib
 openstacksdk==0.23.0
-os-service-types==1.2.0   # via openstacksdk
+os-service-types==1.6.0   # via keystoneauth1, openstacksdk
 ovirt-engine-sdk-python==4.2.4
-packaging==17.1
+packaging==19.0
 paramiko==2.4.2           # via azure-cli-core, ncclient
-pbr==3.1.1                # via keystoneauth1, openstacksdk, os-service-types, stevedore
+pbr==5.2.0                # via keystoneauth1, openstacksdk, os-service-types, stevedore
 pexpect==4.6.0
 psutil==5.4.3
-ptyprocess==0.5.2         # via pexpect
-pyasn1-modules==0.2.3     # via google-auth
-pyasn1==0.4.2             # via paramiko
-pycparser==2.18           # via cffi
-pycurl==7.43.0.1          # via ovirt-engine-sdk-python
-pygments==2.2.0           # via azure-cli-core, knack
-pyjwt==1.6.0              # via adal, azure-cli-core
+ptyprocess==0.6.0         # via pexpect
+pyasn1-modules==0.2.5     # via google-auth
+pyasn1==0.4.5             # via paramiko, pyasn1-modules, rsa
+pycparser==2.19           # via cffi
+pycurl==7.43.0.1
+pygments==2.3.1           # via azure-cli-core, knack
+pyjwt==1.7.1              # via adal, azure-cli-core
 pykerberos==1.2.1         # via requests-kerberos
-pynacl==1.2.1             # via paramiko
-pyopenssl==17.5.0         # via azure-cli-core, pyvmomi, requests-credssp
-pyparsing==2.2.0          # via packaging
+pynacl==1.3.0             # via paramiko
+pyopenssl==19.0.0         # via azure-cli-core, pyvmomi, requests-credssp
+pyparsing==2.4.0          # via packaging
 python-dateutil==2.6.1    # via adal, azure-storage, botocore
 pyvmomi==6.5
 pywinrm[kerberos]==0.3.0
-pyyaml==5.1               # via azure-cli-core, knack, openstacksdk, os-client-config
+pyyaml==5.1               # via azure-cli-core, knack, openstacksdk
 requests-credssp==1.0.2
 requests-kerberos==0.12.0  # via pywinrm
 requests-ntlm==1.1.0      # via pywinrm
-requests-oauthlib==0.8.0  # via msrest
-requests==2.20.0
-requestsexceptions==1.4.0  # via openstacksdk, os-client-config
+requests-oauthlib==1.2.0  # via msrest
+requests==2.21.0
+requestsexceptions==1.4.0  # via openstacksdk
 rsa==4.0                  # via google-auth
 s3transfer==0.1.13        # via boto3
 selectors2==2.0.1         # via ncclient
-six==1.11.0               # via azure-cli-core, bcrypt, cryptography, google-auth, isodate, keystoneauth1, knack, munch, ncclient, ntlm-auth, openstacksdk, ovirt-engine-sdk-python, packaging, pynacl, pyopenssl, python-dateutil, pyvmomi, pywinrm, stevedore
-stevedore==1.28.0         # via keystoneauth1
-tabulate==0.7.7           # via azure-cli-core, knack
+six==1.12.0               # via azure-cli-core, bcrypt, cryptography, google-auth, isodate, keystoneauth1, knack, munch, ncclient, openstacksdk, ovirt-engine-sdk-python, packaging, pynacl, pyopenssl, python-dateutil, pyvmomi, pywinrm, stevedore
+stevedore==1.30.1         # via keystoneauth1
+tabulate==0.8.2           # via azure-cli-core, knack
+typing==3.6.6; python_version < '3' # via msrest
 urllib3==1.24.3           # via requests
 wheel==0.30.0             # via azure-cli-core
-xmltodict==0.11.0         # via pywinrm
+xmltodict==0.12.0         # via pywinrm
 
 # The following packages are considered to be unsafe in a requirements file:
 pip==9.0.1