Note:
These notes are intended for RH AAP upgrades from AAP 2.1 or higher to AAP 2.4. Upgrades previous to 2.1 require a migration of Ansible content for compatibility with AAP 2.4.
Overview:
A. Set up the existing AAP inventory to match the existing environment.
B. Download and run the 2.4 installation program over top of the existing AAP installation.
Automation Controller Upgrade
Warning:
- Credentials or the subscription manifest is required after performing the upgrade on the license page
- Pre-AAP 2.1, RH Enterprise Linux and the AAP Controller upgrades require back-up of the AAP controller data, and then restoration of it.
Automation Hub Upgrade
Warning:
- If you generate a new token instead of adding an existing token, this will invalidate any existing tokens with the hub.
Prerequisites:
1. Backup the current environment. In this case, the location is: /myadminid/aapinstall/aap-setup-bundle-2.3-11-x86_64/
a. Navigate to the last install location still containing the extracted installation folder with the inventory file and the apitoken.
b. Run the backup:
$ sudo su -
# cd /myadminid/aapinstall/aap-setup-bundle-2.3-11-x86_64/
# mkdir /myadmin/aapinstall/aap2-3-11backup/
# ./setup.sh -e 'backup_dest=/myadmin/aapinstall/aap-2-3-11backup/' -b
# exit
2. Ensure you have all your LDAP set-up information.
3. Ensure you AAP is using execution environments (EEs) rather than the older virtual environments.
Convert them, if not.
4. If using EDA, Event Driven Ansible, you'll need to install a separate VM for it.
a. Deploy a RHEL 9 VM, 16 GB RAM, 4 CPU, 40 GB (/ drive)
b. Add an extra disk, via Edit Settings --> Add New Device --> Disk (e.g. 40 GB)
- $ ssh myadminid@aapeda@mindwatering.net
<enter pwd>
- - Make awx folder:
- - - $ sudo mkdir /var/awx/
- - Confirm the device id:
- - - $ sudo lsblk
<view output and confirm entry /dev/sdb 40 GB>
- - Format the new disk:
- - - $ sudo fdisk /dev/sdb
- - - - Command: n -> p 1 -> nnnn - nnnnnnnnnn <enter>
- - - - Command: w
- - Format the OS:
- - - $ sudo mkfs.xfs -f /dev/sdb1
- - Test the /dev/sdb1 partition:
- - - $ sudo mount /dev/sdb1 /var/awx/
- - - $ sudo lsblk
<verify drive mapped: sdc1 ... part /var/awx >
- - - $ systemctl daemon-reload
- - - $ sudo lsblk
<verify drive mapped: sdc1 ... part /var/awx >
- - Get the UUID of the partition for /etc/fstab for the permanent mapping:
- - - $ sudo blkid /dev/sdb1
<view output - /dev/sdb1: UUID12345abc-...12a" TYPE="XFS" PARTUUID="a12b12cd-01" >
- - Map the new disk via fstab as/var/awx using either the UUID or the dev mapping (both entries are listed below, just use one of the lines)
- - - $ sudo vi /etc/fstab
. . .
UUID=12345abc-...12a /var/awx xfs defaults 0 0
/dev/sdb1 /var/awx xfs defaults 0 0
. . .
<esc>:wq (to save)
We have other tasks but we'll do them together below.
5. Verify the RAM on the controllers
Ansible recommends 1GB memory/10 forks, and a 2GB reservation for the AAP controller.
6. AAP Controller requires umask be set to 0022. If is not set, the setup.sh will fail.
This is the default for most Linux environments for normal users.
- To change for all users:
$ sudo vi /etc/profile.d/set-umask-for-all-users.sh
- To adjust just the "current" user
$ vi ~/.bashrc
<a to append>
umask022
<esc>:wq to save and close
7. Verify access to the AAP Controller PostgreSQL 13 db:
$ sudo awx-manage check_db
8. Ensure your ansible_user account set-up in the inventory file is not expired and that the ansible_private_key_file private key is still working.
e.g. if expired:
$ sudo chage -M <max_days> ansible_user
9. Set-up the ansible_user account on the new Event Driven Ansible VM created above. Add the public key to the VM so that this user from the AAP Controller appliance/VM can remotely log-in/ssh. Test a SSH session from the main AAP Controller.
a. Create the ansible_user on the AAP EDA controller:
$ ssh myadminid@aapeda.mindwatering.net
<enter pwd>
$ sudo useradd -d /home/ansible_user -m ansible_user
- OR -
$ sudo useradd -d /users/ansible_user -m ansible_user
$ sudo usermod -a -G wheel ansible_user
$ sudo passwd ansible_user
<enter new pwd>
<enter new pwd confirmation>
$ exit
b. Copy the key to the AAP EDA controller:
$ ssh ansible_user@aapdev.mindwatering.net
<enter pwd>
$ ssh-copy-id ansible_user@aapeda.mindwatering.net
<enter pwd>
c. Test the ssh using the key copied:
$ ssh aapeda.mindwatering.net
<verify auto login worked>
$ exit
(to exit appeda)
$ exit
(to exit appdev)
10. Ensure that /tmp and /var/tmp are mounted as exec
$ ssh myadminid@aapdev.mindwatering.net
$ sudo vi /etc/fstab
...
mount -o remount,exec /tmp
mount -o remount,exec /var/tmp
...
<esc>:wq to save and close
Repeat above on the aaphub.mindwatering.net and the aapeda.mindwatering.net
11. Ensure that /var/tmp is 755
$ ssh myadminid@aapdev.mindwatering.net
$ sudo chmod 755 /var/log
12. If you have your system with yum updates disabled, then you need to enable yum to update RH packages. Update yum.conf before the upgrade, remove the following (if you have it) and then re-add it back afterwards
a. Update yum.conf:
$ ssh myadminid@aapdev.mindwatering.net
$ sudo vi /etc/yum.conf
Remove the line: exclude=kernel* redhat-release*
...
<esc>:wq to save and close
b. Update dnf.conf:
$ sudo vi /etc/dnf/dnf.conf
Remove the line: exclude=kernel* redhat-release*
...
<esc>:wq to save and close
Repeat above on the aaphub.mindwatering.net and the aapeda.mindwatering.net
13. The ansible user and the setup.sh program need sudo to be enabled on the AAP Controller, the AAP Hub, and the AAP EDA Controller:
Uncomment the line below, noting to uncomment the one that does not require a password:
$ ssh myadminid@aapdev.mindwatering.net
$ sudo visudo
...
%wheel ALL=(ALL) NOPASSWD: ALL
...
<esc>:wq to save and close
Repeat above on the aaphub.mindwatering.net and the aapeda.mindwatering.net
14. The upgrade will likely override custom cert/keys. If applicable, perform backups of those files:
a. Backup the main AAP controller tower.key and tower.cert files:
$ ssh myadminid@aapdev.mindwatering.net
$ <enter pwd>
$ sudo vi /etc/nginx/conf.d/automation-controller.nginx.conf
<read location e.g. /etc/tower/tower.cert and /etc/tower/tower.key>
$ sudo cp /etc/tower/tower.cert /etc/tower/tower_backup_yyyymmdd.cert
$ sudo cp /etc/tower/tower.key /etc/tower/tower_backup_yyyymmdd.key
$ exit
b. Backup the AAP Hub key and cert files:
Repeat above on the aaphub.mindwatering.net but also backup the custom cert in the pulp config folder
$ ssh myadminid@aapdev.mindwatering.net
<enter pwd>
$ sudo cp /etc/pulp/certs/pulp_webserver.crt /etc/pulp/certs/pulp_webserver.crt_backup_yyyymmdd.crt
$ sudo cp /etc/pulp/certs/pulp_webserver.key /etc/pulp/certs/pulp_webserver.key_backup_yyyymmdd.key
$ exit
Note:
aapeda.mindwatering.net cert and key do not need to be backed up since the EDA does not exist before this upgrade
Upgrade:
1. Download the RH AAP latest 2.4 version:
Browser --> access.redhat.com/downloads/content/480/ver=2.4/rhel---9/2.4/x86_64/product-software
--> Login
Look for the offline bundle version. It includes dependencies and can make installation a bit easier.
- naming format of the "offline" bundle version:
ansible-automation-platform-setup-bundle-<latest-version>.tar.gz
If not retrieved on the AAP server, transfer the downloaded file to the AAP controller VMs via SSH/SCP (e.g. Filezilla)
e.g. to /myadminid/aapinstall/
Uncompress the tar file:
$ ssh myadminid@aapdev.mindwatering.net
<enter pwd>
$ cd /users/myadminid/aapinstall/aap-setup-bundle-2.3-11-x86_64/
$ tar xvzf ansible-automation-platform-setup-bundle-<latest-version>.tar.gz
2. Update the AAP inventory file in the archive, or re-use the current one if nothing has changed. The inventory file is part of the expanded tar extract. The file path differs based on the tar downloaded ("internet" or "bundle") For the offline bundle version...
$ cd ansible-automation-platform-setup-bundle-2.4-1-x86_64
To reuse the existing inventory file and token from the previous installation:
Move the default file to a backup file.
$ mv inventory inventory_backup
$ cp /users/myadminid/aapinstall/inventory ./
$ cp /users/myadminid/aapinstall/apitoken ./
Inventory file notes:
- The use of localhost for any [automationhub] or [automationcontroller] sections of the file is not allowed.
- Add node_state-deprovision to the end of the node line to deprovision only isolated nodes, not execution nodes.
e.g. hostname.domain ansible_host=192.168.111.115 node_type=hybrid node_state=deprovision
- pg_password can only contains the special characters: !, #, 0, and @. The admin username cannot be changed.
- Reuse the existing token like below:
$ vi inventory
automationhub_api_token=<api_token>
...
<esc>:wq, to save
Example simple inventory file from RH AAP documentation:
Notes:
The following example inventory file uses an embedded (non external database). With an embedded database, the host under the [database] heading is left out, the pg_host is an empty string (e.g. pg_host='') and the port 5432 is still populated (e.g. pg_port=5432).
The following example shows the registry_url, and the registry_username, and its registry_password completed for a machine on a RedHat subscription. If this upgrade is being performed outside of using subscription manager (e.g. with a local repo), then the URL, username, and password fields can be left empty strings. The repo will have to have the AAP available, and the setup.sh script will auto disable the AAP repo afterwards. See below for the instructions to enable the repo using dnf config-manager.
$ sudo vi inventory
[automationcontroller]
aapdev.mindwatering.net
[automationhub]
aaphub.mindwatering.net
[automationedacontroller]
aapeda.mindwatering.net
[database]
[all:vars]
admin_password='guiadminpassword'
pg_host=''
pg_port=5432
pg_database='awx'
pg_username='awx'
pg_password='<awxpassword>'
registry_url='registry.readhat.io'
registry_username='reguserid'
registry_password='regpassword'
automationcontroller_main_url= 'https://aapdev.mindwatering.net'
automationhub_main_url= 'https://aaphub.mindwatering.net'
automationhub_pg_host='hubdev.mindwatering.net'
automationhub_pg_port=5432
automationhub_pg_database="automationhub'
automationhub_pg_username='automationhub'
automationhub_pg_password='HubReallyAmazingPassword'
automationhub_pg_sslmode = 'prefer'
automationhub_authentication_backend = "ldap"
automationhub_ldap_server_uri = "ldaps://ldapserverint.mindwatering.net"
automationhub_ldap_bind_dn = "cn=myldapadmin, dc=mindwatering, dc=net"
automationhub_ldap_bind_password = "MyBetterThanGoodPassword"
automationhub_ldap_user_search_base_dn = "ou=people, dc=mindwatering, dc=net"
automationhub_ldap_group_search_base_dn = "ou=aapgroup, dc=mindwatering, dc=net"
automationedacontroller_admin_password='EDAUIReallyAmazingPassword'
automationedacontroller_pg_host = 'edadev.mindwatering.net'
automationedacontroller_pg_port=5432
automationedacontroller_pg_database ='automationedacontroller'
automationedacontroller_pg_username='automationedacontroller'
automationedacontroller_pg_password='EDAReallyAmazingPassword'
automationedacontroller_pg_sslmode='prefer'
...
# if the server has a non generic name, it can be left out, if there is only one name and it is specified under the [automationedacontroller] heading, it can be left out
# automationedacontroller_allowed_hostnames = 'aapeda.mindwatering.net'
...
<esc>:wq to save and close
3. After the modifications of the inventory are complete, run the installer:
$ ./setup.sh
Post Upgrade.
1. If you had/have custom certs, ssh back into the main AAP Controller, and the AAP Hub, and restore the certificates back on the main AAP controller and hub.
a. Update certificate back on the main AAP controller:
$ ssh myadminid@aapdev.mindwatering.net
<enter pwd>
$ sudo mv /etc/tower/tower.cert /etc/tower/tower_backup_post2-4upgrade.cert
$ sudo mv /etc/tower/tower.key /etc/tower/tower_backup_post2-4upgrade.key
$ sudo mv /etc/tower/tower_backup_yyyymmdd.cert /etc/tower/tower.cert
$ sudo cp /etc/tower/tower_backup_yyyymmdd.key /etc/tower/tower.key
$ sudo systemctl restart nginx.service
Verify the GUI works in the web browser.
$ exit
b. Update the certificate back on the main AAP Hub
$ ssh myadminid@aaphub.mindwatering.net
<enter pwd>
$ sudo mv /etc/pulp/certs/pulp_webserver.crt /etc/pulp/certs/pulp_webserver_backup_post2-4upgrade.crt
$ sudo mv /etc/pulp/certs/pulp_webserver.key /etc/pulp/certs/pulp_webserver_backup_post2-4upgrade.key
$ sudo mv /etc/pulp/certs/pulp_webserver.crt_backup_yyyymmdd.crt /etc/pulp/certs/pulp_webserver.crt
$ sudo mv /etc/pulp/certs/pulp_webserver.crt_backup_yyyymmdd.key /etc/pulp/certs/pulp_webserver.key
$ sudo systemctl restart nginx.service
Verify the GUI works in the web browser.
$ exit
______________
Additional Notes:
Ansible Application (Private) Hub Upgrade Variables for Inventory File:
LDAP Information:
Inventory File Section: ldap_extra_settings
Required settings:
automationhub_authentication_backend = "ldap"
automationhub_ldap_server_uri = "ldaps://ldapserverint.mindwatering.net"
automationhub_ldap_bind_dn = "cn=myldapadmin, dc=mindwatering, dc=net"
automationhub_ldap_bind_password = "MyBetterThanGoodPassword"
automationhub_ldap_user_search_base_dn = "ou=people, dc=mindwatering, dc=net"
automationhub_ldap_group_search_base_dn = "ou=aapgroup, dc=mindwatering, dc=net"
Additional settings:
automationhub_ldap_user_search_scope = "SUBTREE"
automationhub_ldap_user_search_filter = "(uid=%(user)s)"
automationhub_ldap_group_search_scope = "SUBTREE"
automationhub_ldap_group_search_filter = "(objectClass = Group)"
automationhub_ldap_group_type_class = "django_auth_ldap.config:GroupOfNamesType"
Note:
For AD LDAP, the user_search_filter and group_type_class will likely be like:
automationhub_ldap_user_search_filter = "(sAMAccountName=%(user)s)"
automationhub_ldap_group_type_class = "django_auth_ldap.config:NestedActiveDirectoryGroupType"
Example Passing Additional LDAP settings (e.g. a field map) with the setup.sh run:
# vi ldapextras.yml
#ldapextras.yml
---
ldap.extra_settings:
AUTH_LDAP_USER_ATTR_MAP: '{"first_name": "FirstName", "last_name": "LastName", "email": "MailAddress"}'
AUTH_LDAP_REQUIRE_GROUP: 'cn=aapadmins,ou=groups,dc=mindwatering,dc=net'
<esc>:wq (to save)
Note:
For AD LDAP, the user field mapping will be more like:
AUTH_LDAP_USER_ATTR_MAP: '{"first_name": "givenName", "last_name": "sn", "email": "upn"}'
To run the setup with the custom map:
# ./setup.sh -e @ldapextras.yml
Troubleshooting Hub LDAP issues:
a. SSH into the AAP Hub:
$ ssh myadminid@aaphub.mindwatering.net
<enter pwd>
b. Enable logging by adding a line to the settings.py file:
$ sudo vi /etc/pulp/settings.py
...
GALAXY_LDAP_LOGGING=TRUE
<esc>:wq (to save)
c. Restart services
$ sudo systemctl restart pulpcore-api.service
$ sudo systemctl restart nginx.service
d. Watch the logging during a login attempt to the administrative UI:
$ sudo journalctl -f
<watch for error, and use ctnl-c to quit>
e. After seeing issue, update the hub configuration file.
f. Disable logging and remove the logging line added in step b above
$ sudo vi /etc/pulp/settings.py
RH SSO to LDAP for AAP Controller:
Scan for sso lines in the inventory file to comment out or remove:
[sso]
ssoserverint.mindwatering.net
...
# sso_host=''
# sso_custom_keystore_file='/path/to/sso.jks'
# sso_keystore_password='********'
# sso_console_admin_password='********'
Add the LDAP configuration above.
Using dnf config-manager to enable AAP repo for AAP controller setup.sh run.
On the AAP controllers, the AAP execution nodes, the AAP Hubs, and the AAP EDA controller, you can use the dnf to enable an alternate repo. Make sure that the subscription manager is NOT set-up in the OS, and the inventory file has empty string values for the repository variables. if using this approach. Please also note that setup.sh is running the --disable as part of its script, so we did not need to disable it afterwards. (AAP updates should not be done as part of dnf/yum but only by setup.sh, which is why the rep should not be left enabled.)
To enable the AAP repo:
$ sudo dnf config-manager --enable ansible-automation-platform
$ sudo dnf repolist
<view output, confirm that baseOS / appstream and AAP are all listed>
Duplicate Host Issue w/Second setup.sh Run:
The setup.sh can be run after a configuration change or a previous installation/upgrade failed to a configuration problem. Sometimes on the next run, setup.sh will fail because of a "duplicate" host entry.
For example, to clear the duplicate host entry, for hubdev.mindwatering.net:
a. SSH to the primary AAP controller:
$ ssh myadminid@aapdev.mindwatering.net
<enter pwd>
$ awx-manage deprovision_instance -hostname hubdev.mindwatering.net
<wait>
b. Re-run the setup again.
previous page
|