Continue the Network Automation Journey using AWX

I have been working on a project recently to assist a customer in upgrading thousands of devices in their network. Managing such a large-scale upgrade requires automation to ensure consistency, efficiency, and reduced manual intervention. For this, AWX was selected as the UI tool of choice, as it provides a powerful interface for managing Ansible playbooks, job templates, inventories, and credentials.

What is AWX?

AWX is the open-source upstream project for Ansible Tower, providing a web-based UI for managing Ansible automation tasks. It allows you to organize and execute playbooks, define inventories, manage credentials, and create job templates—all from a user-friendly interface. AWX is particularly useful when dealing with complex automation workflows, like network device upgrades, where you need to manage hundreds or thousands of devices.

AWX’s Limitation with Interactive Prompts

One challenge I encountered with AWX is its limitation in handling interactive prompts from network devices. For example, when upgrading Cisco devices, there are scenarios where the device sends a prompt (e.g., asking for confirmation), and AWX is unable to directly interact with such prompts during playbook execution. This can cause the playbook to hang or fail, which is not ideal in an automated environment.

Workaround: Using EEM Scripts

To overcome this limitation, I incorporated Cisco Embedded Event Manager (EEM) scripts into the workflow. EEM scripts run directly on the network devices and can handle interactive prompts or actions that require confirmation during the upgrade process. This allows the playbooks to execute without interruption, ensuring that the device upgrade process remains smooth and fully automated.

---

Building a 3-Node Kubernetes Cluster

To begin the AWX deployment, first set up a 3-node Kubernetes cluster. This was crucial to ensure scalability and resilience for AWX, which I chose to run on Kubernetes. The cluster consisted of one control plane node and two worker nodes. The setup was done using kubeadm, and containerd was selected as the container runtime. A proper pod network was deployed to allow communication between nodes.

Note that you may need to configure storage classes and networking plugins to handle AWX's persistence and networking requirements.

Installing AWX Using AWX Operator

With the Kubernetes cluster ready, the next step was deploying AWX using the AWX Operator. AWX Operator simplifies the deployment and management of AWX on Kubernetes. At the time of writing, I am using AWX version 24.6.1. Please refer to the AWX operator documentation in how to deploy AWX on a Kubernetes cluster.

Setting Up Custom Execution Environments or Using Requirements YAML

AWX provides the flexibility of using Custom Execution Environments (EE) or specifying dependencies in a requirements.yaml file under your project. By default, the Cisco ansible galaxy collection is not included in AWX operator. There are two options you can make AWX working with custom ansible collection:

1. Custom EE: Building a custom execution environment that included specific versions of Python libraries and network-related modules like netmiko and ncclient. This requires certain knowledge in docker - you will need to create a Dockerfile, specify required libraries and build the docker image and then upload to a repository. In AWX, you will then create a Custom EE using the custom docker image.

2. Requirements YAML: Alternatively, set up a requirements.yaml in the project, where you could specify collections and roles required for the playbooks. This was especially useful for dependency management, making sure that all needed Ansible Galaxy roles and collections were installed during the project sync.

---

AWX User Interface Overview

For those new to AWX, the UI offers a clear and structured way to manage automation:

1. Projects: This is where you link the Git repositories containing the Cisco IOS XE upgrade playbooks. Projects can be synced to pull the latest playbook updates automatically. Make sure every time you commit to Git, you will need to hit the sync button in the project. The synchronization is not automatic.

2. Inventories: Devices (routers and switches) were organized into Inventory Groups. This allows separation of devices by type or purpose, ensuring that the correct playbooks are applied to the right devices.

3. Credentials: You will store device access credentials securely within AWX, including SSH keys and passwords, making it easy to automate connections to the Cisco devices during playbook execution.

4. Templates: Playbooks were turned into Job Templates, with surveys used to capture necessary inputs such as firmware versions, file names, and checksum values.

The AWX UI made it straightforward to manage these components and automate the entire playbook execution process.

AWX Job Templates for Cisco IOS XE Upgrades

The playbooks for upgrading Cisco IOS XE devices were ported into AWX as templates. Key templates included:

1. Activate Firmware (activate.yaml): This template used a survey to capture variables like target_version, upgrade_filename, and md5_checksum to ensure a smooth firmware activation process using the install activate command.

2. Flash Cleanup (cleanup-flash.yaml): This template was used to remove old, inactive files from the device’s flash memory via the install remove inactive command.

3. Upload Firmware via TFTP/SCP (upload-image-tftp.yaml): This playbook handled the firmware file upload, with survey options for selecting the transfer mode (TFTP or SCP), specifying the server_ip, and validating available storage space on the device.

The flexibility offered by AWX’s job templates and survey mechanisms made it possible to customize the upgrade process for different types of devices with minimal manual intervention.

Repository

For those who missed my Github repository, visit here: svenuscf/ansible-cisco-upgrade (github.com). A new sub-directory awx is created to store related playbooks for AWX deployment.

Good luck!