Auto Remediation with Zabbix and Ansible Tower Part 1

Objective Goals

1.) Automatically detect new systems and logically add them to a group

2.) If no Agent is found to be running, then to have Ansible Tower run a job to set it up

3.) Create a collection of playbooks that can be used for auto remediation

What we Need

1.) A Zabbix Server

2.) Ansible Tower

3.) Some hosts with agents talking to Zabbix

Zabbix Server Setup

I am using Zabbix version 4.0 for this guide, there is no value for me to run through the setup of the server as its very straight forward and well documented on the Zabbix site

https://www.zabbix.com/documentation/4.0/manual/installation

If you would like to use an Ansible Role to install this for you then look at the Galaxy roles created by dj-wasabi.

For this initial blog I am only using the server and web functions.

We need to install nmap on the Zabbix Server as well as give the zabbix user the ability to run nmap add in the following rule into your sudoers file.

zabbix ALL=(root) NOPASSWD: /usr/bin/nmap


This allows Zabbix to detect the OS that is in use along with more advanced discovery rules.

Auto Discovery

With the Server running, the only thing it is aware of is itself, so we need to start adding our nodes to it there are a couple of ways we could do this.

1.) Go ahead and install agents on all our boxes as soon as the agent starts it will call home and the Zabbix server will see it.

2.) Create a auto discovery rule to scan the network for boxes and then perform some action based on it.

I want full automation for this so option 1 whilst we could have Ansible install the agents on our inventory and any new server we build, does leave the problem of what if I forget to install it on a existing server or someone builds a new server outside of my automation platform, eventually an issue will come up and I wont be aware of it because it has no agent.

Option 2 allows us to do some clever conditions bases on the discovery like is it a linux box or a windows box, what environment is this for ( Prod, Dev, QA etc ) Does it have an application already installed on it ( Web, Database etc ), and we can also have a self healing agent if the agent dies.

To setup the discovery rules in the Zabbix server go to Configuration, Discovery and select the Create discovery rule on the right hand side


I am going to create my first rule to detect any of my production linux servers, I know they are production based on the subnet 10.10.10.X/24 I will know if these are Linux servers if they have SSH enabled port 22, I want to also have a condition based on it, to make sure that the server has been detected and alive for at least an hour before I do anything so we dont work on boxes that have been spun up by mistake etc.

Lets create the Production Windows one also which is the same but instead of SSH I am looking for RDP port 3389

You may ask why split this up, why not just detect for all pingable devices on the subnet, by doing this to ssh and rdp I remove bringing in any box that I wouldnt want Ansible to connect to for now.

Next we want to create an action based on the discovery

Go to Configuration and Actions

Make sure you change the Event Source to Discovery before clicking on create action.

We want to set the conditions here for a production linux server make sure the calculation is set to And and not And/Or.

1.) Lets make sure its actually a Linux server so set the Received value to equal Linux

2.) The Host IP should be on our Production subnet

3.) The Discovered state should still be alive

4.) Finally it should have been up for at least an hour

For Windows Servers I changed the condition to look for Pot 3389 instead of the Received Value being Windows as 2016 + does not return Windows yet.

Click on the Operations Tab , we now specify what to do if the conditions are met.

We want to add to the Linux Host Group and link it to the Linux OS Templates

Again the WIndows one will look the same but change the Host group to Windows Servers and the Template to OS Windows

Once we have set these we will see our discovered rules as such

Ansible Tower Prep

Our Tower server is going to require us to install the zabbix-api python package, simply ssh into your tower server and run the following commands

source /var/lib/awx/venv/ansible/bin/activate
umask 0022
pip install zabbix-api

I would strongly recommend we also use zabbix as a dynamic inventory source, if we will be using this for auto remediation etc then what better inventory than itself.

The custom inventory script can be found on the Ansible github page

https://raw.githubusercontent.com/ansible/ansible/devel/contrib/inventory/zabbix.py

You want to copy this into your Tower server under inventory scripts

We also need to modify this script to be able to connect to our Zabbix Server, if you scroll down you will find the self.zabbix section

def __init__(self):

    self.defaultgroup = 'group_all'
    self.zabbix_server = 'http://zabbix.example.com'
    self.zabbix_username = 'Admin'
    self.zabbix_password = 'password'
    self.validate_certs = True
    self.read_host_inventory = False
    self.use_host_interface = True

I do not personally like how this is set and I will update this guide with a better solution soon

Create a new Inventory called Zabbix and then go to sources and add a new source, this will be our custom script we just created like so

Make sure it scans and completes and you should see your hosts and groups get populated.

If you discovery rule has already detected windows guests then the Windows group will come down, we want to use this to set the winrm connectivity.

My Groups after the sync

Click on the Windows Servers and add the connectivity information into the details pane

Going Forward any box that is put into that host group on Zabbix will now be set to connect over the above winrm settings. You can use this method with other systems, some people may have a different SSH port for Production over Development on the Linux servers this is ideal for that.

Zabbix Agents

We need to create a Job Template that can install the Zabbix Agent on our Linux and Windows Servers

I have created a role based off dj-wasabi Zabbix Agent role, note at time of writting this I found an annoyance in his role for RHEL based servers if they are runing SELINUX but in Permissive mode then it requires to install the policycoreutils-python package, but if this is a new server that isn’t connected to any repos etc then we have a failure, so I took this check out in my forked version found here.

https://github.com/p-avery/Ansible-Zabbix

You want to create a job template for the agent install, the role works for all linux and windows, but we would have a different credential file based on the OS type ( also possible based on lifecycle env ) so tick the prompt on launch for the credential.

Make sure the Limit is set to Prompt on launch, unless this is enabled the API wont honor any limit we send.

Also Make sure you set the Enable Concurrent Jobs, as will be running all of this from an API call we may have multiple calls each time.

Take a note of the job template number for each of the templates you create this can be found in the URL

“https://tower.example.com/#/templates/job_template/123

In the case above its id 123

Create a Zabbix User and Team In Tower

You can have this user pre created externally AD etc etc for this I will have it created as a localuser

Create our Zabbix user, go to Users and click on the green + to create a new one, I am calling it Zabbix

Note that I have made this User a normal user there is no need to make it any higher as it will only be used for the API calls.

Next go to Teams and click on the Green + to add a new Team

I have called the Team Monitoring and placed it into my Default Organization


Assign our Zabbix user to the team by clicking on the Users Tab and click on the Green + and select the Zabbix User give them Member rights.

Finally go to the Permissions Tab on the Team and add in the required projects, job templates, inventories and Credentials needed along with the relevant role for each. ( Note for the Credentials I only need use as this account would never change a credential file )

API Token Authentication

With Ansible Tower 3.3 we can now create our own application tokens for authentication, Prior to 3.3 we had 2 options either authenticate with Username and Password which isn’t ideal or install the tower-cli onto our server ( Zabbix ) in our case again not really ideal. Now though we can use a token.

As an Administrator in the same Organization as the Zabbix user create an Application, by clicking on Applications and clicking on the Green +

Set the Authorization Grant Type to Resource owner password-based and the Client Type set to Confidential. After you save it click on the application and take a note of the url it will give you the id number of it at the end. In my case its 3

“https://tower.example.com/#/applications/3”

Now log out of the Tower server as the admin and log back in as the zabbix user, after authenticating go to the api url for the application tokens

“https://tower.example.com/api/v2/applications/3/tokens/”

Note the above url you will need to change the number from 3 to whatever your application id was and you should see a screen like such

In the Content box at the bottom there is “description”: “”,

in the value for description that is just blank add in a description I have done zabbix-svc_API and then click on Post

After you Post the API Screen will then Display the details of the token

You need the token you can see from the my return value above it is

    "token": "aQVs6tuV2qVOr2u3F3OPyPq33H2k7C",

Leave a Reply

Your email address will not be published. Required fields are marked *