Zabbix Server

Auto Scaling support # expert

How does AWS Auto Scaling work with Zabbix?

To handle Auto Scaling from Zabbix there are three separate things to take care of:

1. Add a new instance to Zabbix when it appears. This can be done with active Zabbix agent auto-registration (6.4, 6.0, 4.0). That way, when a new instance appears, Zabbix adds it automatically as a new host.
2. Store the instance ID in the Zabbix inventory. When the instance goes away (scale in) we receive a message with its ID, so we need that ID in the host inventory to look it up and delete it.
3. Receive the AWS scaling messages. When AWS produces a scaling event it publishes an SNS message to a topic you define. By connecting that topic to an SQS queue we can read the messages from Zabbix and remove the instances that leave the scaling groups.

Configure the client

Two steps are needed to prepare your clients for Auto Scaling in Zabbix: configure the Zabbix agent, and add a file with a user parameter for the instance_id item.

Zabbix agent configuration

Alongside any other change you may need, six settings have to change in your Zabbix agent’s local configuration file, normally at /etc/zabbix/zabbix_agentd.conf:

• Comment out the Server option.
• Add ServerActive=YOUR__ZABBIX__IP__ADDRESS to configure the active server.
• Comment out the Hostname option.
• Set StartAgents to 0.
• Set HostnameItem to system.hostname.
• Set HostMetadataItem to system.uname.

This puts the Zabbix client in active mode (the client pushes information to the server instead of receiving checks from it). It also makes Zabbix use the machine name as the host name, so you don’t have to change it every time a new server boots. Finally, it makes Zabbix send the output of uname as host metadata: AWS instances use a special kernel, so the value aws shows up there and we can use it to register the server.

The instance_id parameter

By default, when a new Auto Scaling server appears in Zabbix it is linked to a special template called AWS EC2 Template, which has an item named instance_id bound to the inventory alias field. To feed it, add a user parameter to your Zabbix client by creating this file:

UserParameter=aws.ec2.instance_id,curl -s http://169.254.169.254/latest/meta-data/instance-id

Configure the server

With the clients configured, the next step is configuring the server to delete removed instances. The fastest and simplest route has two steps: send the scaling messages to an SQS queue, and configure the server to read it.

Send the scaling messages to an SQS queue

We want the notifications sent by the Auto Scaling service stored in an SQS queue, for durability and scale. Since AWS Auto Scaling publishes notifications to an SNS topic, we need to:

• Create an SNS topic.
• Create an SQS queue.
• Configure SNS to deliver the messages to SQS.
• Create an IAM user that can read the messages.

Because that setup is fiddly, you can use our CloudFormation stack to create every element in the region you want. After deploying the stack, the Outputs tab gives you four values to write down:

• SNSTopicARN: the ARN of the SNS topic created.
• SQSQueueURL: the URL of the SQS queue.
• SQSReadIAMUserAccessKey and SQSReadIAMUserSecretKey: the IAM user’s key and secret.

You need those values for the configuration. Edit /etc/zabbix/zabbix_ami.yml, in the block that looks like this:

sqs:
  delay_seconds: 60
  default_region: us-east-1
  queues:
    - url:
      access_key_id:
      secret_access_key:
      region:

For every SQS queue with scaling notifications, add an array entry like the existing one with the values CloudFormation gave you. There is more about this file in the important files section.

Important files # expert

The Zabbix AMI has a number of special files you should know about.

/etc/zabbix/zabbix_ami.yml

This is the most important one, because it drives the configuration of the special features, such as the scaling checks or the SNS notifications. Here is an example (with fake data):

zabbix:
  api:
    user: autoscaler_api_user
    password: i-073672daf01037aea94
sns:
  aws:
    access_key_id: AKK4L6A3OXZZ5HE2POQRU7V
    secret_access_key: cICycAg26hSDpsJwbhZjbaaYlt4Tz3n1xqE7Oxp6X2Hf
    region: us-east-1
  topics:
    not_classified: arn:aws:sns:us-east-1:000000000:Zabbix-General-SNS
    information:    arn:aws:sns:us-east-1:000000000:Zabbix-General-SNS
    warning:        arn:aws:sns:us-east-1:000000000:Zabbix-General-SNS
    average:        arn:aws:sns:us-east-1:000000000:Zabbix-General-SNS
    high:           arn:aws:sns:us-east-1:000000000:Zabbix-General-SNS
    disaster:       arn:aws:sns:us-east-1:000000000:Zabbix-General-SNS
sqs:
  delay_seconds: 60
  default_region: us-east-1
  queues:
    - url: https://sqs.us-east-1.amazonaws.com/000000000/zabbix-pro-sqs-queue
      access_key_id: AKIA3OXZZ5HEWUR5A16CNWH
      secret_access_key: 8ttYO3Ba9K6p0C+aj3fH5cDt2eRLHmlkHbMtlrpv18S7KRbC
      region: us-east-1

The file currently has three separate sections:

• zabbix: since we need to talk to the Zabbix server to delete stale hosts, we need API access. We create a dedicated user for it, and its credentials go here.
• sns: the configuration required to send alarms through SNS. This is where the IAM user credentials and the SNS topics we use are set.
• sqs: the configuration that drives the AWS scaling messages. Here we define how often the queue is polled (delay_seconds) and the different queues to check.

/var/log/zabbix-sqs-reader/zabbix-sqs-reader.log

This file records the Auto Scaling events read from SQS.

Before you configure anything you will see the message Ops! It looks like the file /etc/zabbix/zabbix_ami.yml hasn't been configured yet. We better sleep for 5 minutes. — that is normal, and it goes away once you configure zabbix_ami.yml.

AMI updates # expert

Since Zabbix ships new releases with new features and improvements, the best approach is to update your Zabbix AMI from time to time. Unfortunately, those same features involve database and internal changes that can be painful when upgrading an existing installation.

Our Zabbix AMI is the best way to avoid those problems, because we offer a path between major versions that is faster, simpler and safer than the usual one.

Update at a glance

The update process for our AMI is fairly simple, and boils down to:

• Dump your current database.
• Launch a new AMI on the latest stable release.
• Upload the database dump to the new instance.
• Run our migration script.

This way you keep the current and the new version running at the same time, and you can test as much as you need to be sure everything works before moving the EIPs attached to that instance, or migrating clients gradually.

Update process

Step 1: back up the source server

The first step is taking an image of the source server, to avoid any accidental data loss. You can follow the AWS guide on creating an AMI.

Step 2: dump the database and note the Zabbix version

Log into the instance over SSH and become root:

sudo /bin/su -

Then dump the database to a file:

mysqldump zabbix > zabbix_origin_database.sql

Finally, download that file to your machine so you can upload it to the new server. Note down the Zabbix version you are currently running too: you can see it in the web interface, and it should be a number like 4.0, 4.2, 4.4 or 5.4.

Step 3: launch a new Zabbix instance

Next, create a new Zabbix instance using the latest stable AMI release, which you can find on our AWS Marketplace seller profile. You need at least version 6.0 of our Zabbix AMI.

This new instance is the target server we will move the data to, and it already ships the latest version installed and updated. Once it is up, copy the source server’s database dump onto it.

Step 4: upgrade the database

To import your current database, upgrade it to the new version and add the required configuration, use the /opt/tools/bin/zabbix-ami-upgrade.sh script. Run it as the ubuntu user, passing the original Zabbix version and the dump file. For example, migrating from Zabbix 4.0:

/opt/tools/bin/zabbix-ami-upgrade.sh 4.0 zabbix_origin_database.sql

The process reports each step it takes, and once it finishes you can carry on working with your new Zabbix server.

Step 5: open the web interface

Once the migration is done you can open the web interface and check the overall state of your server. Note that you must use the same users and passwords you had on the previous server.

Step 6: migrate special configuration and point your clients

With the database working, take into account any other configuration you may have installed on the server, such as:

• Monitoring scripts.
• Extra system packages.
• SSL certificates you installed.

You will also want to move the EIP you had attached to that instance, so your clients need no changes.

If you are not using an EIP, this is a good moment to start: otherwise you have to change the new server’s IP on every client.