how AWS autoscaling work with zabbix?
When we want to handle autoscale on Zabbix there are two points that we need to take in.
1. Add a new instance on Zabbix when it appears
We can do that by using Zabbix active agent autoregistration 6.4 version, 6.0 version, 4.0 version. That way, when a new instance apears, Zabbix add them as a zabbix host.
2. Save instance-id on Zabbix inventory
When an active server dissapears (scale in), we receive a message that contains the instance-id of this server, so we need to associate this ami-id on zabbix host inventory, to search and delete them when dissapears.
3. receive AWS autoscale messagess
AWS autoscale send a SNS message when scale in or out to a SNS topic we define. By connect this SNS topic with a SQS queue, we can read this messages from Zabbix and delete hosts that were removed from autoscaling group.
See this simple diagram of how it works:
Configure your client
Two steps are required to config your clients for autoscaling handle on Zabbix:
- Configure your agent config file
- Create a user parameter file for instance_id item
Zabbix agent config file
You need to change (in addition to any other options you wish to configure) six points on your zabbix agent config file (usually locate on /etc/zabbix/zabbix_agentd.conf ):
- Comment option Server
- Add ServerActive=YOUR__ZABBIX__IP__ADDRESS to configure active server
- Comment option Hostname
- Set StartAgents option to 0
- Set option HostnameItem to system.hostname
- Set option HostMetadataItem to system.uname
Whit this configuration, what we are doing is configure our zabbix client as active (Client send info to server, and not receive checks from them). Plus, we define that zabbix use value of “hostname” to send their host name, that way we avoid change it each time server starts and its easier to configure on autoscale. Last, we define that zabbix use value of “uname” command to define host information we send to Zabbix server. As AWS EC2 instances uses special kernel, string “aws” appears on it and server autoregister the instance.
Parameter instance_id
By default, when a new hosts from autoscale are added on Zabbix, this hosts linked with a special template calling AWS EC2 Template, that contains one item calling instance-id associate with “alias” inventory item. For this we need to add in our zabbix client an userparameter who sends the item.
You can do that by create a file on /etc/zabbix/zabbix_agentd.d/template_aws_ec2.conf that contains:
UserParameter=aws.ec2.instance_id,curl -s http://169.254.169.254/latest/meta-data/instance-id
Configure your server
once you have configured your clients, next step is configure your server to delete dissapeared instances. To do so, fast and easiest consists of two steps:
- Send autoscaling messages to a SQS queue
- Configure server to read this SQS queue
Send autoscaling messages to a SQS queue
We need that notificacions sending by autoscaling service were stored in a SQS queue to ensure durability and scale. As AWS autoscaling sends notifications to a SNS topic, we need to:
- create a SNS topic
- create a SQS queue
- configure SNS to send messages to SQS
- create a IAM user who can read SQS messages
As this is a complicated setup, you can use our CloudFormation Stack to create all this elements in the desired region/s After deploy the stack, you will find on Outputs tab four values that you need to note:
- SNSTopicARN: The ARN of SNS topic created
- SQSQueueURL: The URL of SQS queue
- SQSReadIAMUserAccessKey and SQSReadIAMUserSecretKey: User and password for IAM user
you need last two to configure SQS reading. To do so you need to edit file /etc/zabbix/zabbix_ami.yml, on code looks like that:
sqs:
delay_seconds: 60
default_region: us-east-1
queues:
- url:
access_key_id:
secret_access_key:
region:
Per each sqs with autoscaling notifications, you need to add an array element similar to the existing one, with cloudformation output’s data.
You can get more info about this config file on section AMI config file