What this module does #

An instance has plenty to report: fail2ban has banned an IP, aide has seen a system file change, last night's backup failed, the root disk is at 95 %. If every module reports its own way —an email here, a log line there— nobody hears anything, and with ten instances the problem is ten times bigger.

The global-alerts module is imaxe's central alert bus: a single command every notice goes through and a single destination they all reach, an Amazon SNS topic shared by your whole fleet. From there SNS fans out however you like: email, SMS, a Lambda function, an SQS queue, your on-call system. It publishes with the instance IAM role (the sns:Publish permission), so there is no credential to store. And it filters the noise before sending: a severity threshold drops anything below the level you care about, and a deduplication window keeps the same alert from waking you forty times.

If SNS is unavailable —network down, role without permissions yet, region unreachable— the alert is not lost: it is queued on disk and a systemd timer retries it every 5 minutes.

Before you start

You need the ARN of an SNS topic (arn:aws:sns:region:account:topic) and the instance to be able to publish to it. If you launched the AMI from the launcher, the CloudFormation template already creates the topic, subscribes your email, creates the IAM role with sns:Publish and passes the ARN to the instance as a tag: the module configures itself and there is nothing to do here.

Quick startstep-by-step tasks

Common tasks #

Pick what you want to do. Every recipe comes with the command already written — swap the ARN and the text for yours, and hit Copy.

Set the SNS topic Send a test alert Send an alert from a script Check the bus status Turn down the noise See the queue and retry Mute the module

1

Set the SNS topic

Tell the instance where to publish its notices.

Connect to your server over SSH as the ubuntu user.

Point the module at your topic ARN. The region is derived from the ARN itself, so you normally do not need to pass it:

terminal

$ sudo imaxe global-alerts configure \
    --topic-arn arn:aws:sns:eu-west-1:123456789012:imaxe-alerts

Several instances publishing to the same topic? Give each one a recognisable source label with --source (the hostname is used by default):

terminal

$ sudo imaxe global-alerts configure --source web-production-1

The configuration lands in /etc/imaxe/global-alerts.yml and sending is enabled. Carry on with the Send a test alert recipe.

2

Send a test alert

Check that the IAM role really publishes before trusting the channel.

Publish a test alert to the configured topic:

terminal

$ sudo imaxe global-alerts test

The test skips the threshold and the deduplication —it always goes out— and, if something fails, it hands you the real AWS error instead of queueing silently. Check the inbox of the address subscribed to the topic (and the spam folder).

If you see the MessageId and the notice arrives, the channel works. If you get AuthorizationError, the instance is missing the sns:Publish permission on that topic.

3

Send an alert from a script

The same channel the modules use, available for your own things.

An alert with its severity and its source:

terminal

$ sudo imaxe global-alerts send --severity critical \
    --source backup --subject "backup failed" \
    "the nightly database backup finished with an error"

If another command produces the text, pipe it through stdin using - as the message:

terminal

$ df -h / | sudo imaxe global-alerts send --severity warning -

In something that runs every few minutes, give it a stable deduplication key: within the configured window only the first one goes out:

terminal

$ sudo imaxe global-alerts send --severity warning \
    --dedup-key root-disk-full "root disk at 95%"

The alert travels to the topic as JSON, with the instance, the region and the timestamp already included. Without --dedup-key the module derives one from source + severity + subject.

4

Check the bus status

Topic, region, AWS CLI and pending alerts, at a glance.

A summary of the current state:

terminal

$ sudo imaxe global-alerts status

To check which configuration is really in charge —including what arrives via instance tags, which take priority over the file—:

terminal

$ sudo imaxe global-alerts show
$ sudo imaxe global-alerts status --json

You will know whether a topic is configured, which region it publishes to and how many alerts are waiting in the queue. With --json, ready for a dashboard or a script.

5

Turn down the noise

Raise the severity threshold and widen the deduplication window.

If you only want to hear about what matters, drop everything below warning:

terminal

$ sudo imaxe global-alerts configure --min-severity warning

The deduplication window has no flag: it is set in the configuration file. Raise it if the same alert repeats a lot:

/etc/imaxe/global-alerts.yml

dedup_window: 1h   # 30s, 5m, 1h… (5m by default)

Alerts below the threshold are dropped before they go out (they are not queued) and repeats within the window are suppressed. test still publishes every time, so you never lose the way to check the channel.

6

See the queue and retry

What was left pending when SNS did not answer, and how to force the send.

Look at what is pending and the keys sent most recently:

terminal

$ sudo imaxe global-alerts history

A systemd timer already retries every 5 minutes, but you can force it after fixing the permission or the network:

terminal

$ sudo imaxe global-alerts flush
$ systemctl status imaxe-global-alerts-flush.timer

You will see how many went out and how many are still waiting. If the first one fails again, the pass stops there and leaves the rest for the next attempt: nothing is thrown away.

7

Mute the module

Stop publishing without losing the configuration.

Disable alert sending and remove the retry timer:

terminal

$ sudo imaxe global-alerts remove

The topic, the region and the rest of the settings stay in the file: to switch it back on a configure is enough, which re-enables the module.

The instance stops publishing. Later send calls do not fail: they warn on stderr that the module is disabled and exit with code 0.

No IAM role, no alerts

The module publishes with the credentials of the instance role, not with stored keys. If the role does not allow sns:Publish on that topic, alerts will queue up one after another and never arrive. An imaxe global-alerts test tells you right away, with the error exactly as AWS returns it.

CLI referencecommands, flags and files

Synopsis #

usage

imaxe global-alerts <subcommand> [--topic-arn ARN] [--severity LEVEL] [flags]

Every subcommand needs root privileges (use sudo) because they write to /etc/imaxe/, keep state in /var/lib/imaxe/ and manage a systemd unit. There are no secrets to handle: publishing goes through the instance IAM role. Add --json to status, show, history or flush for machine-readable output.

Subcommands #

Subcommand	What it does	Relevant flags
status	State: configured topic, effective region, AWS CLI available and alerts in the queue.	--json
configure	Sets the SNS topic and the sending options. Re-enables the module if it was disabled.	--topic-arn, --region, --source, --min-severity
send	Publishes an alert. This is the channel the operator and the other modules use.	--severity, --source, --subject, --dedup-key
test	Publishes a test alert skipping threshold and deduplication, and reports the real error if it fails.	--severity
show	Shows the effective configuration (file + instance tags already applied).	--json
history	Alerts pending in the queue and deduplication keys sent recently.	--json
flush	Retries the queued alerts. The systemd timer runs this too.	--json
remove	Disables sending and removes the timer. Keeps the configuration.	—

Arguments and flags #

Flag	Type	Default	Description
--topic-arn req.	string	—	ARN of the destination SNS topic (`arn:aws:sns:region:account:topic`). Without it the module cannot publish.
--region	string	from the ARN	AWS region. If omitted, it is derived from the topic ARN; failing that, from IMDS or `AWS_REGION`.
--source	string	hostname	Source label. In `configure`, the instance's; in `send`, that particular alert's (e.g. the module raising it).
--min-severity	string	info	Threshold: drops alerts below this level. Values: `info`, `warning`, `critical`.
--severity	string	info	In `send`/`test`: the level of this alert. The short forms `warn` and `crit` are accepted.
--subject	string	from the message	Short subject. If omitted, it is derived from the message itself.
--dedup-key	string	derived	Deduplication key: suppresses repeats within `dedup_window`. By default it is computed from source + severity + subject.
<message> req.	positional	—	In `send`: the alert text, or `-` to read it from `stdin`.
--json	bool	false	In `status`, `show`, `history` and `flush`, emits the result as JSON on `stdout`.

Dropping is not failing

When an alert is dropped —module disabled, severity below the threshold, or a duplicate within the window— send explains it on stderr and exits with code 0. That way the script that raised it does not break because of a filter you configured yourself.

Configuration through instance tags #

Every deployment needs to point at its topic, and rebuilding the AMI for that would make no sense. So besides the file, the module reads the instance tags prefixed with imaxe.global-alerts. over IMDSv2: if they exist, they win over the YAML. That is what the launcher's CloudFormation template does, and it also requires MetadataOptions.InstanceMetadataTags: enabled so they can be read.

Tag	Equivalent to	Values
imaxe.global-alerts.topic_arn	topic_arn	ARN of the destination SNS topic.
imaxe.global-alerts.region	region	AWS region; empty = derived from the ARN or from IMDS.
imaxe.global-alerts.source	source	Source label; empty = hostname.
imaxe.global-alerts.min_severity	min_severity	`info` · `warning` · `critical`
imaxe.global-alerts.dedup_window	dedup_window	Duration: `30s`, `5m`, `1h`…
imaxe.global-alerts.enabled	enabled	`true`/`false` (also `1`/`0`, `yes`/`no`, `on`/`off`).

Outside AWS, or with IMDS blocked, the read fails in milliseconds and the module carries on with whatever the file says. To see what actually ended up active, imaxe global-alerts show.

Files and paths #

Path	Contents
/etc/imaxe/global-alerts.yml	Module configuration: topic, region, source, threshold and deduplication window.
/var/lib/imaxe/state/global-alerts/spool/	Queue of pending alerts, one per `.json` file, in chronological order.
/var/lib/imaxe/state/global-alerts/sent.json	Record of deduplication keys with the time they were last sent.
/etc/systemd/system/imaxe-global-alerts-flush.timer	Retry timer: starts 2 min after boot and repeats every 5 min.

Example global-alerts.yml:

/etc/imaxe/global-alerts.yml

enabled: true
topic_arn: arn:aws:sns:eu-west-1:123456789012:imaxe-alerts
region: ""            # empty = derived from the ARN or from IMDS
source: ""            # empty = instance hostname
min_severity: info
dedup_window: 5m

State (queue and deduplication record) lives in /var/lib/imaxe/ and not in /etc/ on purpose: it is state, not configuration. Both paths can be moved with the IMAXE_CONFIG_DIR and IMAXE_STATE_DIR environment variables.

Alert format #

The body of the SNS message is versioned JSON (schema: 1), so the subscriber can handle it with a Lambda or a queue as well as read it by email:

message body

{
  "schema": 1,
  "severity": "critical",
  "source": "backup",
  "subject": "backup failed",
  "message": "the nightly database backup finished with an error",
  "host": "web-production-1",
  "instance_id": "i-0abc123def4567890",
  "region": "eu-west-1",
  "ts": "2026-07-25T03:14:07Z",
  "dedup_key": "9f2c1b7e44a0d513"
}

The subject of the SNS message is composed as [imaxe][severity] host: subject, sanitised to printable ASCII and trimmed to 100 characters, which is the limit SNS imposes.

Exit codes and logs #

Every run returns a code you can check with echo $? — handy for chaining in scripts:

0OKOperation completed. Also when the alert is dropped or queued on purpose.

1ERRGeneric error: the test could not be published, or it could neither be published nor queued.

2USAGEInvalid arguments: unknown flag, missing message or invalid severity.

64NO TOPICNo SNS topic configured, neither by file nor by tag.

The module also answers the imaxe health check: if it is enabled but has no topic, health fails, so an imaxe health gives it away before the first alert is even needed.

terminal

$ sudo imaxe global-alerts test; echo "exit: $?"
$ journalctl -u imaxe-global-alerts-flush.service -n 50

Troubleshooting #

Symptom	Likely cause	Fix
You get NO TOPIC (code 64)	Neither the file nor the tags carry a topic ARN.	Run `configure --topic-arn …`, or check that the instance has the `imaxe.global-alerts.topic_arn` tag and that tags are enabled in the metadata.
«SNS unavailable; alert queued» over and over	The instance role is missing `sns:Publish` on that topic, or the ARN belongs to another account or region.	Run `test` to see the exact AWS error, fix the role policy and then `flush`.
«aws CLI available: false» in status	The instance does not have the AWS CLI installed; the module publishes through it.	Install the AWS CLI. imaxe AMIs ship with it; on your own host you have to add it.
You configure something and show keeps showing another value	An `imaxe.global-alerts.*` tag is overriding the file: it takes priority by design.	Change the instance tag (or the CloudFormation stack parameter) instead of the YAML.
Only the first of several identical alerts arrives	The deduplication window is suppressing them.	That is expected. Lower `dedup_window`, or use different `--dedup-key` values if they really are different events.
Nothing arrives and there are no errors	The module is disabled, or the severity is below `min_severity`.	`show` gives you `enabled` and the threshold; `configure` re-enables sending.
Alerts publish but never reach you by email	The SNS topic subscription has not been confirmed.	Look for the AWS confirmation email (check spam) and accept the subscription in the SNS console.

Stuck with the Alerts module?

Write to us with the output of «imaxe <module> status --json» and we'll get back to you fast.

Contact support

A single place to listen to your whole fleet

What this module does #

Common tasks #

Set the SNS topic

Send a test alert

Send an alert from a script

Check the bus status

Turn down the noise

See the queue and retry

Mute the module

Synopsis #

Subcommands #

Arguments and flags #

Configuration through instance tags #

Files and paths #

Alert format #

Exit codes and logs #

Troubleshooting #

Stuck with the Alerts module?