Sensu is a flexible monitoring framework that supports both agent-based and agentless monitoring. This guide covers Sensu Go (the modern version) configuration. Sensu uses a check-based architecture with a powerful event pipeline.
| File/Directory | Path | Purpose |
|---|---|---|
| Backend config | /etc/sensu/backend.yml |
Sensu backend server configuration |
| Agent config | /etc/sensu/agent.yml |
Sensu agent configuration |
| CLI config | /etc/sensu/cli.yml |
sensuctl CLI configuration |
| Asset definitions | /etc/sensu/assets/ |
Runtime asset definitions |
| Check definitions | /etc/sensu/checks/ |
Check configuration files |
| Handler definitions | /etc/sensu/handlers/ |
Event handler configurations |
| Filter definitions | /etc/sensu/filters/ |
Event filter configurations |
| Mutator definitions | /etc/sensu/mutators/ |
Event mutator configurations |
| TLS certificates | /etc/sensu/ssl/ |
TLS/SSL certificates |
| Logs | /var/log/sensu/ |
Sensu log files |
# /etc/sensu/backend.yml
# Backend identification
backend-name: sensu-backend-prod
# API and Dashboard configuration
api-host: 0.0.0.0
api-port: 8080
dashboard-host: 0.0.0.0
dashboard-port: 3000
# Agent communication
agent-host: 0.0.0.0
agent-port: 8081
# etcd configuration (embedded)
etcd-advertise-url: http://localhost:2380
etcd-client-url: http://localhost:2379
etcd-peer-url: http://localhost:2380
etcd-initial-cluster: sensu-backend-prod=http://localhost:2380
etcd-initial-cluster-state: new
etcd-initial-cluster-token: ""
etcd-name: sensu-backend-prod
# Datastore configuration
state-dir: /var/lib/sensu
cache-dir: /var/cache/sensu/sensu-backend
# Logging
log-level: info
log-format: json
# TLS configuration
cert-file: /etc/sensu/ssl/backend.pem
key-file: /etc/sensu/ssl/backend-key.pem
trusted-ca-file: /etc/sensu/ssl/ca.pem
# JWT authentication
jwt-private-key-file: /etc/sensu/jwt/jwt_private_key.pem
jwt-public-key-file: /etc/sensu/jwt/jwt_public_key.pem
# Clustering (for HA setup)
cluster:
- name: sensu-backend-1
advertise-url: http://192.168.1.10:2380
- name: sensu-backend-2
advertise-url: http://192.168.1.11:2380
- name: sensu-backend-3
advertise-url: http://192.168.1.12:2380
# Performance tuning
events-burst-limit: 100
events-rate-limit: 500
# /etc/sensu/agent.yml
# Agent identification
name: web-server-01
namespace: default
subscriptions:
- linux
- webserver
- production
# Backend connection
backend-url:
- wss://sensu-backend.example.com:8081
- wss://sensu-backend-backup.example.com:8081
# Agent settings
api-host: 127.0.0.1
api-port: 3031
socket-host: 127.0.0.1
socket-port: 3032
# Logging
log-level: info
log-format: json
# TLS configuration
cert-file: /etc/sensu/ssl/agent.pem
key-file: /etc/sensu/ssl/agent-key.pem
trusted-ca-file: /etc/sensu/ssl/ca.pem
# Keepalive configuration
keepalive-interval: 20
keepalive-warning-timeout: 120
keepalive-critical-timeout: 180
# Deregistration
deregister: false
deregistration-handler: deregistration
# Labels and annotations
labels:
environment: production
team: platform
location: dc1
rack: a12
annotations:
slack-channel: "#alerts-production"
pagerduty-service: "platform-oncall"
# Statsd configuration (for metrics)
statsd-disable: false
statsd-event-host: 127.0.0.1
statsd-event-port: 8125
statsd-metric-handlers:
- statsd-metric-handler
# /etc/sensu/cli.yml
api-url: https://sensu-backend.example.com:8080
trusted-ca-file: /etc/sensu/ssl/ca.pem
username: admin
password: P@ssw0rd123!
format: tabular
namespace: default
Checks define what Sensu monitors. They can be ad-hoc, scheduled, or proxy checks.
# /etc/sensu/checks/check-cpu.yml
type: CheckConfig
api_version: core/v2
metadata:
name: check-cpu
namespace: default
labels:
team: platform
spec:
command: check-cpu.sh -w 75 -c 90
interval: 60
timeout: 10
handlers:
- slack
- email
subscriptions:
- linux
publish: true
stdin: false
check_hooks:
- type: pipe
command: /usr/local/bin/check-hook.sh
proxy_requests:
entity_attributes:
- "entity.entity_class == 'proxy'"
- "entity.labels.os == 'linux'"
output_metric_format: graphite_plaintext
output_metric_handlers:
- influxdb
round_robin: false
ttl: 0
low_flap_threshold: 5
high_flap_threshold: 20
# /etc/sensu/checks/check-http.yml
type: CheckConfig
api_version: core/v2
metadata:
name: check-http
namespace: default
spec:
command: check-http.rb -u https://example.com -r '200'
interval: 30
timeout: 15
handlers:
- pagerduty
- slack
subscriptions:
- webserver
publish: true
check_hooks:
- type: pipe
command: /opt/sensu/hooks/log-response.sh
# /etc/sensu/checks/check-disk.yml
type: CheckConfig
api_version: core/v2
metadata:
name: check-disk
namespace: default
spec:
command: check-disk.rb -w 80 -c 90 -p / -p /var -p /home
interval: 300
timeout: 30
handlers:
- email
subscriptions:
- linux
- windows
publish: true
output_metric_format: graphite_plaintext
output_metric_handlers:
- graphite
Proxy checks allow monitoring entities that cannot run the Sensu agent:
# /etc/sensu/checks/check-network-device.yml
type: CheckConfig
api_version: core/v2
metadata:
name: check-snmp-interface
namespace: default
spec:
command: check-snmp.rb -H {{ .labels.hostname }} -C {{ .labels.snmp_community }} -O ifOperStatus
interval: 60
timeout: 30
handlers:
- slack
subscriptions:
- network-monitor
publish: true
proxy_requests:
entity_attributes:
- "entity.entity_class == 'proxy'"
- "entity.labels.device_type == 'network'"
splay: true
splay_coverage: 90
Assets are reusable packages of Sensu plugins:
# /etc/sensu/assets/sensu-plugins-disk-checks.yml
type: Asset
api_version: core/v2
metadata:
name: sensu-plugins-disk-checks
namespace: default
spec:
url: https://github.com/sensu-plugins/sensu-plugins-disk-checks/releases/download/5.0.0/sensu-plugins-disk-checks-5.0.0-linux-amd64.tar.gz
sha512: abc123...
filters:
- "entity.system.os == 'linux'"
- "entity.system.arch == 'amd64'"
builds:
- url: https://github.com/sensu-plugins/sensu-plugins-disk-checks/releases/download/5.0.0/sensu-plugins-disk-checks-5.0.0-linux-amd64.tar.gz
sha512: abc123...
filters:
- "entity.system.os == 'linux'"
- "entity.system.arch == 'amd64'"
- url: https://github.com/sensu-plugins/sensu-plugins-disk-checks/releases/download/5.0.0/sensu-plugins-disk-checks-5.0.0-windows-amd64.tar.gz
sha512: def456...
filters:
- "entity.system.os == 'windows'"
Handlers define what happens when an event occurs:
# /etc/sensu/handlers/handler-email.yml
type: Handler
api_version: core/v2
metadata:
name: email
namespace: default
spec:
type: pipe
command: sensu-emailer -t smtp.example.com -p 587 -u alerts@example.com -P password -f alerts@example.com -T {{ .labels.team_email }}
filters:
- is_incident
- not_silenced
timeout: 10
env_vars:
- SENSU_EMAILER_SMTP_HOST=smtp.example.com
- SENSU_EMAILER_SMTP_PORT=587
- SENSU_EMAILER_SMTP_USER=alerts@example.com
- SENSU_EMAILER_SMTP_PASSWORD=password
# /etc/sensu/handlers/handler-slack.yml
type: Handler
api_version: core/v2
metadata:
name: slack
namespace: default
spec:
type: pipe
command: sensu-slack-handler --webhook-url https://hooks.slack.com/services/XXX/YYY/ZZZ --channel #alerts --username Sensu
filters:
- is_incident
- not_silenced
timeout: 10
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ
- SLACK_CHANNEL=#alerts
- SLACK_USERNAME=Sensu
# /etc/sensu/handlers/handler-pagerduty.yml
type: Handler
api_version: core/v2
metadata:
name: pagerduty
namespace: default
spec:
type: pipe
command: sensu-pagerduty-handler --token YOUR_PAGERDUTY_TOKEN
filters:
- is_incident
- is_critical
timeout: 30
env_vars:
- PAGERDUTY_TOKEN=YOUR_PAGERDUTY_TOKEN
# /etc/sensu/handlers/handler-webhook.yml
type: Handler
api_version: core/v2
metadata:
name: webhook
namespace: default
spec:
type: pipe
command: sensu-webhook-handler --endpoint https://api.example.com/alerts --method POST
filters:
- is_incident
timeout: 15
env_vars:
- WEBHOOK_ENDPOINT=https://api.example.com/alerts
- WEBHOOK_METHOD=POST
- WEBHOOK_AUTHORIZATION=Bearer YOUR_TOKEN
# /etc/sensu/handlers/handler-influxdb.yml
type: Handler
api_version: core/v2
metadata:
name: influxdb
namespace: default
spec:
type: pipe
command: sensu-influxdb-handler --scheme http --host influxdb.example.com --port 8086 --db sensu --user sensu --password password
event_types:
- metric
timeout: 10
filters:
- has_metrics
Filters determine which events are processed:
# /etc/sensu/filters/filter-is-incident.yml
type: EventFilter
api_version: core/v2
metadata:
name: is_incident
namespace: default
spec:
action: allow
when: event.check.status == 1 || event.check.status == 2
# /etc/sensu/filters/filter-not-silenced.yml
type: EventFilter
api_version: core/v2
metadata:
name: not_silenced
namespace: default
spec:
action: allow
when: event.check.silenced == false
# /etc/sensu/filters/filter-is-critical.yml
type: EventFilter
api_version: core/v2
metadata:
name: is_critical
namespace: default
spec:
action: allow
when: event.check.status == 2
# /etc/sensu/filters/filter-business-hours.yml
type: EventFilter
api_version: core/v2
metadata:
name: business_hours
namespace: default
spec:
action: allow
when: hour() >= 8 && hour() <= 18 && weekday() >= 1 && weekday() <= 5
# /etc/sensu/filters/filter-has-metrics.yml
type: EventFilter
api_version: core/v2
metadata:
name: has_metrics
namespace: default
spec:
action: allow
when: event.metrics != null && event.metrics.points != null && length(event.metrics.points) > 0
Mutators transform event data before sending to handlers:
# /etc/sensu/mutators/mutator-json.yml
type: Mutator
api_version: core/v2
metadata:
name: json
namespace: default
spec:
command: sensu-json-mutator
timeout: 10
# /etc/sensu/mutators/mutator-influxdb-line-protocol.yml
type: Mutator
api_version: core/v2
metadata:
name: influxdb-line-protocol
namespace: default
spec:
command: sensu-influxdb-line-protocol-mutator
timeout: 10
# Route alerts based on severity
type: Handler
api_version: core/v2
metadata:
name: route-by-severity
namespace: default
spec:
type: set
handlers:
- slack-warning
- pagerduty-critical
filters:
- is_incident
# Warning alerts to Slack
type: Handler
api_version: core/v2
metadata:
name: slack-warning
namespace: default
spec:
type: pipe
command: sensu-slack-handler --webhook-url https://hooks.slack.com/services/XXX/YYY/ZZZ --channel #warnings
filters:
- is_warning
# Critical alerts to PagerDuty
type: Handler
api_version: core/v2
metadata:
name: pagerduty-critical
namespace: default
spec:
type: pipe
command: sensu-pagerduty-handler --token YOUR_PAGERDUTY_TOKEN
filters:
- is_critical
# Escalation handler
type: Handler
api_version: core/v2
metadata:
name: escalation
namespace: default
spec:
type: set
handlers:
- slack-initial
- pagerduty-escalation
timeout: 300 # 5 minutes before escalation
# Validate backend configuration
sensu-backend config validate /etc/sensu/backend.yml
# Validate agent configuration
sensu-agent config validate /etc/sensu/agent.yml
# Validate check configuration
sensuctl check list
# Validate asset configuration
sensuctl asset list
# Test check execution locally
/usr/bin/check-cpu.sh -w 75 -c 90
# Restart Sensu backend
sudo systemctl restart sensu-backend
# Restart Sensu agent
sudo systemctl restart sensu-agent
# Check service status
sudo systemctl status sensu-backend
sudo systemctl status sensu-agent
# View logs
sudo journalctl -u sensu-backend -f
sudo journalctl -u sensu-agent -f
# Configure sensuctl
sensuctl configure -n \
--url https://sensu-backend.example.com:8080 \
--username admin \
--password P@ssw0rd123! \
--namespace default
# Create resources from YAML files
sensuctl create -f /etc/sensu/checks/
sensuctl create -f /etc/sensu/handlers/
sensuctl create -f /etc/sensu/filters/
sensuctl create -f /etc/sensu/assets/
# Bulk create all resources
sensuctl create -f /etc/sensu/
# Verify resources
sensuctl check list
sensuctl handler list
sensuctl asset list
# Check agent status
sensuctl entity list
# Check specific entity
sensuctl entity info web-server-01
# View agent events
sensuctl event list --entity web-server-01
# Execute check manually
sensuctl check execute check-cpu --entity web-server-01
# View check history
sensuctl check history check-cpu
# View check output
sensuctl event info web-server-01 check-cpu --format json
# Test handler with sample event
echo '{"check": {"name": "test", "status": 1}}' | sensu-slack-handler --webhook-url https://hooks.slack.com/services/XXX/YYY/ZZZ
# View handler execution logs
sudo journalctl -u sensu-backend | grep handler
# Check metrics are being received
sensuctl event list --format json | jq '.[] | select(.metrics != null)'
# Query InfluxDB for Sensu metrics
influx -execute 'SELECT * FROM sensu.autogen.check_cpu LIMIT 10'
Squeezing every bit of performance from your Sensu installation? Our experts help with:
Optimize your setup: office@linux-server-admin.com | Contact Us