Prometheus is a central metrics collection and storage system that can expose sensitive infrastructure metadata if not properly secured. This guide covers security measures for production Prometheus deployments.
Prometheus operates as both a metrics collector (scraping targets) and an HTTP server (serving queries and UI). Key security concerns include:
Configure firewall rules to restrict Prometheus access:
# Allow Prometheus server port (9090) only from trusted networks
ufw allow from 10.0.0.0/8 to any port 9090 proto tcp
ufw allow from 192.168.1.0/24 to any port 9090 proto tcp
# Block external access to Prometheus
ufw deny from any to any port 9090 proto tcp
Bind Prometheus to specific interfaces:
# prometheus.yml
global:
external_labels:
environment: 'production'
# Start Prometheus with specific address binding
prometheus --web.listen-address="127.0.0.1:9090"
For containerized deployments:
# docker-compose.yml
services:
prometheus:
ports:
- "127.0.0.1:9090:9090"
Place Prometheus behind an authenticated reverse proxy:
Nginx with Basic Auth:
location / {
auth_basic "Prometheus";
auth_basic_user_file /etc/nginx/.htpasswd;
proxy_pass http://localhost:9090;
}
Nginx with OAuth2:
location / {
auth_request /oauth2/auth;
proxy_pass http://localhost:9090;
}
While Prometheus lacks native RBAC, implement access control at the proxy level:
/adminIntegrate with enterprise identity providers:
Enforce multi-factor authentication for:
Enable TLS for Prometheus server:
prometheus \
--web.tls-cert=/etc/prometheus/certs/server.crt \
--web.tls-key=/etc/prometheus/certs/server.key \
--web.tls-client-ca=/etc/prometheus/certs/ca.crt
For high-security environments, require client certificates:
prometheus \
--web.tls-cert=/etc/prometheus/certs/server.crt \
--web.tls-key=/etc/prometheus/certs/server.key \
--web.tls-client-ca=/etc/prometheus/certs/ca.crt \
--web.tls-client-auth=RequireAndVerifyClientCert
Configure TLS for scraping targets:
# prometheus.yml
scrape_configs:
- job_name: 'secure-targets'
scheme: https
tls_config:
ca_file: /etc/prometheus/certs/ca.crt
cert_file: /etc/prometheus/certs/client.crt
key_file: /etc/prometheus/certs/client.key
insecure_skip_verify: false
static_configs:
- targets: ['target1:9100', 'target2:9100']
remote_write:
- url: "https://remote-storage:9201/write"
tls_config:
ca_file: /etc/prometheus/certs/ca.crt
cert_file: /etc/prometheus/certs/client.crt
key_file: /etc/prometheus/certs/client.key
Restrict access to sensitive API endpoints:
| Endpoint | Risk Level | Access Control |
|---|---|---|
/api/v1/query |
Medium | Authenticated users |
/api/v1/query_range |
Medium | Authenticated users |
/api/v1/series |
High | Admin only |
/api/v1/labels |
Low | Authenticated users |
/api/v1/targets |
High | Admin only |
/api/v1/rules |
Medium | Authenticated users |
/api/v1/admin/* |
Critical | Admin only |
--web.enable-admin-api only when neededprometheus --web.enable-admin-api=false --web.enable-lifecycle=false
Secure federation endpoints:
# Limit federation to specific match patterns
scrape_configs:
- job_name: 'federate'
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
scheme: https
tls_config:
ca_file: /etc/prometheus/certs/ca.crt
static_configs:
- targets: ['federated-prometheus:9090']
Prometheus TSDB data should be encrypted:
Never store secrets in Prometheus configuration:
# Use environment variable substitution
remote_write:
- url: "https://remote-storage:9201/write"
basic_auth:
username: ${PROMETHEUS_REMOTE_USER}
password_file: /etc/prometheus/secrets/remote_password
Remove sensitive labels before storage:
# prometheus.yml
metric_relabel_configs:
- action: labeldrop
regex: 'password|secret|token|api_key'
- action: labelkeep
regex: 'job|instance|__name__|env|service'
Limit data exposure with retention policies:
prometheus --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=10GB
Enable query logging for audit purposes:
prometheus --web.enable-admin-api --log.level=debug
Monitor query patterns for anomalies:
# Alert on excessive query rates
- alert: HighQueryRate
expr: rate(prometheus_http_requests_total[5m]) > 100
for: 5m
labels:
severity: warning
annotations:
summary: "High query rate detected"
Configure reverse proxy logging:
access_log /var/log/nginx/prometheus_access.log combined;
log_format combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
# Alert on authentication failures
- alert: PrometheusAuthFailures
expr: increase(nginx_http_requests_total{status="401"}[5m]) > 10
for: 2m
labels:
severity: critical
# Alert on configuration changes
- alert: PrometheusConfigReloadFailed
expr: prometheus_config_last_reload_successful == 0
for: 1m
labels:
severity: warning