I have a 4 nodes clusters:
- MonitoringCenter, hosting
- a Grafana, connected to all Prometheus and Loki instances + local AlertManager
- a Prometheus scraping local NodeExporter/AlertManager/Loki/Promtail (one job per target), with rules
- a Loki, a Promtail, a NodeExporter, a AlertManger configured with Slack
- Production/Staging/Dev, hosting
- our Application
- a Redis, a PostGreSQL
- a Prometheus scraping local NodeExporter/AlertManager/Loki/Promtail/Redis/PostGreSQL (one job per target), with rules
- a Loki, a Promtail, a NodeExporter, a AlertManger configured with Slack
- a Traefik to expose Prometheus/Loki
I was wondering having 1 Prometheus/Loki/AlertManager per node is the right way to do it, or if I should only have one for all the infrastructure, having multiple targets per Prometheus job?