When to use Thanos/Cortex over Vanilla Prometheus?

Question

Generally when I look around or try to understand the HA setup for Prometheus the most common search results are Cortex & Thanos. I've also seen M3 and Victoriametrics on the list.

What I understand is that Cortex and Thanos support is mainly for HA, Multitenancy and Long term storage.

So what situations would be best to go with the vanilla setup of Prometheus with the HA setup (of alertmanager) recommended by them, which also support certain amounts on Long Term.

Basically, when would these solutions be the best option over vanilla Alertmanager HA and when would they be an overkill?

Would the basic setup be fine if we don't need to consider multitenancy and storage (30 days or less)

Answer 1

If you're happy with your Prometheus right now (durability, query speed, resource usage), then don't bother with switching to anything else. Just run two Proms as HA pair and you're gold.

Thanos/Cortex/Mimir are usually used when you can't scale Prometheus vertically anymore, or if you want infinite retention while storing data in S3. Switching to Thanos/Cortex/Mimir will bring much more complexity into your life. Not because they're bad products, but because they're distributed systems. Distributed systems are always not easy.

VictoriaMetrics is a good alternative if you want to reduce resource usage and get clustering out-of-the box. I'm biased on this option because I'm one of maintainers of VictoriaMetrics project. But if you're curious, read the following blogpost.

M3 seems pretty much a "dead" project to me. Check the rate of commits to their repo and communication in their slack. It looks abandoned.