Do You Really Need All Those Metrics ?

In Kubernetes, we use Service Monitors to collect metrics from various apps and system components. Sometimes, vmagent scrapes more than we need, which can cause unnecessary load.

When I wrote this post, I was on a hunt for unnecessary metrics to clean up our monitoring. Instead of deleting unwanted Service Monitors, I wanted to stop vmagent from scraping them directly.

Before scaling or tuning performance, it’s often enough to simply stop vmagent from scraping unwanted Service Monitors.

In this post, I’ll show you how to exclude specific Service Monitors from vmagent scraping using label selectors.

In my case, I’m excluding Cilium and Hubble metrics from being collected. Especially Hubble generates a lot of metrics. Let’s take a quick look at what’s there.

kubectl get servicemonitors -A -l app.kubernetes.io/part-of=cilium

NAMESPACE     NAME              AGE
kube-system   cilium-agent      61d
kube-system   cilium-operator   61d
kube-system   hubble            61d

Yes, we can stop scraping these. For that, we will use serviceScrapeSelector in vmagent.

If you only use serviceScrapeSelector, vmagent will scrape Service Monitors only in the same namespace where it runs. To avoid this, you also need to set serviceScrapeNamespaceSelector. By setting its value to empty, you allow vmagent to scrape Service Monitors from all namespaces.

We need to configure vmagent like this

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: victoria-metrics-agent
spec:
...
serviceScrapeNamespaceSelector: {}
serviceScrapeSelector:
  matchExpressions:
  - key: app.kubernetes.io/part-of
    operator: NotIn
    values:
      - cilium
...

Yes, it’s that simple. This way, we no longer collect metrics from these three Service Monitors. You can verify this by checking the targets section in the vmagent UI. Or you can check the sample rate from the dashboard.

For more detailed information, please refer to the documentation.