The Prometheus Operator

The main purpose of this operator is to simplify and automate the configuration and management of the Prometheus monitoring stack running on a Kubernetes cluster. Essentially it is a custom controller that monitors the new object types introduced through the following CRDs:

• Prometheus: defines the desired Prometheus deployments as a StatefulSet
• Alertmanager: defines a desired Alertmanager deployment
• ServiceMonitor: declaratively specifies how groups of Kubernetes services should be monitored
• PodMonitor: declaratively specifies how groups of pods should be monitored
• Probe: declaratively specifies how groups of ingresses or static targets should be monitored
• PrometheusRule: defines a desired set of Prometheus alerting and/or recording rules
• AlertmanagerConfig: declaratively specifies subsections of the Alertmanager configuration

#Adding additional scrape configuration - prometheus-additional-job.yaml
» cat prometheus-additional-job.yaml                                                                             (microk8s/default)
- job_name: "my_server"
  scrape_interval: 30s
  static_configs:     
    - targets: ['192.168.1.11:9100']

» kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional-job.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml

» more additional-scrape-configs.yaml                                                                            (microk8s/default)
apiVersion: v1
data:
  prometheus-additional-job.yaml: LSBqb2JfbmFtZTogImJzcGVnYXN1cyIKICBzY3JhcGVfaW50ZXJ2YWw6IDMwcwogIHN0YXRpY19jb25maWdzOiAgICAgCiAgICAtIHRhcmdldHM6IFsnMTkyLjE2OC4xLjExOjkxMDA
nXQ==
kind: Secret
metadata:
  creationTimestamp: null
  name: additional-scrape-configs

» kubectl -n observability apply -f additional-scrape-configs.yaml
» kubectl -n observability get secrets | grep scrape                                                             
additional-scrape-configs                     Opaque               1      71m

Now lets edit the CR to add the new configs:

» kubectl -n observability edit prom  

# Add additionalScrapeConfigs:
spec:
  serviceAccountName: prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  podMonitorSelector: {}
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional-job.yaml

Mostly sourced from here:

https://blog.container-solutions.com/prometheus-operator-beginners-guide