Setup Prometheus and Grafana On Kubernetes And Make Persistent Using ConfigMap

Harshet Jain

Aug 13, 2020·6 min read

Hola everyone,

Today, I would like to show you some great monitoring tools Prometheus and Grafana. As when I talk about the DevOps world these two tools are very important for that because monitoring is also important for the DevOps world. Monitoring helps us to see the faults, time latencies, Centralised control, Data visualisation and many more…

Before, I give you today’s agenda. I want to tell you something about Prometheus and Grafana.

So, What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. Prometheus collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. For more info you can click here.

What is Grafana?

Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources. It is expandable through a plug-in system. End users can create complex monitoring dashboards using interactive query builders. Grafana collect metrics from Prometheus and give you a interactive interface. For more info you can click here.

Agenda!!!

Deploy Prometheus and Grafana on the top of Kubernetes and create a Persistent system so If accidentally any pod is deleted so it creates a new pod within a second and also with the same data.

Steps Involved

• Create a server of Prometheus and Grafana on the top of Kubernetes.
• Use a node exporter for collecting the metrics of system.
• Create a persistent storage for Prometheus and Grafana.

Let’s Go

Step 1:

For creating a Prometheus and Grafana we need an Image or a Software. Here, I am using an Image.

First, setup the Kubernetes: I am using minikube(single-node cluster) for this

Then, configure the Prometheus and Grafana. I am creating a deployment of image vimal13/prometheus and vimal13/grafana. Use command:

kubectl create deployment (name) --image=vimal13/prometheus
kubectl create deployment (name) --image=vimal13/grafana

After that you need to expose the services of Prometheus and Grafana. Use command:

kubectl expose deployment (deploymentname) --type=NodePort --port=9090kubectl expose deployment (deploymentname) --type=NodePort --port=3000

Now, your Prometheus and Grafana are ready to work.

Step 2:

For monitoring a system we need an agent in that system which collect metrics. In Prometheus, agent name is node exporter. I am using RHEL8 and download a compatible version of RHEL8.

Now, you need to extract that. For this use command:

tar -xzf node_exporter-1.0.0.linux-amd64.tar.gz

After that, you see a folder is created of the same name of software. As you see in the above image also. Now, you need to go inside the folder and for a run this agent use command:

nohup ./node_exporter &

After that, you need to tell Prometheus about this node-exporter agent. For this first, go to configure file of Prometheus(/etc/prometheus/prometheus.yml) and set context.

Now, setup is ready for monitoring but, you need to make this setup permanent as because If by chance pod is deleted so you need to configure again for this …

Step 3:

Main Step

For making this setup persistent you need to create a PVC and mount it with the config file. But there is some interesting thing comes up. After you mount the config file and then your pod is deleted so because of deployment a new pod is created at a time but you see your setup is not there in the config file.

But why???

Because PVC is not meant for mount any file. It’s meant for mount any folder. There is a another concept for this which is subpaths and configMap.

Now, you need to setup context of configMap.

configMap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: prom-graf-config
data:
  prometheus.yml: |-
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']
    - job_name: 'RHEL8'
      static_configs:
      - targets: ['192.168.43.197:9100']
  
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://192.168.99.100:30002

Now, there is a pre-requisite for use a configMap. You can say that also it’s a condition. We use configMap only at the time of creating any pod or deployment. So, I create two YML files one for Prometheus and one for Grafana.

Prometheus.yml

apiVersion: v1
kind: Service
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  ports:
    - port: 9090
      nodePort: 30002
  selector:
    app: prometheus
  type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-prometheus
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1 
kind: Deployment
metadata:
  name: prom-deploy 
  labels:
    app: prometheus
spec:
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - image: prom/prometheus
        name: prometheus
        ports:
        - containerPort: 9090
          name: prometheus
        volumeMounts:
        - name: prom-config
          mountPath: /etc/prometheus/prometheus.yml
          subPath: prometheus.yml
        - name: pv-prometheus
          mountPath: /prometheus
      volumes:
      - name: prom-config
        configMap:
          name: prom-graf-config
      - name: pv-prometheus
        persistentVolumeClaim:
          claimName: pvc-prometheus

Grafana.yml

apiVersion: v1
kind: Service
metadata:
  name: grafana
  labels:
    app: grafana
spec:
  ports:
    - port: 3000
      nodePort: 30003
  selector:
    app: grafana
  type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-grafana
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1 
kind: Deployment
metadata:
  name: graf-deploy 
  labels:
    app: grafana
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - image: grafana/grafana
        name: grafana
        ports:
        - containerPort: 3000
          name: grafana
        volumeMounts:
        - name: graf-config
          mountPath: /etc/grafana/provisioning/datasources/datasource.yml
          subPath: datasource.yml
        - name: pv-grafana
          mountPath: /var/lib/grafana
      volumes:
      - name: graf-config
        configMap:
          name: prom-graf-config
      - name: pv-grafana
        persistentVolumeClaim:
          claimName: pvc-grafana

Finally, we are in last stage of this task. Now, I create a kustomization.yml file. So, I can launch all the things in one single shot.

kustomization.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - configmap.yml
  - Prometheus.yml
  - Grafana.yml

After create that file we only need to just run command:

kubectl create -k .

Now, you can see the web app of Prometheus and Grafana…

For just confirmation, I write a query of free ram in Grafana ((node_memory_MemFree_bytes)/1024)/1024

That means everything works properly. Now, If by chance any pod is deleted so it create again with in a sec and also with all the configuration which I set.

For your reference I upload my codes in my GitHub repository.

Thank you for reading …

Connect with me On Linkedin For FurThur Queries Or Suggestion’s if u Have Any !!!