By default, Metrics API is not enabled in Kubernetes. There is a software called Metrics Server, by which metrics can be enabled. Metrics Server software is available here.

kubectl create -f . – command that executes all *.yaml files in current directory

It takes Metrics Server for about 1 minute to collect the data from the cluster and send it to Api-Server.

kubectl top pods / nodes – command that shows CPU and Memory utilization by pods (in docker in-built docker stats command is used for the same purposes)

But these is very superficial information. Prometheus is one of the best tools showing container data. Prometheus is a good collector, but not a good visualizer. For better visualizing, Prometheus is used in combination with Grafana. Prometheus collects the data from Kubernetes, Grafana visualizes that data, with using Grafana dashboards. Prometheus and Grafana work for Docker also.

Not that Splunk is actually a log aggregator. It is not a monitoring tool. Prometheus is alerting tool – if something goes wrong, it alerts users. Splunk is not proactive, but reactive tool. ElasticSearch, Kibana are also considered to be reactive, log aggregation tools.

This folder enables all the metrics required by Prometheus. Commonly used metrics are located in standard subfolder. kubectl create -f . command launched from standard subfolder executes all *.yaml files, thus creates set of metrics. This folder contains data for installing Prometheus. kubectl create -f prometheus.yaml command launched from this folder creates set of Kubernetes entities (Deployment, Replicaset, Service, Pod) related to Prometheus in a newly created namespace, monitoring. kubectl create -f grafana.yaml command creates Deployment, Replicaset, Service and Pod for Grafana in the same namespace, monitoring.

So, the sequence of steps for providing metrics environment is

1. Install Metrics Server
2. Install Kube Metrics
3. Install Prometheus
4. Install Grafana

After that, the relationship between Prometheus and Grafana must be established. Grafana should be accessed via created Kubernetes Service (see above), in Grafana’s Welcome Screen Add data source must be clicked, and Prometheus must be selected from sources list. After that Prometheus URL should be written in URL field (Prometheus is accessed the same way as Grafana, Prometheus host and port match those of Grafana).

Next step is creating Grafana dashboard. When creating dashboard, the query from Prometheus is chosen in Query list alternatives and metric name is written in Metrics field (one of metrics created on Step 2, see above), for instance, it can be kube_pod_created, that shows list of created pods in chosen time frame. Note that metrics in Prometheus are created using PQL (Prometheus Query Language). There are different visualization options on Grafana dashboard (graph, stat, gauge, bar gauge, table, etc).

There are many Grafana dashboards readily available. Kubernetes dashboard for Grafana as well. To get a dashboard, its json code can be downloaded or the number matching particular dashboard must be remembered for later use. On Grafana Welcome Screen, Import must be chosen instead of Create. On the Import page, json can be uploaded or previously remembered number can be inserted. This is a useful link for metrics examples.

Alerting rules can also be written in Grafana.