Kubernetes - Monitoring and Logging

Built-in monitoring

Google’s open source projects

• Contrib is a catch-all for a variety of components that are not part of core Kubernetes. It is found at https://github.com/kubernetes/contrib.
• LevelDB is a key store library that was used in the creation of InfluxDB. It is found at https://github.com/google/leveldb.
• Heapster runs in a container on one of the minion nodes and aggregates the data from kublet. A simple REST interface is provided to query the data.
• InfluxDB is an open-source distributed time series database with no external dependencies.
• Grafana, which provides a dashboard and graphing interface for the data stored in InfluxDB.

Exploring Heapster

First, we can list the pods to find the one running Heapster as follows:

$ kubectl get pods --namespace=kube-system

Run a describe command to see which node it is running on as follows:

$ kubectl describe pods/<Heapster monitoring Pod> --namespace=kube-system

Figure 6.2. Heapster pod details

Next, we can SSH to this box with the familiar gcloud ssh command as follows:

$ gcloud compute --project "<Your project ID>" ssh --zone "<your gce zone>" "<kubernetes minion from describe>"

Let’s see the list now by issuing a curl command to the pod IP address we saved from the describe command as follows:

$ curl -G <Heapster IP from describe>:8082/api/v1/metric-export-schema/

Customizing our dashboards

Figure 6.3. Heapster pod details

Figure 6.4. Singlestat options

Figure 6.5. Custom dashboard panels

FluentD and Google Cloud Logging

FluentD

FluentD is a collector. It can be configured to have multiple sources to collect and tag logs, which are then sent to various output points for analysis, alerting, or archiving.

The AWS Kubernetes setup also uses FluentD, but instead forwards events to Elasticsearch.

Maturing our monitoring operations

Beyond system monitoring with Sysdig

Sysdig stands out for its ability to dive deep not only into system operations but specifically containers.

Figure 6.9. Sysdig Cloud Explore page

• Detailed views
• Topology views
• Metrics
• Alerting

Figure 6.10. Sysdig Cloud network topology view

Figure 6.11. The Sysdig Cloud network topology detailed view

Figure 6.12. Sysdig Cloud capacity estimate view

Figure 6.13. Sysdig Cloud new alert