Kubernetes Monitoring

monitoring metrics and logs within a kubernetes cluster is extremely important to ensure everything is running well, resources are being used efficiently, and the overall health of the cluster is maintained note if you're using managed kubernetes services like aks (azure kubernetes service), eks (elastic kubernetes service) , or gke (google kubernetes engine) you can follow the same steps outlined in this guide for monitoring the opentelemetry based setup applies uniformly across these platforms by monitoring different parts of the system and the numbers that represent their performance, you can learn useful things, fix problems, make sure that resources such as memory and cpu are being used wisely, and ensure your kubernetes setup is working nicely in this guide, we're going to look at the main things that kubernetes keeps an eye on, the critical numbers that tell us how they are doing, who is allowed to see these stats, and how to set up a tool called opentelemetry collector to collect these numbers and send them to a particular place called prometheus or cortex, which helps us to make sense of it all kubernetes components to monitor the opentelemetry collector offers a variety of tools to help with monitoring kubernetes some of the most important components for collecting kubernetes data kubeletstats receivers k8sobjects receiver hostmetrics receiver kubernetes cluster receiver filelog receiver prometheus receiver these components are crucial components to monitoring the kubernetes cluster following are the steps to integrate opentelemetry with your cluster step 1 install certificate manager kubectl apply f https //github com/cert manager/cert manager/releases/download/v1 12 0/cert manager yaml step 2 install open telemetry operator kubectl apply f https //github com/open telemetry/opentelemetry operator/releases/latest/download/opentelemetry operator yaml step 3 enable kubestate metrics and node exporter by which prometheus receiver can scrape the available k8s resources and node level metrics helm repo add prometheus community https //prometheus community github io/helm charts helm repo update helm install ksm prometheus community/kube state metrics n "default" helm install nodeexporter prometheus community/prometheus node exporter n "default" step 4 enable kubernetes audit log for better log response click here to enable the audit log step 5 the below configuration will create the required configuration and send kubernetes telemetry data to kloudmate \# this manifest is for kubernetes metrics and logs to the km backend apiversion v1 kind serviceaccount metadata name otel collector namespace default \ apiversion rbac authorization k8s io/v1 kind clusterrole metadata name otel collector namespace default rules \ apigroups \ "" \ apps \ autoscaling \ batch \ extensions \ policy \ rbac authorization k8s io resources \ componentstatuses \ configmaps \ nodes/proxy \ daemonsets \ deployments \ events \ cronjobs \ statefulsets \ endpoints \ horizontalpodautoscalers \ ingress \ jobs \ limitranges \ namespaces \ nodes \ pods \ nodes/stats \ persistentvolumes \ persistentvolumeclaims \ resourcequotas \ replicasets \ replicationcontrollers \ serviceaccounts \ services verbs \ get \ list \ watch \ apiversion rbac authorization k8s io/v1 kind clusterrolebinding metadata name otel collector namespace default subjects \ kind serviceaccount name otel collector namespace default roleref kind clusterrole name otel collector apigroup rbac authorization k8s io \ apiversion v1 kind configmap metadata name otel collector namespace default data config yaml | receivers prometheus config scrape configs \ bearer token file /var/run/secrets/kubernetes io/serviceaccount/token job name integrations/kubernetes/cadvisor kubernetes sd configs \ role node relabel configs \ replacement kubernetes default svc cluster local 443 target label address \ regex ( +) replacement /api/v1/nodes/$${1}/proxy/metrics/cadvisor source labels \ meta kubernetes node name target label metrics path scheme https tls config ca file /var/run/secrets/kubernetes io/serviceaccount/ca crt insecure skip verify true server name kubernetes \ bearer token file /var/run/secrets/kubernetes io/serviceaccount/token job name integrations/kubernetes/kubelet kubernetes sd configs \ role node relabel configs \ replacement kubernetes default svc cluster local 443 target label address \ regex ( +) replacement /api/v1/nodes/$${1}/proxy/metrics source labels \ meta kubernetes node name target label metrics path scheme https tls config ca file /var/run/secrets/kubernetes io/serviceaccount/ca crt insecure skip verify true server name kubernetes \ job name integrations/kubernetes/kube state metrics kubernetes sd configs \ role pod relabel configs \ action keep regex kube state metrics source labels \ meta kubernetes pod label app kubernetes io name \ job name integrations/node exporter kubernetes sd configs \ namespaces names \ default role pod relabel configs \ action keep regex prometheus node exporter source labels \ meta kubernetes pod label app kubernetes io name \ action replace source labels \ meta kubernetes pod node name target label instance \ action replace source labels \ meta kubernetes namespace target label namespace k8s events namespaces \[] filelog include \ /var/log/pods/ / / log \ /var/log/containers/ / log exclude \ /var/log/pods/ /otel collector/ log start at beginning include file path true include file name false operators \ type router id get format routes \ output parser docker expr 'body matches "^\\\\{"' \ output parser crio expr 'body matches "^\[^ z]+ "' \ output parser containerd expr 'body matches "^\[^ z]+z"' \ type regex parser id parser crio regex '^(?p\<time>\[^ z]+) (?p\<stream>stdout|stderr) (?p\<logtag>\[^ ] ) ?(?p\<log> )$' output extract metadata from filepath timestamp parse from attributes time layout type gotime layout '2006 01 02t15 04 05 999999999z07 00' \ type regex parser id parser containerd regex '^(?p\<time>\[^ ^z]+z) (?p\<stream>stdout|stderr) (?p\<logtag>\[^ ] ) ?(?p\<log> )$' output extract metadata from filepath timestamp parse from attributes time layout '%y %m %dt%h %m %s %lz' \ type json parser id parser docker output extract metadata from filepath timestamp parse from attributes time layout '%y %m %dt%h %m %s %lz' \ type move from attributes log to body \ type regex parser id extract metadata from filepath regex '^ \\/(?p\<namespace>\[^ ]+) (?p\<pod name>\[^ ]+) (?p\<uid>\[a f0 9\\ ]{36})\\/(?p\<container name>\[^\\ ]+)\\/(?p\<restart count>\d+)\\ log$' parse from attributes\["log file path"] cache size 128 \ type move from attributes stream to attributes\["log iostream"] \ type move from attributes container name to resource\["k8s container name"] \ type move from attributes namespace to resource\["k8s namespace name"] \ type move from attributes pod name to resource\["k8s pod name"] \ type move from attributes restart count to resource\["k8s container restart count"] \ type move from attributes uid to resource\["k8s pod uid"] processors batch send batch size 10000 timeout 60s k8sattributes auth type "serviceaccount" passthrough true filter node from env var kube node name extract metadata \ k8s pod name \ k8s pod uid \ k8s deployment name \ k8s namespace name \ k8s node name \ k8s pod start time labels \ tag name app label component key app kubernetes io/component from pod pod association \ sources \ from resource attribute name k8s pod ip \ sources \ from resource attribute name k8s pod uid \ sources \ from connection resourcedetection detectors \ env \ ec2 \ system \ docker timeout 5s override false attributes/metrics actions \ key cluster value 'test metrics' #add the cluster name action insert attributes/logs actions \ key source from attribute name action upsert \ key source from attribute operator type action upsert \ key source from attribute log file name action upsert \ key source from attribute fluent tag action upsert \ key source from attribute service name action upsert \ key source from attribute project name action upsert \ key source from attribute servicename action upsert \ key source from attribute projectname action upsert \ key source from attribute pod name action upsert \ key source from attribute container name action upsert \ key source from attribute namespace action upsert \ key servicename value 'k8s logs' action insert exporters otlphttp endpoint 'https //otel kloudmate com 4318' headers authorization xxxxxxxxxxxxxxxxxxxxx #add the required auth key timeout 10s retry on failure enabled true service pipelines metrics receivers \[prometheus] processors \[batch, k8sattributes, resourcedetection, attributes/metrics] exporters \[otlphttp] logs receivers \[filelog, k8s events] processors \[batch, k8sattributes, resourcedetection, attributes/logs] exporters \[otlphttp] \ apiversion apps/v1 kind daemonset metadata name otel collector namespace default spec selector matchlabels app otel collector template metadata labels app otel collector spec serviceaccountname otel collector containers \ name otel collector image otel/opentelemetry collector contrib\ latest args \[" config=/etc/otel collector config/config yaml"] env \ name kube node name valuefrom fieldref fieldpath spec nodename resources requests memory "200mi" cpu "50m" limits memory "400mi" cpu "150m" volumemounts \ name config vol mountpath /etc/otel collector config \ name varlogpods mountpath /var/log/pods readonly true \ name varlibdockercontainers mountpath /var/lib/docker/containers readonly true \ name varlogcontainers mountpath /var/log/containers readonly true volumes \ name config vol configmap name otel collector \ name varlogpods hostpath path /var/log/pods \ name varlibdockercontainers hostpath path /var/lib/docker/containers \ name varlogcontainers hostpath path /var/log/containers securitycontext runasuser 0 this configuration provided for the opentelemetry collector collects metrics and logs from all the mentioned components and then transmits these metrics to the kloudmate backend by using opentelemetry collector alongside appropriate rbac authorization, you can effectively gather and send out metrics this process provides valuable information about resource usage, performance, and the overall well being of the cluster