Kubernetes Agent

18 min

this document will guide you to instrument kubernetes using the kloudmate agent via daemonset & deployment what the kubernetes agent monitors once installed, the kubernetes agent collects cluster metrics cpu, memory, disk, network usage node metrics resource usage, health, availability namespace metrics workloads, events per namespace pod metrics cpu, memory, restarts, and lifecycle events workload metrics deployment and replica status, resource utilization container logs stdout/stderr from pods cluster events pod failures, scaling events, and other events optional apm metrics instrumented applications (python, node js, java, net, go) these are the raw metrics and logs collected by the agent visualization, alerts, and dashboards are accessed separately accessing your data view all kubernetes data in these kloudmate sections 1\ kubernetes monitoring explore metrics and events for your cluster, nodes, namespaces, pods, and workloads track availability, resource usage, pod restarts, and workload health docid\ lebi770tguwagtaffmw2k 2\ dashboards use the following pre built dashboards to visualize kubernetes metrics https //templates kloudmate com/k8s cluster dashboard/index html https //templates kloudmate com/k8s node dashboard/index html https //templates kloudmate com/k8s namespace dashboard/index html https //templates kloudmate com/k8s pod dashboard/index html build custom dashboards for specific workloads, namespaces, or pods docid\ ok ep0fq7ewvljdwp4mi8 3\ log explorer search and filter pod logs and cluster events correlate logs with metrics to troubleshoot container or application issues docid 8xd7dr5xgyamqh5umhnlr install agent the commands below are for reference only log in to kloudmate and copy your account specific command to install the agent step 1 getting started use the kloudmate kubernetes agent to monitor your kubernetes infrastructure and optionally your applications (apm) without any code changes pre requisites kubernetes version 1 24 or above helm version 3 0 or above cert manager version v1 18 2 or above step 2 install cert manager skip this step if you already have cert manager installed helm install \\ cert manager oci //quay io/jetstack/charts/cert manager \\ \ namespace cert manager \\ \ create namespace \\ \ set crds enabled=true step 3 enter cluster details provide a name for your kubernetes cluster this name will be used to identify the cluster inside kloudmate step 4 set preferences configure how kloudmate should collect data from this cluster collect container logs toggle on/off based on whether you want to ingest container logs from the cluster enable auto instrumentation for apm turn this on to automatically instrument supported application runtimes (python, node js, java, net, go) for apm monitored namespaces enter a comma‑separated list of namespaces to monitor for events and apm (for example, redis, kafka, prod app ) leave this field empty to monitor all namespaces (system namespaces are excluded by default) step 5 run the installation command the ui will generate a helm command based on your inputs run this command in your terminal to install the kloudmate agent in your kubernetes cluster helm repo add kloudmate https //charts kloudmate com helm repo update helm install kloudmate release kloudmate/km kube agent \\ \ namespace km agent create namespace \\ \ set api key="\<your api key>" \\ \ set collector endpoint="https //otel kloudmate dev 4318" \\ \ set clustername="test" \\ \ set monitorednamespaces="default, prod, dev" \\ \ set featuresenabled logs="true" \\ \ set featuresenabled apm="false" installing the agent on tainted nodes if your kubernetes cluster has nodes with taints, the agent pods must be assigned matching tolerations to be scheduled successfully by default, the agent does not include any tolerations, but you can configure them during installation by using helm parameters example helm installation command with tolerations helm install kloudmate release kloudmate/km kube agent namespace km agent create namespace \\ \ set api key="\<your api key>" set collector endpoint="https //otel kloudmate com 4318" \\ \ set clustername="\<your cluster name>" \\ \ set "monitorednamespaces={monitored ns}" \\ \ set tolerations\[0] key="env" set tolerations\[0] operator="equal" set tolerations\[0] value="uat" set tolerations\[0] effect="noschedule" \\ \ set tolerations\[1] key="env" set tolerations\[1] operator="equal" set tolerations\[1] value="sb uat node" set tolerations\[1] effect="noschedule" \\ in this example, two separate tolerations are defined to match the taints configured on the cluster nodes, enabling the agent pods to be scheduled on all required nodes you can add any number of tolerations by incrementing the index ( tolerations\[0] , tolerations\[1] , etc ) ensure that the key, operator, value, and effect match the taints applied to your nodes if your cluster does not have taints, these parameters can be omitted the agent pods will then be scheduled on untainted nodes by default apm setup instructions there are two ways to enable application performance monitoring (apm) in your cluster 1\ enable apm via dashboard after installing the agent, open the apm configuration tab in the dashboard use the toggle button to enable or disable apm once enabled, the agent will automatically detect all running microservices and identify their respective programming languages after turning on apm, wait a few minutes to begin receiving metrics, then verify them on the dashboard 2\ enable apm via the patch method in some cases, the agent may not automatically detect certain applications in such scenarios, use deployment annotations to enable apm for your applications steps 1\ identify each application that requires apm 2\ you can enable apm using either of the following methods java option 1 – patch command kubectl patch deployment \<name> n \<ns> p '{"spec" {"template" {"metadata" {"annotations" {"instrumentation opentelemetry io/inject java" "km agent/km agent instrumentation crd"}}}}}' option 2 – add annotation annotations instrumentation opentelemetry io/inject java km agent/km agent instrumentation crd python option 1 – patch command kubectl patch deployment \<name> n \<ns> p '{"spec" {"template" {"metadata" {"annotations" {"instrumentation opentelemetry io/inject python" "km agent/km agent instrumentation crd"}}}}}' option 2 – add annotation annotations instrumentation opentelemetry io/inject python km agent/km agent instrumentation crd node js option 1 – patch command kubectl patch deployment \<name> n \<ns> p '{"spec" {"template" {"metadata" {"annotations" {"instrumentation opentelemetry io/inject nodejs" "km agent/km agent instrumentation crd"}}}}}' option 2 – add annotation annotations instrumentation opentelemetry io/inject nodejs km agent/km agent instrumentation crd net option 1 – patch command kubectl patch deployment \<name> n \<ns> p '{"spec" {"template" {"metadata" {"annotations" {"instrumentation opentelemetry io/inject dotnet" "km agent/km agent instrumentation crd"}}}}}' option 2 – add annotation annotations instrumentation opentelemetry io/inject dotnet km agent/km agent instrumentation crd 3\ restart the deployments after adding the annotations next steps open kubernetes monitoring to view metrics and events for your cluster , nodes , namespaces , pods , and workloads use dashboards to create visualizations or view pre built dashboards use log explorer to investigate logs and correlate them with metrics configure alerts for any of these components based on thresholds