Java Instrumentation (Running on Kubernetes)

this document explains how to instrument a java spring boot application with kloudmate, using opentelemetry this instrumentation can be done using the opentelemetry operator, which supports injecting and configuring auto instrumentation libraries for net, java, node js, python, and go service prerequisites running a kubernetes cluster the cert manager must be installed if you use the helm chart, there is an option to generate a self signed cert instead step 1 first, install the opentelemetry operator into your cluster you can do this with the operator release manifest , the operator helm chart , or with operator hub bash kubectl apply f https //github com/open telemetry/opentelemetry operator/releases/latest/download/opentelemetry operator yaml step 2 create an opentelemetry collector to send telemetry from containers to a collector instead of directly to a backend for example, here is a demo collector yaml kubectl apply f <\<eof apiversion opentelemetry io/v1beta1 kind opentelemetrycollector metadata name demo namespace kloudmate backend #change namespace spec config receivers otlp protocols grpc endpoint 0 0 0 0 4317 http endpoint 0 0 0 0 4318 processors memory limiter check interval 1s limit percentage 75 spike limit percentage 15 batch send batch size 10000 timeout 10s exporters debug {} otlphttp endpoint 'https //otel kloudmate com 4318' headers authorization service pipelines traces receivers \[otlp] processors \[memory limiter, batch] exporters \[debug, otlphttp] metrics receivers \[otlp] processors \[memory limiter, batch] exporters \[debug, otlphttp] logs receivers \[otlp] processors \[memory limiter, batch] exporters \[debug, otlphttp] eof note change your kloudmate organization api key to configure the collector to export data to kloudmate step 3 configure automatic instrumentation to be able to manage automatic instrumentation, the operator needs to be configured to know what pods to instrument and which automatic instrumentation to use for those pods this is done via the instrumentation crd for our example, we have used java sdk auto instrumentation the following command creates a basic instrumentation resource that is configured for instrumenting java services yaml kubectl apply f <\<eof apiversion opentelemetry io/v1alpha1 kind instrumentation metadata name demo instrumentation namespace kloudmate backend #change namespace spec exporter endpoint http //demo collector 4317 propagators \ tracecontext \ baggage sampler type parentbased traceidratio argument "1" eof note the endpoint used under exporter must be the same as the service name of the collector created in step 2 step 4 now opt in your services to automatic instrumentation this is done by updating your service’s spec template metadata annotations to include a language specific annotation java instrumentation opentelemetry io/inject java "true" to patch your existing java application with the necessary annotation use the below command bash kubectl patch deployment otlp http deployment n kloudmate backend p '{"spec" {"template" {"metadata" {"annotations" {"instrumentation opentelemetry io/inject java" "true"}}}}}' following the above steps, you can get the necessary telemetry data on kloudmate following is a sample yaml file where the annotation is used yaml apiversion apps/v1 kind deployment metadata name spring boot k8s spec selector matchlabels app spring boot k8s replicas 1 template metadata annotations instrumentation opentelemetry io/inject java "true" labels app spring boot k8s spec containers \ name spring boot k8s image springboot instrument 01 imagepullpolicy ifnotpresent ports \ containerport 8080 jvm metrics name description unit jvm memory used measure of memory used bytes jvm memory committed measure of memory committed bytes jvm memory limit measure of max obtainable memory bytes jvm memory used after last gc measure of memory used, as measured after the most recent garbage collection event on this pool bytes jvm gc duration duration of jvm garbage collection actions seconds jvm thread count number of executing platform threads thread jvm class loaded number of classes loaded since jvm start class jvm class unloaded number of classes unloaded since jvm start class jvm class count number of classes currently loaded class jvm cpu time cpu time used by the process as reported by the jvm seconds jvm cpu count number of processors available to the java virtual machine cpu jvm cpu recent utilization recent cpu utilization for the process as reported by the jvm one jvm memory init measure of initial memory requested bytes jvm system cpu utilization recent cpu utilization for the whole system as reported by the jvm one jvm system cpu load 1m average cpu load of the whole system for the last minute as reported by the jvm run queue item jvm buffer memory used measure of memory used by buffers bytes jvm buffer memory limit measure of total memory capacity of buffers bytes jvm buffer count number of buffers in the pool buffer