Go Instrumentation (Running on Kubernetes)

this document explains how to instrument the go application with kloudmate, using opentelemetry this instrumentation can be possible using the opentelemetry operator, which supports injecting and configuring auto instrumentation libraries for net, java, node js, python, and go service pre requisites running a kubernetes cluster the cert manager must be installed if you use the helm chart, there is an option to generate a self signed cert instead step 1 first, install the opentelemetry operator into your cluster you can do this with the operator release manifest , the operator helm chart , or with operator hub bash kubectl apply f https //github com/open telemetry/opentelemetry operator/releases/latest/download/opentelemetry operator yaml step 2 create an opentelemetry collector to send telemetry from containers to a collector instead of directly to a backend for example, here is a demo collector used kubectl apply f <\<eof apiversion opentelemetry io/v1beta1 kind opentelemetrycollector metadata name demo namespace default #change namespace spec config receivers otlp protocols grpc endpoint 0 0 0 0 4317 http endpoint 0 0 0 0 4318 processors memory limiter check interval 1s limit percentage 75 spike limit percentage 15 batch send batch size 10000 timeout 10s exporters debug {} otlphttp endpoint 'https //otel kloudmate com 4318' headers authorization kloudmate api key service pipelines traces receivers \[otlp] processors \[memory limiter, batch] exporters \[debug, otlphttp] metrics receivers \[otlp] processors \[memory limiter, batch] exporters \[debug, otlphttp] logs receivers \[otlp] processors \[memory limiter, batch] exporters \[debug, otlphttp] eof change your kloudmate workspace api key to configure the collector to export data to kloudmate step 3 configure automatic instrumentation to be able to manage automatic instrumentation, the operator needs to be configured to know what pods to the instrument and which automatic instrumentation to use for those pods this is done via the instrumentation crd for example, we used go sdk auto instrumentation the following command creates a basic instrumentation resource that is configured for instrumenting go services to get metrics, logs, and traces kubectl apply f <\<eof apiversion opentelemetry io/v1alpha1 kind instrumentation metadata name demo instrumentation namespace default #change namespace spec exporter endpoint http //demo collector 4318 propagators \ tracecontext \ baggage sampler type parentbased traceidratio argument "1" eof the endpoint used under exporter must be the same as the service name of the collector created in step 2 use only one instrumentation command as per the requirement step 4 now opt in your services to automatic instrumentation this is done by updating your service’s spec template metadata annotations spec template metadata annotations to include a language specific annotation go instrumentation opentelemetry io/inject go "true" "true" to patch your existing go application with the necessary annotation and use the below command bash kubectl patch deployment \<deployment name> n \<namespace> p '{"spec" {"template" {"metadata" {"annotations" {"instrumentation opentelemetry io/inject go" "true"}}}}}'