MongoDB integration with KloudMate using Opentelemetry

mongodb is a widely used nosql database designed for scalability and flexibility it stores data in a json like format (bson), which allows for dynamic schema design and handling of unstructured data known for its high performance in read and write heavy environments, mongodb is ideal for applications like real time analytics, content management, and iot systems available in both open source and enterprise editions, it offers features like horizontal scaling through sharding and built in replication for high availability this document will guide you on collecting mongodb server metrics using the opentelemetry (otel) mongodb metrics receiver this receiver can also help capture metrics on a per database or per collection basis from mongodb instances running on aws ec2, azure vms, or on premise servers pre requisites the mongodb server must be running this receiver supports mongodb versions 4 0+, 5 0, 6 0, 7 0 install the opentelemetry collector on the specific server that requires metric monitoring see installing and configuring opentelemetry collector mongodb recommends setting up a least privilege user (lpu) with a clustermonitor role in order to collect metrics please refer to lpu sh for an example of how to configure these permissions to create this please follow the below steps $mongosh >use admin >db createuser({ user "user", pwd "password", roles \[{ role "clustermonitor", db "admin" }] }) >exit step 1 configure the receivers to scrape metrics as well as logs to start monitoring mongodb with otel collector, you need to configure the mongodb receiver linux users open the file located at /etc/otelcol contrib/config yaml using your preferred text editor windows users create a new file called config yaml in the c \program files\opentelemetry collector folder you can use notepad or any text editor to do this add the suitable extensions extensions health check pprof endpoint 0 0 0 0 1777 zpages endpoint 0 0 0 0 55679 file storage create directory true in this step, two receivers are integrated to retrieve telemetry data from mongodb mongodb receiver to retrieve metrics file log receiver to retrieve logs to get the metrics use only the mongodb receiver receivers mongodb hosts \ endpoint localhost 27017 username otel #change the correct username password passwd #change the correct password collection interval 60s initial delay 1s tls insecure true insecure skip verify true filelog include \[/var/log/mongodb/ log] storage file storage retry on failure enabled true step 2 set up the processor component to identify resource information from the host and either append or replace the resource values in the telemetry data with this information please choose one of the following configuration options based on your provider (aws ec2, azure vm, or on premises server) server(can be on premise, non cloud, or cloud) processors batch send batch size 5000 timeout 10s resourcedetection detectors \[system] system resource attributes host name enabled true host id enabled true os type enabled false resource attributes \ key service name action insert from attribute host name aws ec2 (optional) to retrieve aws ec2 instance tags along with logs and metrics, users need to associate an iam role with the ec2 instance that includes the ec2\ describetags policy processors batch send batch size 5000 timeout 10s resourcedetection/ec2 detectors \["ec2"] ec2 tags \ ^tag1$ \ ^tag2$ \ ^label $ \ ^name$ resource attributes \ key service name action insert from attribute ec2 tag name \ key service instance id action insert from attribute host id azure virtual machines processors batch send batch size 5000 timeout 10s resourcedetection/azure detectors \["azure"] azure tags \ ^tag1$ \ ^tag2$ \ ^label $ \ ^name$ resource attributes \ key service name action insert from attribute azure vm name \ key service instance id action insert from attribute host id step 3 configure the exporter, and extension and save the configuration set up the kloudmate backend on the exporter part of the open telemetry configuration file and configure the pipeline exporters debug verbosity detailed otlphttp endpoint 'https //otel kloudmate com 4318' headers authorization xxxxxxxx # use the auth key service pipelines metrics receivers \[mongodb] processors \[batch, resourcedetection, resource] # apply the correct resourcedetection exporters \[otlphttp] logs receivers \[filelog] processors \[batch, resourcedetection, resource] # apply the correct resourcedetection exporters \[otlphttp] extensions \[health check, pprof, zpages, file storage] step 4 to restart and verify the status of the opentelemetry (otel) collector, follow these steps for linux execute the following commands sudo systemctl restart otelcol contrib sudo systemctl status otelcol contrib these commands will restart the otel collector and display its current status for windows open the services window press win + r , type services msc , and press ok alternatively, search for " services " in the windows start menu in the services window, locate the " opentelemetry collector " service right click the service and select " restart " subsequently, monitor the metrics on the kloudmate dashboard and set up an alarm to receive notifications if the potential metrics for a specific application rise default metrics for mongodb name description type unit mongodb cache operations the number of cache operations of the instance sum number mongodb collection count the number of collections sum number mongodb data size the size of the collection data compression does not affect this value sum bytes mongodb connection count the number of connections sum number mongodb extent count the number of extents sum number mongodb global lock time the time the global lock has been held sum milliseconds mongodb index count the number of indexes sum number mongodb index size the sum of the space allocated to all indexes in the database, including free index space sum bytes mongodb index size the amount of memory used sum bytes mongodb object count the number of objects sum number mongodb operation latency time the latency of operations gauge microseconds mongodb operation count the number of operations executed sum number mongodb operation repl count the number of replicated operations executed sum number mongodb storage size the total amount of storage allocated to this collection sum bytes mongodb database count the number of existing databases sum number mongodb index access count the number of times an index has been accessed sum number mongodb document operation count the number of document operations executed sum number mongodb network io receive the number of bytes received sum bytes mongodb network io transmit the number of bytes transmitted sum bytes mongodb network request count the number of requests received by the server sum number mongodb operation time the total time spent performing operations sum microseconds mongodb session count the total number of active sessions sum number mongodb cursor count the number of open cursors maintained for clients sum number mongodb cursor timeout count the number of cursors that have timed out sum number mongodb lock acquire count number of times the lock was acquired in the specified mode sum number mongodb lock acquire wait count the number of times the lock acquisitions encountered waits because the locks were held in a conflicting mode sum number mongodb lock acquire time cumulative wait time for the lock acquisitions sum microseconds mongodb lock deadlock count the number of times the lock acquisitions encountered deadlocks sum number mongodb health the health status of the server gauge number mongodb uptime the amount of time that the server has been running sum microseconds for full details on metrics, click here