MongoDB Integration using KloudMate Agents

mongodb is a widely used nosql database designed for scalability and flexibility it stores data in a json like format (bson), which enables dynamic schema design and the handling of unstructured data known for its high performance in read and write heavy environments, mongodb is ideal for applications such as real time analytics, content management, and iot systems available in both open source and enterprise editions, it offers features like horizontal scaling through sharding and built in replication for high availability this document will guide you on collecting mongodb server metrics using the kloudmate agent(based on opentelemetry) the agent enables you to capture metrics at the server, database, and collection level from mongodb instances running on aws ec2, azure vms, or on premise servers pre requisites the mongodb server must be running this receiver supports mongodb versions 4 0+, 5 0, 6 0, 7 0 kloudmate agent installed on the mongodb host (see agent installation for https //docs kloudmate com/linux agent and https //docs kloudmate com/windows agent ) mongodb recommends setting up a least privilege user (lpu) with the clustermonitor https //www mongodb com/docs/v5 0/reference/built in roles/#mongodb authrole clustermonitor https //www mongodb com/docs/v5 0/reference/built in roles/#mongodb authrole clustermonitor to collect metrics please refer to https //github com/open telemetry/opentelemetry collector contrib/blob/main/receiver/mongodbreceiver/testdata/integration/scripts/lpu sh for an example of how to configure these permissions to create this, please follow the steps below $mongosh >use admin >db createuser({ user "user", pwd "password", roles \[{ role "clustermonitor", db "admin" }] }) >exit step 1 access agents and opentelemetry collector configuration navigate to the kloudmate platform and log in to your account go to settings > agents section here you can view all installed agents access configuration settings for each agent review the default configurations provided by kloudmate to edit an agent's collector configuration select the desired agent click the actions menu (or agent action tab) choose collector configuration a text editor ui will open, allowing you to edit the agent’s yaml configuration directly within the platform add the suitable extensions extensions health check pprof endpoint 0 0 0 0 1777 zpages endpoint 0 0 0 0 55679 file storage create directory true in this step, two receivers are integrated to retrieve telemetry data from mongodb mongodb receiver to retrieve metrics file log receiver to retrieve logs to get the metrics, use only the mongodb receiver receivers mongodb hosts \ endpoint localhost 27017 username otel #change the correct username password passwd #change the correct password collection interval 60s initial delay 1s tls insecure true insecure skip verify true filelog include \[/var/log/mongodb/ log] storage file storage retry on failure enabled true step 2 configure the processor component to extract resource information from the host and either append or replace the resource values in the telemetry data with this information please choose one of the following configuration options based on your provider (aws ec2, azure vm, or on premises server) server(can be on premise, non cloud, or cloud) processors batch send batch size 5000 timeout 10s resourcedetection detectors \[system] system resource attributes host name enabled true host id enabled true os type enabled false resource attributes \ key service name action insert from attribute host name aws ec2 (optional) to retrieve aws ec2 instance tags along with logs and metrics, users need to associate an iam role with the ec2 instance that includes the ec2\ describetags policy processors batch send batch size 5000 timeout 10s resourcedetection/ec2 detectors \["ec2"] ec2 tags \ ^tag1$ \ ^tag2$ \ ^label $ \ ^name$ resource attributes \ key service name action insert from attribute ec2 tag name \ key service instance id action insert from attribute host id azure virtual machines processors batch send batch size 5000 timeout 10s resourcedetection/azure detectors \["azure"] azure tags \ ^tag1$ \ ^tag2$ \ ^label $ \ ^name$ resource attributes \ key service name action insert from attribute azure vm name \ key service instance id action insert from attribute host id step 3 configure the exporter and extension and save the configuration set up the exporter as the kloudmate backend by specifying the endpoint and authorisation key in the agent configuration file, and configure the pipeline exporters debug verbosity detailed otlphttp endpoint 'https //otel kloudmate com 4318' headers authorization xxxxxxxx # use the auth key service pipelines metrics receivers \[mongodb] processors \[batch, resourcedetection, resource] # apply the correct resourcedetection exporters \[otlphttp] logs receivers \[filelog] processors \[batch, resourcedetection, resource] # apply the correct resourcedetection exporters \[otlphttp] extensions \[health check, pprof, zpages, file storage] step 4 save configuration and restart agent after updating the configuration, click “ save configuration ” in the ui or, from your server’s console, run for linux execute the following commands systemctl restart kmagent systemctl status kmagent these commands will restart the kloudmate agent and display its current status for windows open the services window press win + r , type services msc , and press ok alternatively, search for " services " in the windows start menu in the services window, locate the " kloudmate agent " service right click the service and select " restart " subsequently, monitor the metrics on the kloudmate dashboard and set up an alarm to receive notifications if the potential metrics for a specific application rise post‑integration data validation verify that metrics are flowing into kloudmate using the explore view; the metrics will be visible here only if the integration is successful log in to your kloudmate account go to explore , select opentelemetry – metrics , choose a metric exposed by the integration, and click run query to see time‑series data confirming telemetry is available standard mongodb dashboards mongodb monitoring dashboards are available in the https //templates kloudmate com/mongo db/index html to import and start using these templates, follow the steps described in https //docs kloudmate com/creating a dashboard#sfk3g default metrics for mongodb name description type unit mongodb cache operations the number of cache operations of the instance sum number mongodb collection count the number of collections sum number mongodb data size the size of the collection data compression does not affect this value sum bytes mongodb connection count the number of connections sum number mongodb extent count the number of extents sum number mongodb global lock time the time the global lock has been held sum milliseconds mongodb index count the number of indexes sum number mongodb index size the sum of the space allocated to all indexes in the database, including free index space sum bytes mongodb index size the amount of memory used sum bytes mongodb object count the number of objects sum number mongodb operation latency time the latency of operations gauge microseconds mongodb operation count the number of operations executed sum number mongodb operation repl count the number of replicated operations executed sum number mongodb storage size the total amount of storage allocated to this collection sum bytes mongodb database count the number of existing databases sum number mongodb index access count the number of times an index has been accessed sum number mongodb document operation count the number of document operations executed sum number mongodb network io receive the number of bytes received sum bytes mongodb network io transmit the number of bytes transmitted sum bytes mongodb network request count the number of requests received by the server sum number mongodb operation time the total time spent performing operations sum microseconds mongodb session count the total number of active sessions sum number mongodb cursor count the number of open cursors maintained for clients sum number mongodb cursor timeout count the number of cursors that have timed out sum number mongodb lock acquire count number of times the lock was acquired in the specified mode sum number mongodb lock acquire wait count the number of times the lock acquisitions encountered waits because the locks were held in a conflicting mode sum number mongodb lock acquire time cumulative wait time for the lock acquisitions sum microseconds mongodb lock deadlock count the number of times the lock acquisitions encountered deadlocks sum number mongodb health the health status of the server gauge number mongodb uptime the amount of time that the server has been running sum microseconds for full details on metrics, click https //github com/open telemetry/opentelemetry collector contrib/blob/main/receiver/mongodbreceiver/documentation md