Azure Application Gateway Log Monitoring using Event Hub

this guide explains how to stream azure application gateway logs to kloudmate using azure event hub forward diagnostic logs from your gateway (or other azure resources) to event hub, then ingest them using the kloudmate agent for centralized monitoring, analysis, and troubleshooting on the kloudmate platform use case this setup enables pushing azure logs (application gateway, databases, apis, etc ) through event hub so all events can be monitored from a single platform architecture overview azure resource → diagnostic settings → event hub → kloudmate agent → kloudmate platform prerequisites azure subscription with application gateway configured kloudmate agent installed on linux vm docid 4wisf1tqcgkdc1gx8irdl permissions to create event hub resources kloudmate api key step 1 set up event hub namespace and policies in azure portal, search for event hubs > create select subscription, resource group, unique namespace name, and region click review + create in the namespace go to settings > shared access policies > + add set policy name and enable manage (includes send & listen) create and copy the primary connection string (namespace level, for diagnostic settings) under entities > event hubs > + event hub enter event hub name and partition count (default is fine) create in the new event hub go to settings > shared access policies > + add set policy name and enable manage create and copy the primary connection string (event hub level, for kloudmate agent) the namespace hosts event hub entities; use these connection strings in later steps step 2 configure diagnostic settings on application gateway open your application gateway > monitoring > diagnostic settings > + add diagnostic setting select logs access logs , performance logs , and firewall logs (if waf enabled) choose send to event hub select namespace from step 1 choose namespace level shared access policy from step 1 click save gateway logs now stream to the event hub in real time step 3 install and configure kloudmate agent ssh to your linux vm and install curl s https //install kloudmate com | bash verify kloudmate agent status edit /etc/kloudmate/config yaml (use sudo nano ) sample configuration extensions health check pprof endpoint 0 0 0 0 1777 zpages endpoint 0 0 0 0 55679 receivers azureeventhub connection endpoint=\<primary connection string> # event hub level string from step 1 format "azure" processors resource attributes \ action upsert from attribute azure resource id key service name transform/appgw log statements \ context log statements \ set(resource attributes\["azure appgw\ name"], split(resource attributes\["azure resource id"], "/")) where resource attributes\["azure resource id"] != nil \ set(resource attributes\["service name"], resource attributes\["azure appgw\ name"]) where resource attributes\["azure appgw\ name"] != nil \ set(body, attributes\["azure properties"]) where attributes\["azure properties"] != nil batch send batch size 5000 timeout 60s exporters debug verbosity detailed otlphttp endpoint 'https //otel kloudmate dev 4318' headers authorization \<api key> # your kloudmate api key service pipelines logs receivers \[azureeventhub] processors \[batch, resource, transform/appgw] exporters \[debug, otlphttp] extensions \[health check, pprof, zpages] restart sudo systemctl restart kloudmate agent benefits real time, searchable application gateway logs centralized monitoring for azure resources (e g , sql, storage, vms, apis) enhanced troubleshooting and visibility extending to other azure services this event hub + kloudmate pipeline monitors any azure service beyond application gateway, no agent reconfiguration needed purpose apply the same setup across multiple resources to centralize all azure logs in kloudmate how it works event hub accepts diagnostic logs from any azure service identical config yaml pipeline processes all logs automatically repeat only step 2 (diagnostic settings) per service, targeting your existing event hub applicable services azure sql databases → query/performance/error logs storage accounts → blob/container access logs app services/apis → http request/response metrics virtual machines → platform/os logs any diagnostic enabled azure service result single agent monitors your entire azure estate