Amazon ECS Container Logs to KloudMate

this document provides a guide on how to send amazon ecs container (fargate launch type) logs to kloudmate using opentelemetry in the described setup, fluentbit is used as the log exporting agent to send logs from the application container firelens is the log driver used for exporting logs the opentelemetry plugin allows you to take logs from fluentbit and submit them to a kloudmate http endpoint step 1 prerequisites fluentbit binaries and packages as per your system requirement an ecs cluster running containers with the desired logs to be sent recommendation use the latest fluentbit image to send container logs create a custom aws fluentbit image with a proper configuration as well as required parsers step 2 create a dockerfile for fluentbit from fluent/fluent bit\ latest add logdestinations conf /fluentbit config conf add parsers conf /parsers conf recommendation instead of using aws firelens image use the official fluentbit image step 3 structure the configuration file for successful log delivery \[service] flush 5 log level debug daemon off parsers file /parsers conf \[input] name forward listen 0 0 0 0 port 2020 buffer chunk size 1m buffer max size 6m tag prefix ecs \[output] name opentelemetry match host otel kloudmate com port 4318 logs uri /v1/logs log response payload true header authorization \<auth token> tls on tls verify off \# add user defined labels add label app fluent bit add label \<container tag> step 4 add parsers file as per use case here is an example of nginx parsers here an example nginx parsers \[parser] name nginx format regex regex ^(?\<remote>\[^ ] ) (?\<host>\[^ ] ) (?\<user>\[^ ] ) \\\[(?\<time>\[^\\]] )\\] "(?\<method>\s+)(? +(?\<path>\[^\\"] ?)(? +\s )?)?" (?\<code>\[^ ] ) (?\<size>\[^ ] )(? "(?\<referer>\[^\\"] )" "(?\<agent>\[^\\"] )") time key time time format %d/%b/%y %h %m %s %z step 5 push this image to the amazon elastic container registry (ecr) step 6 task definition to update the service this task definition contains two containers, one is nginx and the other is a fluentbit make sure to configure the config file of fluentbit before proceeding with ecs tasks "containerdefinitions" \[ { "name" "nginx", "image" "\<image uri>", "cpu" 0, "essential" true, "environment" \[], "mountpoints" \[], "volumesfrom" \[], "logconfiguration" { "logdriver" "awsfirelens", "options" { "host" "127 0 0 1", "name" "forward", "port" "2020", "retry limit" "2" } } }, { "name" "log router", "image" "\<image uri custom 2 fluent bit\ latest>", "cpu" 0, "portmappings" \[], "essential" true, "environment" \[], "mountpoints" \[], "volumesfrom" \[], "user" "0", "logconfiguration" { "logdriver" "awslogs", "options" { "awslogs create group" "true", "awslogs group" "firelens container", "awslogs region" "us east 2", "awslogs stream prefix" "firelens" } }, "firelensconfiguration" { "type" "fluentbit", "options" { "config file type" "file", "config file value" "/fluentbit config conf" } } } ],