Skip to content
KloudMate Docs

Amazon ECS Container Logs

This guide explains how to send Amazon ECS container logs from Fargate workloads to KloudMate using Fluent Bit, FireLens, and the OpenTelemetry output plugin.

In this setup:

  • Fluent Bit forwards log records
  • FireLens routes container logs to Fluent Bit
  • The OpenTelemetry output plugin sends those logs to KloudMate

Prerequisites

Section titled “Prerequisites”

Step 1: Create a Fluent Bit Image

Section titled “Step 1: Create a Fluent Bit Image”

Create a Dockerfile for Fluent Bit:

FROM fluent/fluent-bit:latest
ADD logDestinations.conf /fluentbit-config.conf
ADD parsers.conf /parsers.conf

Step 2: Configure Fluent Bit Output

Section titled “Step 2: Configure Fluent Bit Output”
[SERVICE]
    Flush                5
    Log_Level            debug
    Daemon               off
    Parsers_File         /parsers.conf

[INPUT]
    Name                 forward
    Listen               0.0.0.0
    Port                 2020
    Buffer_Chunk_Size    1M
    Buffer_Max_Size      6M
    Tag_Prefix           ecs.

[OUTPUT]
    Name                 opentelemetry
    Match                *
    Host                 otel.kloudmate.com
    Port                 4318
    Logs_uri             /v1/logs
    Log_response_payload True
    Header               Authorization <Auth token>
    Tls                  On
    Tls.verify           Off
    add_label            app fluent-bit
    add_label            <Container tag>

Step 3: Add Parsers

Section titled “Step 3: Add Parsers”

Add a parser file that matches your application logs. For example, the parser below handles common NGINX access logs:

[PARSER]
    Name          nginx
    Format        regex
    Regex         ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
    Time_Key      time
    Time_Format   %d/%b/%Y:%H:%M:%S %z

Step 4: Push the Image to ECR

Section titled “Step 4: Push the Image to ECR”

Push the custom Fluent Bit image to Amazon Elastic Container Registry (ECR).

Step 5: Update the ECS Task Definition

Section titled “Step 5: Update the ECS Task Definition”

The task definition below uses two containers: the application container and a Fluent Bit log router. Make sure the Fluent Bit configuration file is included in the image before deploying the task.

"containerDefinitions": [
        {
            "name": "nginx",
            "image": "",
            "cpu": 0,
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "Host": "127.0.0.1",
                    "Name": "forward",
                    "Port": "2020",
                    "Retry_Limit": "2"
                }
            }
        },
        {
            "name": "log_router",
            "image": "",
            "cpu": 0,
            "portMappings": [],
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "user": "0",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "firelens-container",
                    "awslogs-region": "us-east-2",
                    "awslogs-stream-prefix": "firelens"
                }
            },
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "config-file-type": "file",
                    "config-file-value": "/fluentbit-config.conf"
                }
            }
        }
    ],