AWS Auto Scaling & Elastic Beanstalk

Auto Scaling groups and Elastic Beanstalk environments create and replace instances automatically. The usual install flow assumes a stable host. You install the agent, then choose which apps to instrument in the KloudMate UI. That does not work when instances come and go. There is no lasting host to configure, and a setting made on one instance is not carried to the next.

Autonomous auto-instrument mode removes the per-instance UI step. The agent discovers and instruments everything on the instance itself. You provide its configuration as files through the platform’s provisioning step: user-data for an Auto Scaling group, or a .platform hook for Elastic Beanstalk. Your API key identifies the telemetry, so a new instance starts reporting as soon as it boots.

An Elastic Beanstalk environment runs on an Auto Scaling group underneath. The agent configuration is the same for both. Only the way you deliver it differs.

What autonomous mode collects

With auto-instrument turned on, every instance gets the following, with no UI step:

• eBPF monitoring. HTTP server spans, RED metrics, and the service map for every app, whatever the language, with no code change. It’s also the only tracing method for runtimes with no SDK injection, such as Ruby and Go. See eBPF observability.
• SDK auto-instrumentation. Deeper, in-process traces for Java, .NET, Node.js, Python, and PHP. The agent injects the SDK through a systemd config file and restarts each app once to pick it up.
• Host metrics and logs. The metrics the agent always collects, plus any log files you configure below.

Each app’s service.name comes from its own stable identity, not from the instance. So every instance of one app rolls up as a single service in KloudMate, and a per-instance breakdown is still available. See Telemetry identity.

Before you start

You need:

• Your KloudMate API key.
• The collector endpoint, https://otel.kloudmate.com:4318.
• Instances on a supported Linux (Amazon Linux 2 or 2023, Ubuntu, Debian, or the RHEL family). eBPF monitoring needs a modern kernel, which the standard AWS AMIs have.
• Network access and permissions from Before you start if your outbound traffic is restricted.

Both examples below put the API key directly into the provisioning step. To keep it out of the launch template, read it from SSM Parameter Store or Secrets Manager instead and set KM_API_KEY from there. That is the only line that changes.

Auto Scaling group

Put this in the launch template’s user-data. It writes the agent configuration, installs the agent with autonomous mode on, and starts it. Running the script again changes nothing, so it’s safe to run on every instance launch.

#!/bin/bash
set -euo pipefail

export KM_API_KEY="YOUR_API_KEY"
export KM_COLLECTOR_ENDPOINT="https://otel.kloudmate.com:4318"
export KM_AUTO_INSTRUMENT=true   # autonomous from the first start: no UI step, no per-instance registration

# 1) Optional log collection: one YAML list of sources per file, read when the agent starts (see below).
install -d -m 755 /etc/kmagent/logs.d
cat > /etc/kmagent/logs.d/app.yaml <<'EOF'
- name: my-app
  include: [ /var/log/my-app/*.log ]
  format: json
  parsing: { timestamp_field: ts, severity_field: level, message_field: msg }
  attributes: { service.name: my-app }
EOF

# 2) Install and start. The installer detects the OS, pulls the package, and sets up the systemd service.
curl -fsSL https://cdn.kloudmate.com/scripts/install_linux.sh | bash

Elastic Beanstalk

Elastic Beanstalk runs a provisioning hook on each instance. On the current Amazon Linux 2 and 2023 platforms, use a platform hook. Your environment properties are available there.

1. Set the environment properties on the environment (Console → Configuration → Software → Environment properties, or eb setenv):

KM_API_KEY = YOUR_API_KEY
KM_COLLECTOR_ENDPOINT = https://otel.kloudmate.com:4318

2. Add a predeploy hook to your application source bundle at .platform/hooks/predeploy/01-kmagent.sh, and make it executable with chmod +x:

#!/bin/bash
set -euo pipefail

# Bring the Elastic Beanstalk environment properties into the shell.
[ -f /opt/elasticbeanstalk/deployment/env ] && { set -a; . /opt/elasticbeanstalk/deployment/env; set +a; }
: "${KM_API_KEY:?set KM_API_KEY as an Elastic Beanstalk environment property}"
: "${KM_COLLECTOR_ENDPOINT:=https://otel.kloudmate.com:4318}"
export KM_AUTO_INSTRUMENT=true

# 1) Log collection. nginx logs every request to /var/log/nginx/access.log — the standard, app-agnostic
#    source on the web tier. Add more sources under /etc/kmagent/logs.d/.
install -d -m 755 /etc/kmagent/logs.d
cat > /etc/kmagent/logs.d/beanstalk.yaml <<'EOF'
- name: nginx-access
  include: [ /var/log/nginx/access.log ]
  format: regex
  parsing:
    regex: '^(?P<remote_addr>\S+) - (?P<remote_user>\S+) \[(?P<time>[^\]]+)\] "(?P<request>[^"]*)" (?P<status>\d+)'
    timestamp_field: time
    timestamp_layout: custom
    custom_layout: '%d/%b/%Y:%H:%M:%S %z'
    message_field: request
  attributes: { service.name: my-app }
EOF

# 2) Install once. On later deploys the agent is already there, so restart it to pick up any logs.d change.
if ! command -v kmagent >/dev/null 2>&1; then
  export KM_API_KEY KM_COLLECTOR_ENDPOINT
  curl -fsSL https://cdn.kloudmate.com/scripts/install_linux.sh | bash
else
  systemctl restart kmagent
fi

Configure log collection

Autonomous mode has no UI, so the agent reads its log sources from files. At startup it reads every *.yaml or *.yml file in /etc/kmagent/logs.d/. (Point it at a different directory with KM_LOG_SOURCES_DIR.) Each file is a YAML list of sources. Add as many files as you need; they merge in filename order. An empty directory means no log collection, so it’s entirely opt-in.

Each source takes these fields:

Field	Required	Notes
name	Yes	Source name, also stamped on every record as km.log.source.
include	Yes	List of file globs to tail.
exclude	No	List of globs to skip.
format	Yes	plaintext, json, or regex.
parsing	For json and regex	Parsing rules: timestamp_field, timestamp_layout (iso8601, rfc3339, unix, unix_ms, or custom with custom_layout), severity_field, message_field, regex, multiline_pattern, encoding.
attributes	No	Extra resource attributes. Set service.name to the app’s name so its logs join its traces.
include_file_path	No	Add the source file’s path as an attribute.

A JSON application log:

# /etc/kmagent/logs.d/checkout.yaml
- name: checkout
  include: [ /var/log/checkout/*.log ]
  format: json
  parsing: { timestamp_field: ts, severity_field: level, message_field: msg }
  attributes: { service.name: checkout }

An nginx access log, parsed with a regex:

# /etc/kmagent/logs.d/nginx.yaml
- name: nginx
  include: [ /var/log/nginx/access.log ]
  format: regex
  parsing:
    regex: '^(?P<remote_addr>\S+) - (?P<remote_user>\S+) \[(?P<time>[^\]]+)\] "(?P<request>[^"]*)" (?P<status>\d+)'
    timestamp_field: time
    timestamp_layout: custom
    custom_layout: '%d/%b/%Y:%H:%M:%S %z'
    message_field: request
  attributes: { service.name: nginx }

Tip

Set each log source’s service.name to the same value as the app it belongs to, so its logs, traces, and metrics line up in KloudMate. Lines that don’t match are kept as the raw record body, never dropped. Tailing starts at the end of each file, so there’s no historical backfill on launch.

For the managed-mode version, with the same parsing choices in a UI wizard, see Log monitoring.

Name your services and tune eBPF

eBPF names a service after the process it instruments. On the Beanstalk web tier that’s nginx — your app runs behind it on a unix socket — so Ruby, Go, and other eBPF-traced apps show up as nginx. Name and scope the service with two environment properties, written to the agent’s environment file in the hook:

# in the predeploy hook:
{ echo "KM_EBPF_OPEN_PORTS=${KM_EBPF_OPEN_PORTS:-80}"
  echo "KM_EBPF_SERVICE_NAME=${KM_EBPF_SERVICE_NAME}"; } > /etc/kmagent/kmagent.env

KM_EBPF_OPEN_PORTS=80 scopes tracing to your app’s port, so background processes — the SSM agent, cfn-hup, the health app — stay out. KM_EBPF_SERVICE_NAME=checkout names the traces checkout. SDK runtimes (Java, Node.js, Python, .NET) read OTEL_SERVICE_NAME from the app process itself, so for those you set that property and nothing else.

Environment property	Effect
KM_EBPF_SERVICE_NAME	Name for the eBPF-traced service. Use with KM_EBPF_OPEN_PORTS.
KM_EBPF_OPEN_PORTS	Ports eBPF instruments. Default: all listening ports.
KM_EBPF_ENABLED	false turns off eBPF traces, keeps metrics and logs.
KM_EBPF_NETWORK_ENABLED	false turns off the network topology map.
KM_LOG_SOURCES_DIR	Log-source directory. Default: /etc/kmagent/logs.d.

Using the SDK on the web tier

If you add the OpenTelemetry SDK to an app on the web tier, set KM_EBPF_ENABLED=false for it. The SDK traces the app, and eBPF traces the nginx front end — two different processes — so without this the app is traced twice. (The agent drops eBPF automatically when it finds an SDK on the same process, which doesn’t happen behind a reverse proxy.)

Verify

On any instance:

systemctl status kmagent
journalctl -u kmagent | grep -Ei "auto-instrument mode ON|log monitor"

You should see auto-instrument mode enabled and, if you configured log collection, the log sources being merged. In KloudMate, the host appears with its cloud attributes, your services show traces and RED metrics, and log sources appear under their service.name.

If an instance isn’t reporting, check the agent’s own error logs. It ships them under service.name=kmagent. See See the agent’s own errors.

Notes and limits

• One restart on first boot. Auto-instrument restarts each app once to inject the SDK. That applies to the SDK languages, not to eBPF monitoring. To skip the restart at boot, bake the agent into your AMI so the instrumentation is already configured when the instance starts.
• Configuration is fixed per instance. In autonomous mode the agent doesn’t take configuration from the KloudMate UI. You change what’s instrumented, or your log sources, by updating the launch template or the .platform hook and redeploying. The agent still reports its health to KloudMate.
• Databases aren’t auto-instrumented. They need credentials, which can’t be configured automatically, so monitor RDS or a separate database tier through database monitoring.
• Log sources are read at startup. The agent reads logs.d once, when it starts. Instances that are replaced rather than reconfigured don’t need it re-read.