Configuring AppManager OTel Collector for Apache Traces

Overview

This article covers setting up the AppManager OTel Collector to receive traces from Apache, authenticate them with your API key, and forward them to the ManageEngine APM backend.

Because the OpenTelemetry C agent for Apache does not support custom OTLP export headers, an otelcol-contrib proxy is used as an intermediary to inject the required api-key header.

Architecture:

Apache HTTP Server
  └─► OTel Webserver SDK (gRPC spans)
        └─► otelcol-contrib proxy    (port 4320 — adds api-key header)
              └─► AppManagerOtelCollector    (port 4318 — validates + forwards)
                    └─► ManageEngine APM Backend (https://<your-server>:<port>)

Prerequisites

• Apache instrumented with the OTel Webserver SDK (see Instrumenting Apache HTTP Server with OpenTelemetry C Agent)
• AppManagerOtelCollector binary available
• Your API key from the ManageEngine APM portal
• Your APM server URL (e.g. https://app-server:8446)
• sudo / root access

Obtaining the API Key (License Key)

Before installing, obtain your API key from the Applications Manager UI:

1. In Applications Manager, navigate to New Monitor → Add New Monitor → APM Insight → OpenTelemetry.
2. Copy the license key displayed in the right panel.

This key (format: APMI_<hex-string>) is required for both the collector and the proxy configuration.

Part A: Install and Configure AppManagerOtelCollector

There are two ways to install the collector:

• Option 1 (Recommended): Automated install script from ManageEngine
• Option 2: Manual placement of a pre-downloaded binary

Option 1: Automated Install (Recommended)

Download and run the official installer script. The script handles binary placement, directory creation, and service registration.

cd /tmp
wget -O InstallOtelCollector.zip \
  https://www.manageengine.com/products/applications_manager/54974026/InstallOtelCollector.zip
unzip InstallOtelCollector.zip

Set ME_SERVER_ENDPOINT to point to your Applications Manager server, then run the installer:

export ME_SERVER_ENDPOINT="https://<HOST-NAME>:<SSL-PORT>"
sudo -E sh InstallOtelCollector.sh

Where:

• <HOST-NAME> — Hostname or IP of your Applications Manager server
• <SSL-PORT> — SSL port of Applications Manager (e.g. 8443 or 8446)

Example:

export ME_SERVER_ENDPOINT="https://apm-prod-server:8443"
sudo -E sh InstallOtelCollector.sh

After installation, skip to Step 4: Verify the Collector is Running below.

Option 2: Manual Installation

If you already have the AppManagerOtelCollector package (e.g. from the Applications Manager product bundle), place it manually:

Step 1: Place the Collector Binary

Copy the AppManagerOtelCollector package to a suitable location and create a symlink at /opt/AppManagerOtelCollector:

sudo ln -s /path/to/AppManagerOtelCollector /opt/AppManagerOtelCollector

The directory must contain:

/opt/AppManagerOtelCollector/
├── bin/
│   ├── AppManagerOtelCollector    (main binary)
│   ├── service.sh                 (service wrapper)
│   ├── .env                       (environment config)
│   └── otel-collector-config.yaml (collector pipeline config)
└── logs/
    └── collector.log

Step 2: Configure the .env File

Edit /opt/AppManagerOtelCollector/bin/.env and set the required values:

sudo nano /opt/AppManagerOtelCollector/bin/.env

Key settings to configure:

# Listening addresses for this collector
ME_HTTP_LISTENING_ADDRESS="0.0.0.0:4318"
ME_GRPC_LISTENING_ADDRESS="0.0.0.0:4317"

# ManageEngine APM backend URL (required)
ME_SERVER_ENDPOINT="https://<your-apm-server>:<port>"

# TLS (set true if APM server uses a self-signed certificate)
ME_INSECURE_SKIP_VERIFY="true"

# Log file directory
ME_LOG_FILE_DIRECTORY="/opt/AppManagerOtelCollector/logs"

Step 3: Fix Directory Permissions (if running as a restricted user)

If the collector runs as a non-root user and the binary resides under a home directory, ensure the user can traverse the path:

# Example: if binary is under /home/someuser/SCP/...
sudo chmod o+x /home/someuser
sudo chmod o+x /home/someuser/SCP
# ... and so on for each parent directory

Using the symlink approach in Step 1 (pointing to /opt/AppManagerOtelCollector) is the recommended alternative.

Create a systemd Service

sudo tee /etc/systemd/system/AppManagerOtelCollector.service <<'EOF'
[Unit]
Description=AppManagerOtelCollector service
After=network.target

[Service]
Type=forking
ExecStart=/opt/AppManagerOtelCollector/bin/service.sh service_start
ExecStop=/opt/AppManagerOtelCollector/bin/service.sh service_stop
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

Enable and Start

sudo systemctl daemon-reload
sudo systemctl enable AppManagerOtelCollector
sudo systemctl start AppManagerOtelCollector
sudo systemctl status AppManagerOtelCollector --no-pager

Step 4: Verify the Collector is Running

Verify it is listening:

ss -tlnp | grep -E '4317|4318'

Expected:

LISTEN  0.0.0.0:4317   (gRPC)
LISTEN  0.0.0.0:4318   (HTTP)

Part B: Install otelcol-contrib Proxy

This proxy receives spans from Apache on port 4320 and forwards them to the AppManager collector on port 4318 with the api-key header injected.

Step 1: Download otelcol-contrib

cd /tmp
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.99.0/otelcol-contrib_0.99.0_linux_amd64.tar.gz
tar -xzf otelcol-contrib_0.99.0_linux_amd64.tar.gz
sudo mv otelcol-contrib /usr/local/bin/

Verify:

otelcol-contrib version

Step 2: Create the Proxy Configuration

sudo mkdir -p /etc/otelcol-contrib

sudo tee /etc/otelcol-contrib/config.yaml <<'EOF'
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4320
      http:
        endpoint: 0.0.0.0:4321

exporters:
  otlphttp:
    endpoint: http://127.0.0.1:4318
    tls:
      insecure: true
    headers:
      "api-key": "YOUR_API_KEY_HERE"

  logging:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp, logging]
    metrics:
      receivers: [otlp]
      exporters: [otlphttp, logging]
    logs:
      receivers: [otlp]
      exporters: [otlphttp, logging]
EOF

Step 3: Create a Dedicated Service User

sudo useradd --system --no-create-home otelcol

Step 4: Create the systemd Service

sudo tee /etc/systemd/system/otelcol-proxy.service <<'EOF'
[Unit]
Description=OTel Collector Proxy (adds api-key for Apache traces)
After=network.target AppManagerOtelCollector.service

[Service]
User=otelcol
Environment="NO_PROXY=localhost,127.0.0.1"
Environment="no_proxy=localhost,127.0.0.1"
Environment="http_proxy="
Environment="https_proxy="
ExecStart=/usr/local/bin/otelcol-contrib --config /etc/otelcol-contrib/config.yaml
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

Step 5: Enable and Start

sudo systemctl daemon-reload
sudo systemctl enable otelcol-proxy
sudo systemctl start otelcol-proxy
sudo systemctl status otelcol-proxy --no-pager

Verify it is listening:

ss -tlnp | grep 4320

Part C: Remove Any System HTTP Proxy (if present)

A system HTTP proxy can intercept connections to localhost/127.0.0.1 and break the pipeline. Check for and remove any proxy settings:

# Check current proxy
echo $http_proxy
echo $https_proxy

# Check if set in ~/.bashrc
grep -n "http_proxy\|https_proxy" ~/.bashrc

If proxy lines are present in ~/.bashrc, remove them:

sed -i '/export http_proxy=/d; /export https_proxy=/d' ~/.bashrc
unset http_proxy https_proxy

Part D: Update Apache OTel Endpoint

Ensure Apache's OTel module is pointed at the proxy (port 4320), not directly at the collector.

In /etc/apache2/conf-available/opentelemetry_module.conf:

ApacheModuleOtelExporterEndpoint     http://localhost:4320

Then restart Apache:

sudo systemctl restart apache2

Verification

Check all services are running

systemctl is-active apache2 AppManagerOtelCollector otelcol-proxy

Expected: three lines of active.

Check ports are listening

ss -tlnp | grep -E '4317|4318|4320'

Send test traffic

for i in $(seq 1 5); do curl -s -o /dev/null http://localhost/; done

Confirm traces arrived at the collector

sudo tail -20 /opt/AppManagerOtelCollector/logs/collector.log | python3 -c "
import sys, json
for line in sys.stdin:
    line = line.strip()
    if not line: continue
    try:
        d = json.loads(line)
        code = d.get('response_code')
        if code:
            print(d.get('level'), '|', d.get('msg'), '| code=', code, '| url=', d.get('URL',''))
    except: pass
"

Expected output showing successful flow:

info | outbound (response) | code= 200 | url= https://<server>/otel/validate
info | inbound (response) | code= 200 | url= /v1/traces
info | outbound (response) | code= 200 | url= https://<server>/arh/connect

Check otelcol-proxy logs for span content

sudo journalctl -u otelcol-proxy --no-pager -n 30

You should see spans with attributes such as:

http.method: GET
http.target: /portfolio/
http.status_code: 200

Troubleshooting

AppManager collector shows "apikey":""

Cause: The API key header is not reaching the collector, or ME_SERVER_ENDPOINT is empty.

Fix:

1. Verify ME_SERVER_ENDPOINT is set in .env.
2. Verify the proxy config has the correct api-key header value.
3. Confirm no system proxy is intercepting localhost traffic (see Part C).

otelcol-proxy shows EOF error when exporting

Cause: System HTTP proxy routing the connection through an external proxy.

Fix: Set the exporter endpoint in /etc/otelcol-contrib/config.yaml to http://127.0.0.1:4318 (not http://localhost:4318), and ensure the NO_PROXY environment variable is set in the service (Part B, Step 4).

401 Unauthorized from the collector

Cause: Incorrect API key value.

Fix: Double-check the key in /etc/otelcol-contrib/config.yaml matches exactly what the APM portal issued (format: APMI_<hex-string>).

No traces in the proxy logs at all

Cause: Apache OTel module is not sending to port 4320.

Fix: Verify ApacheModuleOtelExporterEndpoint in the Apache config is http://localhost:4320, then restart Apache.

File Reference