How to configure Prometheus inside Dockerswarm to scrape necessary metrics for Applications Manager?

How to configure Prometheus inside Dockerswarm to scrape necessary metrics for Applications Manager?


STEP 1 — Ensure Overlay Network Exists
The monitoring overlay network must exist before deploying any services.
Check if it already exists:
                       docker network ls
Expected output:
      NETWORK ID     NAME         DRIVER    SCOPE
      m91dq75djyhc   monitoring   overlay   swarm

If the `monitoring` network does not exist, create it:
        docker network create --driver overlay --attachable monitoring 
Note: The `--attachable` flag allows standalone containers to join the network if needed.

STEP 2 — Deploy Node Exporter (Global)
Node Exporter collects host-level metrics (CPU, memory, disk, network) from every Swarm node.
Deploy it in global mode so it runs on all nodes automatically.
Execute the below on the Manager Node:

docker service create \
  --name node-exporter \
  --mode global \
  --hostname "{{.Node.Hostname}}" \
  --network monitoring \
  --mount type=bind,source=/proc,target=/host/proc,readonly \
  --mount type=bind,source=/sys,target=/host/sys,readonly \
  --mount type=bind,source=/,target=/rootfs,readonly \
  prom/node-exporter \
  --path.procfs=/host/proc \
  --path.sysfs=/host/sys \
  --path.rootfs=/rootfs

Verify:
         docker service ls
Should show:
        node-exporter   global   n/n

STEP 3 — Deploy cAdvisor (Global)
cAdvisor collects container-level metrics (CPU, memory, network per container).
Deploy it in global mode so it runs on all nodes automatically.
Execute the below on the Manager Node:
docker service create \
  --name cadvisor \
  --mode global \
  --network monitoring \
  --mount type=bind,source=/,target=/rootfs,readonly \
  --mount type=bind,source=/var/run,target=/var/run \
  --mount type=bind,source=/sys,target=/sys,readonly \
  --mount type=bind,source=/var/lib/docker/,target=/var/lib/docker,readonly \
  --constraint 'node.platform.os == linux' \
  --cap-add SYS_ADMIN \
  gcr.io/cadvisor/cadvisor:latest

Verify:
       docker service ls

Should show:
     cadvisor   global   2/2

STEP 4 — Deploy Docker Swarm Exporter

Docker Swarm Exporter collects Swarm-level metrics (services, tasks, nodes, replicas).
It must run on a Manager Node as it needs access to the Docker socket.
Execute the below on the Manager Node:
docker service create \
  --name docker-swarm-exporter \
  --network monitoring \
  --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock,readonly \
  --constraint 'node.role == manager' \
  --publish 9000:9000 \
  nicholasjackson/swarm-exporter:latest

Verify:
     docker service ls
Should show:
    docker-swarm-exporter   replicated   1/1
Verify metrics endpoint:
> Note: Look for metrics like `swarm_services_running`, `swarm_nodes_total` in the output.

STEP 5 — Enable Docker Engine Metrics (On ALL Nodes)

Perform the following steps on every node (Manager + Workers):
Edit Docker daemon config:
  sudo vi /etc/docker/daemon.json

Add the following:
{
  "metrics-addr": "127.0.0.1:9323",
  "experimental": true
}

> Note: Use `"metrics-addr": "0.0.0.0:9323"` only if you need to access metrics
> from a remote host. For security, restrict to internal IP in production.

Restart Docker:
sudo systemctl restart docker
Verify the metrics endpoint:
> Verify: Look for `engine_daemon_engine_info` in the output to confirm it is working.

 STEP 6 — Create Prometheus Configuration

On the Manager Node:

mkdir -p ~/prometheus
cd ~/prometheus
vi prometheus.yml

add the below configuration:
global:
  scrape_interval: 15s

scrape_configs:

  - job_name: 'dockerswarm'
    dns_sd_configs:
      - names:
          - 'tasks.cadvisor'
        type: 'A'
        port: 8080
      - names:
          - 'tasks.node-exporter'
        type: 'A'
        port: 9100
      - names:
          - 'tasks.docker-swarm-exporter'
        type: 'A'
        port: 9000
    static_configs:
      - targets:
          - '<manager-ip>:9323'    # Replace with actual Manager Node IP
          - '<worker-ip>:9323'     # Replace with actual Worker Node IP
        labels:
          exporter: docker-engine

    relabel_configs:
      # Tag each target by exporter type
      - source_labels: [__address__]
        regex: '.*:8080'
        target_label: exporter
        replacement: cadvisor

      - source_labels: [__address__]
        regex: '.*:9100'
        target_label: exporter
        replacement: node-exporter

      - source_labels: [__address__]
        regex: '.*:9000'
        target_label: exporter
        replacement: docker-swarm-exporter

      # Force instance = "crc" for ALL targets
      - source_labels: [__address__]
        regex: '.+'
        target_label: instance
        replacement: 'crc'

    metric_relabel_configs:
      # Ensure instance stays "crc" after scrape
      - target_label: instance
        replacement: 'crc'

> Note: Replace `<manager-ip>` and `<worker-ip>` with your actual node IPs.

STEP 7 — Deploy Prometheus

Execute the below on the Manager Node:
docker service create \
  --name prometheus \
  --network monitoring \
  --publish 9090:9090 \
  --constraint 'node.role == manager' \
  --mount type=bind,source=$HOME/prometheus/prometheus.yml,target=/etc/prometheus/prometheus.yml \
  prom/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.enable-lifecycle

Verify:
docker service ls
Should show:
Example Numbers:
prometheus              replicated   1/1
cadvisor                global       n/n
node-exporter           global       n/n
docker-swarm-exporter   replicated   1/1
STEP 8 — Verify in Browser
Open: `http://<manager-ip>:9090`
Go to: Status → Targets

You should see all targets as UP:

cadvisor                         n/n   UP
node-exporter              n/n   UP
docker-swarm-exporter   n/n   UP
docker-engine           n/n   UP

All targets will carry the unified labels:

instance="crc"   job="dockerswarm"   exporter="<exporter-type>"



Troubleshooting

| Issue                                           | Command                                                                                   | What to Look For |

| Service not starting                   | `docker service logs prometheus`                                            | `error loading config` |
| Target showing DOWN             | `http://<ip>:9090/targets`                                                        | Error message next to target |
| DNS not resolving tasks            | `docker exec -it <prometheus_id> nslookup tasks.cadvisor` | Should return IP |
| Wrong instance label                 | `http://<ip>:9090/targets`                                                         | Check labels column |
| Docker Engine metrics missing | `curl http://localhost:9323/metrics`                                         | Should return metric lines |
| Swarm Exporter unreachable    | `curl http://localhost:9000/metrics`                                         | Should return swarm metrics |

----------------------------------------------------------------------------------------------------
How to Configure it in External Prometheus: 
 Step 1 – Enable Docker Engine Metrics
Docker does NOT expose metrics by default.
Edit daemon.json on every swarm node:
  1. sudo vi /etc/docker/daemon.json
Add:
  1. {
  2.   "metrics-addr": "0.0.0.0:9323",
  3.   "experimental": true
  4. }

Test:
http://<node-ip>:9323/metrics

Step 2 – Deploy Node Exporter (for Node Metrics)
Deploy globally in swarm:
  1. docker service create \
  2.   --name node-exporter \
  3.   --mode global \
  4.   --mount type=bind,source=/proc,target=/host/proc \
  5.   --mount type=bind,source=/sys,target=/host/sys \
  6.   --mount type=bind,source=/,target=/rootfs \
  7.   --publish 9100:9100 \
  8.   prom/node-exporter:latest \
  9.   --path.procfs=/host/proc \
  10.   --path.sysfs=/host/sys \
  11.   --path.rootfs=/rootfs
Now accessible at:
  1. http://<node-ip>:9100/metrics

Step 3 – Deploy cAdvisor (Container Metrics)
  1. docker service create \
  2.   --name cadvisor \
  3.   --mode global \
  4.   --mount type=bind,source=/,target=/rootfs,readonly \
  5.   --mount type=bind,source=/var/run,target=/var/run \
  6.   --mount type=bind,source=/sys,target=/sys,readonly \
  7.   --mount type=bind,source=/var/lib/docker,target=/var/lib/docker,readonly \
  8.   --publish 8080:8080 \
  9.   gcr.io/cadvisor/cadvisor:latest
Metrics:
  1. http://<node-ip>:8080/metrics
Step 4 – Configure External Prometheus
Now in your external Prometheus server, edit:
  1. vi prometheus.yml

Example config:

  1. global:
  2.   scrape_interval: 15s

  4. scrape_configs:

  6.   - job_name: 'docker-engine'
  7.     static_configs:
  8.       - targets:
  9.           - 'NODE1_IP:9323'
  10.           - 'NODE2_IP:9323'

  12.   - job_name: 'node-exporter'
  13.     static_configs:
  14.       - targets:
  15.           - 'NODE1_IP:9100'
  16.           - 'NODE2_IP:9100'

  18.   - job_name: 'cadvisor'
  19.     static_configs:
  20.       - targets:
  21.           - 'NODE1_IP:8080'
  22.           - 'NODE2_IP:8080'

Restart Prometheus. and check in port 9090.

Check:

  • Status → Targets
  • All should be UP ✅
Step 5 – Useful PromQL Queries (Check all these queries in PromQL)

  1. engine_daemon_engine_info
  2. container_last_seen
  3. engine_swarm_service_tasks_running
  4. 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
  5. container_memory_usage_bytes



      New to M365 Manager Plus?
      New to M365 Manager Plus?
        New to RecoveryManager Plus?
        New to RecoveryManager Plus?
          New to Exchange Reporter Plus?
          New to Exchange Reporter Plus?
            New to SharePoint Manager Plus?
            New to SharePoint Manager Plus?
                See all integrations
                New to ADManager Plus?
                See all integrations
                New to ADManager Plus?

                  New to ADSelfService Plus?

                  Start your free trial
                  Start your free trial

                    • Related Articles

                    • How to configure Prometheus inside Openshift to scrape necessary metrics for Applications Manager?

                      To setup Prometheus inside a Openshift Cluster and scrape metrics for monitoring the same in Applications Manager, kindly follow the steps given below: Create a new Project "monitor" inside the Desired Cluster to be monitored using the below command. ...

                    • REST API Monitor Troubleshooting Guide

                      Whether you're adding a new REST API monitor or troubleshooting an existing one, the following steps can help resolve common issues. Troubleshooting 4xx Error Codes (e.g., 401, 403) Check Request Configuration: Verify that the correct HTTP method ...

                    • Service Now Event Integration using Webhook / Rest API Action

                      Steps to perform in Service Now 1) Login to your ServiceNow Instance(dev*****133.service-now.com) 2) Search for the Rest API Explorer and open it 3) In the Rest API Explorer page choose the Namespace and API Name with the proper API version. For ...

                    • REST API Monitor - FAQS

                      1. What to do when Basic Authentication fails in REST API monitor? When Basic Authentication fails in the REST API monitor, follow the below steps to troubleshoot the error. Ensure the credentials provided in Applications Manager (Username and ...

                    • How to enable docker remote API on docker host?

                      Navigate to /lib/systemd/system in your terminal and open docker.service file vi /lib/systemd/system/docker.service Find the line which starts with ExecStart and adds -H=tcp://0.0.0.0:4243 to make it look like ExecStart=/usr/bin/dockerd -H=fd:// ...

                      Related Products