Physical memory utilization
Sample Output:
total used free shared buff/cache available
Mem: 16338247680 6590242816 1372995584 662192128 8375009280 8734982144
Swap: 100 10 0
Calculation:
Physical memory % = 100 -(((MemFree + Buffers + Cached)*100)/maxValue) =100-((1372995584 + 8375009280 + 8375009280)*100/16338247680)
swap memory % = (10/100)*100
Sample Output:
total used free shared buff/cache available
Mem: 16338247680 7217897472 537305088 593297408 8583045120 8176562176
Swap: 0 0 0
Calculation:
physical memory MB= maxValue-(MemFree + Buffers + Cached) / (1024*1024)= 16338247680 - (537305088+593297408+8583045120)/(1024*1024)
swap memory MB = Used/1024*1024
Thread Count
Command:
- ps -eo nlwp | awk '{ threadcount += $1 } END { print threadcount }' = 2555
Disk Utilization
Command:
- /bin/df -Pm |awk '{print $(NF-3) " " $(NF-2) " " $(NF-1) " " $NF}'
Sample Output:
Available Capacity Mounted on
0 7756 0% /dev
3 1556 1% /run
18003 77760 19% /
332 7460 5% /dev/shm
1 5 1% /run/lock
0 7791 0% /sys/fs/cgroup
5 0 100% /snap/gnome-calculator/406
Disk IO Stats
Command:
- LANG=C iostat -d;echo '-----DISK_EXTENDED_STATS-----';iostat -d -x 1 3
Sample Output:
Linux 4.15.0-88-generic (dinesh-7897) 03/02/20 _x86_64_ (12 CPU)
Device tps kB_read/s kB_wrtn/s kB_read kB_wrtn
loop0 0.00 0.01 0.00 328 0
loop1 0.00 0.01 0.00 128 0
loop2 0.01 0.02 0.00 500 0
loop3 0.00 0.00 0.00 46 0
loop4 0.00 0.01 0.00 328 0
loop5 0.02 0.03 0.00 800 0
loop6 0.01 0.05 0.00 1144 0
loop7 0.01 0.05 0.00 1289 0
nvme0n1 33.00 207.34 7747.66 5155956 192658293
dm-0 5.04 249.24 3540.62 6197808 88043200
-----DISK_EXTENDED_STATS-----
Linux 4.15.0-88-generic (test-user) Monday 02 March 2020 _x86_64_ (12 CPU)
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.11 0.00 0.00 9.11 0.00 0.00 0.00
loop1 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.08 0.00 0.00 2.42 0.00 0.00 0.00
loop2 0.01 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.16 0.00 0.00 2.44 0.00 0.02 0.00
loop3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.42 0.00 0.00 2.42 0.00 0.00 0.00
loop4 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.11 0.00 0.00 9.11 0.00 0.00 0.00
loop5 0.02 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 1.58 0.00 0.00 0.00
loop6 0.01 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 8.67 0.00 0.00 0.00
loop7 0.01 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 4.52 0.00 0.00 0.00
nvme0n1 4.32 28.69 207.35 7747.66 0.00 0.53 0.07 1.83 0.47 0.90 0.02 48.04 270.08 0.14 0.46
dm-0 1.54 3.49 249.24 3540.62 0.00 0.00 0.00 0.00 0.67 11.85 0.04 161.37 1013.68 0.58 0.29
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 0.00 25.00 0.00 6456.00 0.00 0.00 0.00 0.00 0.00 0.80 0.02 0.00 258.24 0.48 1.20
dm-0 0.00 25.00 0.00 11244.00 0.00 0.00 0.00 0.00 0.00 0.96 0.02 0.00 449.76 0.48 1.20
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 0.00 4.00 0.00 24.00 0.00 0.00 0.00 0.00 0.00 8.00 0.02 0.00 6.00 2.00 0.80
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Note: These are the attributes that we display in APM:
- avgqu-sz - Average queue length
- %util - % Busy Time
- r/s - Transfers/sec
- wkB/s - Writes/Sec
- rrqm/s - Reads/Sec
Inode Usage
Inode is the total count of files created or currently being viewed in a partition
Sample Output:
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 1985370 631 1984739 1% /dev
tmpfs 1994415 1279 1993136 1% /run
/dev/mapper/nvme0n1p1_crypt 0 0 0 - /
tmpfs 1994415 1112 1993303 1% /dev/shm
tmpfs 1994415 6 1994409 1% /run/lock
tmpfs 1994415 18 1994397 1% /sys/fs/cgroup
Calculation:
Total inodes=Used inodes+free inodes
Free%=100-used%
System Load(CPU load due to the jobs handled periodically by the CPU) = uptime
18:12:08 up 7:35, 1 user, load average: 0.77, 0.88, 0.89
jobsPerMin = 0.77*100
jobsPer5Min = 0.88*100
jobsPer15Min = 0.89*100
CPU Utilization
Command:
- /usr/bin/vmstat 1 3
Sample Output:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 701848 26156 8415072 0 0 16 84 61 50 4 2 94 0 0
0 0 0 700104 26156 8415584 0 0 0 0 3263 10345 4 1 95 0 0
0 0 0 700600 26156 8415828 0 0 0 0 2969 9716 2 2 96 0 0
UserTime=us=4
SystemTime=sy=2
WaitTime=wa
IdleTime=id
RunQueue=r
BlockedProcess=b
IntrsPerSec=in
CPU Core Utilization
Command:
- mpstat -P ALL 1 3
Sample Output:
Linux 4.15.0-88-generic (test-user) Monday 02 March 2020 _x86_64_ (12 CPU)
06:22:44 IST CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
06:22:45 IST all 5.74 0.00 2.53 0.00 0.00 0.08 0.00 0.00 0.00 91.64
06:22:45 IST 0 2.06 0.00 3.09 0.00 0.00 0.00 0.00 0.00 0.00 94.85
06:22:45 IST 1 1.03 0.00 1.03 0.00 0.00 0.00 0.00 0.00 0.00 97.94
06:22:45 IST 2 8.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 88.00
06:22:45 IST 3 2.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.94
06:22:45 IST 4 1.01 0.00 2.02 0.00 0.00 1.01 0.00 0.00 0.00 95.96
06:22:45 IST CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
06:22:46 IST all 1.93 0.00 1.85 0.00 0.00 0.08 0.00 0.00 0.00 96.13
06:22:46 IST 0 0.00 0.00 2.04 0.00 0.00 0.00 0.00 0.00 0.00 97.96
06:22:46 IST 1 1.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 96.00
06:22:46 IST 2 2.04 0.00 2.04 0.00 0.00 0.00 0.00 0.00 0.00 95.92
06:22:46 IST 3 2.02 0.00 2.02 0.00 0.00 0.00 0.00 0.00 0.00 95.96
06:22:46 IST 4 2.04 0.00 1.02 0.00 0.00 0.00 0.00 0.00 0.00 96.94
06:22:46 IST CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
06:22:47 IST all 1.44 0.00 1.19 0.00 0.00 0.00 0.00 0.00 0.00 97.37
06:22:47 IST 0 1.02 0.00 1.02 0.00 0.00 0.00 0.00 0.00 0.00 97.96
06:22:47 IST 1 1.02 0.00 1.02 0.00 0.00 0.00 0.00 0.00 0.00 97.96
06:22:47 IST 2 2.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.94
06:22:47 IST 3 1.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.97
06:22:47 IST 4 4.95 0.00 1.98 0.00 0.00 0.00 0.00 0.00 0.00 93.07
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 3.04 0.00 1.86 0.00 0.00 0.06 0.00 0.00 0.00 95.04
Average: 0 1.02 0.00 2.05 0.00 0.00 0.00 0.00 0.00 0.00 96.93
Average: 1 1.02 0.00 1.69 0.00 0.00 0.00 0.00 0.00 0.00 97.29
Average: 2 4.07 0.00 2.03 0.00 0.00 0.00 0.00 0.00 0.00 93.90
Average: 3 1.71 0.00 0.68 0.00 0.00 0.00 0.00 0.00 0.00 97.61
Average: 4 2.68 0.00 1.68 0.00 0.00 0.34 0.00 0.00 0.00 95.30
Uptime
Command:
- Uptime: commands - uptime|cut -d ',' -f1,2|tr -s ' ' '^'|cut -d '^' -f 2-
Sample Output:
18:37:26^up^8:00,^1^user
Uptime 18 days 37 hours 26 minutes
Process Count
Commands:
- LinuxProcessCount = /bin/ps -ef --no-headers | wc -l
- LinuxZombieProcessCount = /bin/ps -eo fname,pcpu,pmem,command|egrep '%CPU|defunct'
Sample Output:
COMMAND %CPU %MEM COMMAND
grep 0.0 0.0 grep -E --color=auto %CPU|defunct
xdg-open 0.0 0.0 [xdg-open] <defunct>
Process
Command:
- /bin/ps -eo fname,pcpu,pmem,command|cat;echo 'NUMBER_OF_PROCESSORS';cat /proc/cpuinfo|grep 'processor'|wc -l
Sample Output:
COMMAND %CPU %MEM COMMAND
systemd 0.0 0.0 /sbin/init splash
kthreadd 0.0 0.0 [kthreadd]
kworker/ 0.0 0.0 [kworker/0:0H]
mm_percp 0.0 0.0 [mm_percpu_wq]
ksoftirq 0.0 0.0 [ksoftirqd/0]
rcu_sche 0.3 0.0 [rcu_sched]
rcu_bh 0.0 0.0 [rcu_bh]
migratio 0.0 0.0 [migration/0]
watchdog 0.0 0.0 [watchdog/0]
cpuhp/0 0.0 0.0 [cpuhp/0]
cpuhp/1 0.0 0.0 [cpuhp/1]
watchdog 0.0 0.0 [watchdog/1]
Break-up of CPU Utilization for Linux servers in SNMP mode:
For attributes , User Time(%),System Time(%),Idle Time(%)
.1.3.6.1.4.1.2021.11.9.0 = INTEGER: 0 UCD-SNMP-MIB::ssCpuUser.0
.1.3.6.1.4.1.2021.11.10.0 = INTEGER: 0 UCD-SNMP-MIB::ssCpuSystem.0
.1.3.6.1.4.1.2021.11.11.0 = INTEGER: 99 UCD-SNMP-MIB::ssCpuIdle.0
For every value of above oids, the calculation is done as follows:
value = ((Current value for OID - Previous value for OID[From Previous Polling])/(totalRawVal-prevtotRawVal)) * 100
where totalRawVal is calculated as shown below:
totalRawVal=User+Nice+System+Idle+wait;
UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 127401 [.1.3.6.1.4.1.2021.11.50.0]
UCD-SNMP-MIB::ssCpuRawNice.0 = Counter32: 1693 [.1.3.6.1.4.1.2021.11.51.0]
UCD-SNMP-MIB::ssCpuRawSystem.0 = Counter32: 43464 [.1.3.6.1.4.1.2021.11.52.0]
UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 2291205 [.1.3.6.1.4.1.2021.11.53.0]
UCD-SNMP-MIB::ssCpuRawWait.0 = Counter32: 3558 [.1.3.6.1.4.1.2021.11.54.0]
.1.3.6.1.4.1.2021.11.7.0 = INTEGER: 2139 interrupts/s UCD-SNMP-MIB::ssSysInterrupts.0 = INTEGER: 2639 interrupts/s