Learn System Activity Report sar command with real-world scenario examples. Understand CPU, Memory reporting using this command.
In the last post of sar command, we have seen its data file structure, how to extract data from it, and time formats to be used with this command. In this post, let’s see how to get CPU, memory utilization reports from data files, or real-time using sar command.
Read other parts of sar tutorial :
- sar command : Introduction, datafiles and time formats (part I)
- sar command : Disk and network reporting (part III)
sar command follows below format :
# sar [ options ] [ <interval> [ <count> ] ]
We have already seen what is interval and count in the last post. Now we will see different options that can be used to get different system resource utilization stats. Also, we have seen how to get historic data from sar data files, I will be using only real-time commands (i.e. without -f option) for all below examples.
Before we start with resource reporting here is a quick tip about start and end time of reports when you are extracting data from datafiles. Below two options can be used with sar command (in conjunction with -f) so that specific time window data can be extracted.
-s hh:mm:ssStart time of the report. Sar starts output records tagged to this time or very next available time-tagged record. Default start time is 08:00:00-e hh:mm:ssThe end time of the report. The default end time is 18:00:00
CPU utilization reporting using sar
For CPU statistics, sar command has -u option. Executing sar command with -u gives us below utilization matrices (highlighted values are more commonly observed for performance monitoring) :
%user: CPU % used by user processes%nice: CPU % used by user processes with nice priority%system: CPU % used by system processes%iowait: % of the time when CPU was idle (since processes were busy in IO)%steal: % of time wait by virtual CPU while hypervisor servicing another CPU (virtualization aspect)%idle: CPU % idle.
# sar -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2) 12/20/2016 _x86_64_ (16 CPU)
03:34:51 AM CPU %user %nice %system %iowait %steal %idle
03:34:53 AM all 33.00 0.00 9.99 32.22 0.00 24.80
03:34:55 AM all 37.32 0.00 10.44 32.95 0.00 19.29
03:34:57 AM all 36.04 0.00 11.90 29.83 0.00 22.24
Average: all 35.46 0.00 10.78 31.66 0.00 22.11
See the above example to get the values of the parameters explained above. The output starts with a line that has OS kernel version details, hostname in brackets, date, architecture, and the total number of CPUs. Followed by a data with interval (2 sec) and count (3) specified in the command. Finally, it also gives us the average value for all parameters. Column CPU denoting value all indicates these are averaged out values of all 16 CPU for the given time instance.
If you are interested in seeing values for each processor then -P option (per processor reporting) can be used with CPU number of your choice or ‘ALL’. When ALL is specified each processor’s data is shown or only specified CPU’s data is processed.
# sar -P ALL -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsvr2) 12/20/2016 _x86_64_ (16 CPU)
04:08:33 AM CPU %user %nice %system %iowait %steal %idle
04:08:35 AM all 34.50 0.00 6.38 7.66 0.00 51.45
04:08:35 AM 0 32.66 0.00 9.55 4.52 0.00 53.27
04:08:35 AM 1 76.24 0.00 2.48 5.45 0.00 15.84
04:08:35 AM 2 24.62 0.00 10.05 6.53 0.00 58.79
04:08:35 AM 3 38.50 0.00 14.50 5.50 0.00 41.50
04:08:35 AM 4 3.05 0.00 4.06 0.51 0.00 92.39
04:08:35 AM 5 1.99 0.00 1.49 9.45 0.00 87.06
04:08:35 AM 6 99.00 0.00 1.00 0.00 0.00 0.00
04:08:35 AM 7 1.50 0.00 1.00 0.00 0.00 97.50
04:08:35 AM 8 62.00 0.00 13.00 13.50 0.00 11.50
04:08:35 AM 9 91.96 0.00 6.53 0.00 0.00 1.51
04:08:35 AM 10 34.67 0.00 10.55 18.09 0.00 36.68
04:08:35 AM 11 57.00 0.00 4.00 4.00 0.00 35.00
04:08:35 AM 12 11.50 0.00 5.50 20.50 0.00 62.50
04:08:35 AM 13 6.47 0.00 2.49 15.42 0.00 75.62
04:08:35 AM 14 3.00 0.00 2.00 3.50 0.00 91.50
04:08:35 AM 15 7.54 0.00 15.08 15.58 0.00 61.81
----- output clipped -----
See the above example where I mentioned ALL with -P option and got each processor’s utilization report. In the below example, only CPU number 2 data is extracted.
# sar -P 2 -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsvr2) 12/20/2016 _x86_64_ (16 CPU)
04:11:11 AM CPU %user %nice %system %iowait %steal %idle
04:11:13 AM 2 49.75 0.00 3.98 26.87 0.00 19.40
04:11:15 AM 2 97.50 0.00 1.50 1.00 0.00 0.00
04:11:17 AM 2 97.50 0.00 1.50 0.50 0.00 0.50
Average: 2 81.53 0.00 2.33 9.48 0.00 6.66
Another small stats regarding processor is power stats. Here sar shows you processor clock frequency at given instance of time. This helps in calculating power being used by CPU. -m option gives this data and it can be used per -processor reporting option too.
# sar -m 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2) 12/20/2016 _x86_64_ (16 CPU)
04:15:39 AM CPU MHz
04:15:41 AM all 1970.50
04:15:43 AM all 1845.81
04:15:45 AM all 1587.93
Average: all 1801.41
# sar -P ALL 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2) 12/20/2016 _x86_64_ (16 CPU)
04:15:52 AM CPU %user %nice %system %iowait %steal %idle
04:15:54 AM all 15.67 0.00 7.59 7.02 0.00 69.72
04:15:54 AM 0 20.00 0.00 7.00 37.50 0.00 35.50
04:15:54 AM 1 27.86 0.00 19.40 6.97 0.00 45.77
04:15:54 AM 2 47.50 0.00 15.50 2.50 0.00 34.50
04:15:54 AM 3 2.49 0.00 1.99 1.99 0.00 93.53
04:15:54 AM 4 3.02 0.00 5.03 1.01 0.00 90.95
04:15:54 AM 5 1.00 0.00 7.00 1.00 0.00 91.00
04:15:54 AM 6 0.51 0.00 0.51 0.00 0.00 98.97
04:15:54 AM 7 1.00 0.00 0.50 0.00 0.00 98.50
04:15:54 AM 8 35.18 0.00 21.61 20.10 0.00 23.12
04:15:54 AM 9 51.24 0.00 22.89 4.98 0.00 20.90
04:15:54 AM 10 28.64 0.00 8.04 14.57 0.00 48.74
04:15:54 AM 11 12.94 0.00 5.97 11.94 0.00 69.15
04:15:54 AM 12 8.50 0.00 3.50 7.50 0.00 80.50
04:15:54 AM 13 7.04 0.00 2.51 1.51 0.00 88.94
04:15:54 AM 14 1.01 0.00 0.51 1.52 0.00 96.97
04:15:54 AM 15 1.49 0.00 0.99 0.00 0.00 97.52
----- output clipped -----
Memory utilization reporting using sar
Memory stats can be extracted with -r option. When sar runs with -r option, it presents below parameters (highlighted values are more commonly observed for performance monitoring) :
kbmemfree: Free memory available in kilobytes.kbmemused: Memory used (excluding kernel usage)%memused: Percentage of memory usedkbbuffers: memory used as buffers by the kernel in kilobyteskbcached: memory used to cache data by the kernel in kilobytes- kbcommit: memory in kilobytes needed for the current workload. (commitment!)
%commit: % of memory needed for the current workload in relation to the total memory (RAM+swap)
# sar -r 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2) 12/20/2016 _x86_64_ (16 CPU)
03:42:07 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
03:42:09 AM 119133212 145423112 54.97 1263568 109109560 106882060 32.23
03:42:11 AM 119073748 145482576 54.99 1263572 109135424 107032108 32.27
03:42:13 AM 119015480 145540844 55.01 1263572 109162404 106976556 32.25
Average: 119074147 145482177 54.99 1263571 109135796 106963575 32.25
The output above shows the parameter values. It sections the same as explained above (CPU report example); first-line details, last avg row, etc. Note that, %commit can be 100%+ too since kernel always over-commit to avoid out of memory situation.
Paging statistics can be obtained using -B option. Normally, parameters shown in this option’s output are not observed by sysadmin. But if in-depth troubleshooting or monitoring is required then only this option is used. It shows the below parameters:
pgpgin/s: Number of kilobytes the system paged in from disk per second.pgpgout/s: Number of kilobytes the system paged out to disk per second.fault/s: Number of page faults per second.majflt/s: Number of major page faults per second.pgfree/s: Number of pages placed on the free list by the system per second.pgscank/s: Number of pages scanned by thekswapddaemon per second.pgscand/s: Number of pages scanned directly per second.pgsteal/s: Number of pages the system has reclaimed from cache per second.%vmeff: This is a metric of the efficiency of page reclaim.
# sar -B 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2) 12/21/2016 _x86_64_ (4 CPU)
12:59:41 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
12:59:43 AM 3.05 40.10 120411.68 0.00 99052.28 0.00 0.00 0.00 0.00
12:59:45 AM 2.99 7.46 42649.75 0.00 37486.57 0.00 0.00 0.00 0.00
12:59:47 AM 3.00 47.50 43.00 0.00 80.50 0.00 0.00 0.00 0.00
Average: 3.01 31.61 54017.22 0.00 45257.86 0.00 0.00 0.00 0.00
Swap statistics can be obtained with -S option. Swap is another aspect of memory hence its utilization monitoring is as important as memory. Swap utilization reports are shown with -S option. It shows below parameters (highlighted values are more commonly observed for performance monitoring) :
kbswpfree: Free swap in kilobyteskbswpused: Used swap in kilobytes%swpused: % of swap usedkbswpcad: Amount of cached swap memory in kilobytes%swpcad: % of cached swap memory in relation to used swap.
# sar -S 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2) 12/21/2016 _x86_64_ (4 CPU)
01:01:45 AM kbswpfree kbswpused %swpused kbswpcad %swpcad
01:01:47 AM 8388604 0 0.00 0 0.00
01:01:49 AM 8388604 0 0.00 0 0.00
01:01:51 AM 8388604 0 0.00 0 0.00
Average: 8388604 0 0.00 0 0.00
In the above example, you can see a total of 8GB swap available on server and nothing of it used. The swap will get hits only if memory gets completely utilized.
This concludes the second part of the sar tutorial. We will be seeing network-related reporting in the next part.
Share Your Comments & Feedback: