Category Archives: Linux

Difference between LVM and LVM2: Linux interview question explained

Leave a reply

Post stating point-wise differences between LVM and LVM2 explained. This is one of the frequently asked questions for the Linux interview.

One of the frequently asked Linux interview questions is ‘what is the difference between LVM and LVM2?’ to which most of the candidate’s flanks. In this article, we will be discussing this question.

First of all, what is LVM? LVM is a logical volume manager. It is a widely used volume manager in Linux and Unix like VxVM (Veritas volume manager). As a sysadmin, you must be knowing all tasks which can be done in LVM but not a theoretical question like the difference between LVM and LVM2.

LVM and LVM2 are two different versions of LVM. Obviously, LVM2 being the latest. Their differences can be identified in terms of their functions, services offered, availability, etc. Let’s see one by one :

Availability :

LVM is available in most of the famous distributions like HPUX, Redhat, Suse, etc. LVM2 is available only kernel version 2.6.9 and above like RHEL4 and later. You can even build LVM modules as per your kernel (if its not available in your distributions). Setup information can be found here.

Compatibility :

LVM2 has backward compatibility with LVM. It has the same command modules and infrastructure like LVM with enhanced features. LVM being the previous version don’t have compatibility with LVM2 features.

Size limits :

Different size limits are increased in the new LVM2. The maximum file system/device size was capped to 2TB in LVM whereas its 16TB(32-bit CPU) & 8EB (64 bit CPU) in LVM2.

Max LV and PV were 255 in LVM. In LVM2 it has no limit (snippet from vgcreate manpage below).

 -l, --maxlogicalvolumes MaxLogicalVolumes
              Sets the maximum number of logical volumes allowed in this  vol-
              ume  group.  The setting can be changed with vgchange.  For vol-
              ume groups with metadata in lvm1 format, the limit  and  default
              value  is  255.   If  the metadata uses lvm2 format, the default
              value is 0 which removes this  restriction:  there  is  then  no
              limit.

 -p, --maxphysicalvolumes MaxPhysicalVolumes
              Sets  the  maximum number of physical volumes that can belong to
              this volume group.  The setting can be  changed  with  vgchange.
              For  volume  groups  with metadata in lvm1 format, the limit and
              default value is 255.  If the metadata  uses  lvm2  format,  the
              value  0  removes  this restriction: there is then no limit.

Extra features :

LVM2 is bundled with below extra features over LVM :

  1. Volume mirroring support
  2. Shared volume support with GFS
  3. Transnational metadata for fast recovery
  4. Cluster suite failover supported
  5. Striped volume expansion

This sums up the difference between LVM and LVM2 versions. Write to us in comments if you have corrections/suggestions.

Recover forgotten root password in RHEL with screenshots

1 Reply

Learn how to recover forgotten root password in Linux RHEL. Step by step procedure to reset the root password by booting system in single-user mode.

There are many times when we set the root password too complex and later on forgot it. Many people used to work on consoles with no timeout values. In such cases, when they logged out of the console or reboot system, they most of the times forgot the root password to log in.

So it becomes extremely important to have one more account with root privileges on the system. Normally in corporate environments, they do keep such 2-3 accounts so that forgetting root password shouldn’t stop operations. But like in test environments or labs sometimes these practice is not followed and it becomes necessary to know how to recover forgotten root password.

A root password can not be recovered once forgotten! It can only be reset to new value! Now the question is if you don’t know the root password how will you reset it when you won’t able to login in the first place. The answer is the single-user mode!

Yes, to reset root password (when you forgot it) you need to boot the server into single-user mode. Why single user mode? Simple single user mode boots and present your root prompt without any requirement to log in! Simple eh? So resetting forgotten root password follows simple steps :

  1. Edit boot kernel path with single-user mode parameter (GRUB)
  2. Boot kernel into single user mode
  3. Reset root password
  4. Get into normal multi-user run-level

Lets see all these steps below with screenshots :

1. Edit boot kernel path

Restart server with the power button and halt the boot process by pressing any key when you see press any key to interrupt boot message on the console.

If you have installed more than one OS on your system, you will see more than one entries in the above screen. Use arrow keys to select (highlight) your kernel. If your bootloader is protected with password then press “p” you will be asked for a password if not then just press enter. After entering the password or pressing enter, you will be presented with the next screen:

Here, press the “e” key to edit the selected entry. You will be presented with a kernel line selection screen like below.

Select the kernel line and press “e” again to edit that entry. Now on new screen append letter S or single at end of the line so instruct the kernel to boot into single-user mode.

2. Boot kernel into single user mode

After typing above entry, press ‘enter’ key to save and you will back to the previous screen. Select kernel and press “b” to boot this edited kernel. That’s it! your server is booting in a single-user mode which will give you access to the system without the need of login.

Observe that, you haven’t asked for a password, and you are presented with root prompt!

3. Reset root password

Do I have to write anything for it? You have a root prompt already! go change your root password like you normally do.

# passwd root

Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

You can also check here that you are in single-user mode by checking run level with who command.

4. Get into multi user mode

You can do it by rebooting the system. But you need to go through all the above steps again and remove the single user mode entry you made previously. Or else the server will boot into again single-user mode.

Or you can simply type “init 3” to change your run-level from single to multi-user mode!

You have successfully reset the forgotten root password to a new one. Keep a new password in a safe place and don’t forget again! Since forgetting takes your almost 10-20 mins of downtime.

Ulimit value : All you need to know

Leave a reply

Understand what is ulimit? How to set it? Which all system resources can be limited using ulimit control? and how to view current ulimit settings.

In this article, we are going to see everything about bash built-in ulimit value. This is your key to keep the system safe from fork bombs or malicious codes aimed at hung systems by crunching resources.

What is ulimit?

It can roughly be called a user limit! Using this value you are limiting shell and its forked processes to use certain defined system resources. This helps in managing system resources and in turn processes efficiently. Using you can make sure that all-important processes on the server always get resources while the least important once cant hog more than what they should get. There are different parameters can be defined under ulimit umbrella which we will see ahead.

To view your current ulimit setting run below command :

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 95174
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

As you can see in the above output, the left column denotes parameters that can be limited using ulimit, along with their measuring unit and option to be used in braces and the last column shows current set value.

ulimit controlled parameters :

See below list of parameters that can be limit using ulimit and their details. The list is from man page and parameters are self-explanatory. Since this is bash built-in; when you check the manpage, you will see all bash commands. You have to scroll that man page all the way to bottom (since its alphabetically sorted) to get to ulimit section. There you will find these parameters.

Different ulimit parameters

OptionParameter
-a
 All current limits are reported
-b
 The maximum socket buffer size
-c
 The maximum size of core files created
-d
 The maximum size of a process’s data segment
-e
 The maximum scheduling priority (“nice”)
-f
 The maximum size of files written by the shell and its children
-i
 The maximum number of pending signals
-l
 The maximum size that may be locked into memory
-m
 The maximum resident set size (many systems do not honor this limit)
-n
 The maximum number of open file descriptors (most systems do not allow this value to be set)
-p
 The pipe size in 512-byte blocks (this may not be set)
-q
 The maximum number of bytes in POSIX message queues
-r
 The maximum real-time scheduling priority
-s
 The maximum stack size
-t
 The maximum amount of cpu time in seconds
-u
 The maximum number of processes available to a single user
-v
 The maximum amount of virtual memory available to the shell
-x
 The maximum number of file locks
-T
 The maximum number of threads

To set specific parameter limit values, you can issue the command :

# ulimit -option <value>

Once done, it will limit this parameter for the current shell (shell from where the command was run) and it’s all forked processes. A more efficient way to implement limits is through profiles that are discussed next.

How to setup ulimit :

The most common use in corporate Infra is for database servers. Since we all know that DB is resource-hungry application. So many times ulimits specified to it in terms of  -p or -n etc. This setting is done in DB owner user id like Oracle’s (user id with which DB application runs on the server) .bash_profile in its home or /etc/profile or through custom scripts which loads when DB starts. Find below code snippet which can be used in /etc/profile:

if [ $USER = "oracle" ] || [ $USER = "oradb" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 10000 -n 35000
fi
fi

Whenever users logged in, /etc/profile gets executed. It checks if the user is oracle or oradb and if yes with Ksh shell, it sets respective ulimits values for it!

This ensures parameters are set for that user’s shell when DB application starts in the user’s shell. Since,ulimit limits resources for shell and its forked processes, these values get imposed for DB apps running under that user’s shell!

How to zip, unzip files and directories in Linux / Unix

Leave a reply

Learn how to zip, unzip files and directories in Linux or Unix. Compressing files helps in log management and makes data transfer easy.

Zipping files or directories compress data within them using  Lempel-Ziv coding (LZ77). This reduces the size of the resulting file. Lower size means lower storage requirement (log management) and faster transfer (FTP).

In Linux or Unix platforms gzip is widely available utility mostly native to OS which is used to zip, unzip files. In this post, we will see how to zip and unzip files using gzip utility with examples.

Compressing files

Zipping files i.e compressing is achieved by gzip without any option. You need to submit filename.xyz to gzip command. It will compress the file and the resulting file will have a name as filename.xyz.gz Point here to note is gzip removes the original file and keep new gz file in place.

# ll
total 12
-rw-r--r-- 1 root users  40 Jan  3 00:46 file2

# gzip file2

# ll
total 12
-rw-r--r-- 1 root users  63 Jan  3 00:46 file2.gz

Note in the above output after zipping original file file2 is vanished from the system and compressed archive file2.gz came in existence.

This command support wildcards like * or ? too. Also, you can supply a list of files to it and it will compress all the files supplied in the argument.

# gzip file2 file3

# ll
total 12
-rw-r--r-- 1 root users 63 Jan  3 00:46 file2.gz
-rw-r--r-- 1 root users 134 Jan  3 00:46 file3.gz

You can use forceful operation with -f option. This is helpful in case files to be compressed has multiple links in existence.

gzip also supports -v option i.e. verbose mode which shows all details about the operation being done.

# gzip -v *
file1:   45.7% -- replaced with file1.gz
file2:   22.5% -- replaced with file2.gz
file3:   10.5% -- replaced with file3.gz

In the above example, we zipped all files (hence *) within a directory using wild cards. Here verbose mode printed compression ratio for each file along with which file it replaced after the operation.

Compressing directories

Like files, even directories can be compressed recursively. When -r option is used, gzip command read through the given directory to its subtree structure and zips all the files it founds within.

# ll /tmp/dir3
total 12
-rw-r--r-- 1 root users  35 Jan  3 00:46 file1
-rw-r--r-- 1 root users  40 Jan  3 00:46 file2
-rw-r--r-- 1 root users 114 Jan  3 00:46 file3

# gzip -r /tmp/dir3

# ll /tmp/dir3
total 12
-rw-r--r-- 1 root users  51 Jan  3 00:46 file1.gz
-rw-r--r-- 1 root users  63 Jan  3 00:46 file2.gz
-rw-r--r-- 1 root users 134 Jan  3 00:46 file3.gz

In the above output, it recursively zipped all files within a given directory. This is a helpful option where there are hundreds of files in the directory.

Checking compressed files

To test the compressed archive you can use -t option with gunzip. If there are any issues with the compressed files it will report or else it will return you shell prompt.

# gzip -t file2.gz

You can even view compression details of this file using -l option with gunzip command. It shows, uncompressed and compressed size, compression ratio (0% if not known) and name of uncompressed file i.e. filename before compression

# gzip -l file2.gz
         compressed        uncompressed  ratio uncompressed_name
                 63                  40  22.5% file2

Un-Compressing files

To gain the original file from a compressed archive, -d option needs to be used and gz file to be supplied in the argument. It works vice versa and removes gz file and keeps the original file in the directory here. Recursive -r option works with -d too.

# gzip -d file2.gz

# ll
total 12
-rw-r--r-- 1 root users  40 Jan  3 00:46 file2

You can see file2 is back available now and the gz file has been removed. Using verbose mode prints more information about operations being done.

# gzip -v -d *
file1.gz:        45.7% -- replaced with file1
file2.gz:        22.5% -- replaced with file2
file3.gz:        10.5% -- replaced with file3

In the above output, we de-compressed three files in the same directory using wildcard *. It shows the compression ratio with which file replaced which file after decompression.

How to change process priority in Linux or Unix

Leave a reply

Learn how to change process priority in Linux or Unix. Using the renice command understands how to alter the priority of running processes.

Processes in the kernel have their own scheduling priorities and using which they get served by the kernel. If you have a loaded system and need to have some processes serve before others you need to change process priority. This process is also called renicing since we use renice command to change process priority.

There are nice values defined from 0 to 20. 20 being the highest. The process with low nice values gets service before the process with a high nice value. So if you want a particular process to get served first you need to lower its nice value. Administrators may also choose to mark negative nice value (down up to -20) to speed up processes furthermore. Let’s see how to change process priority –

There are 3 ways to select a target for renice command. Once can submit process id (PID) or user ID or process group ID. Normally we use PID and UID in the real-world so we will see these options. New priority or nice value can be defined with option -n. Current nice value can be viewed in top command under NI column Or it can be checked using below command :

# ps -eo pid,user,nice,command | grep 30411
30411 root       0 top
31567 root       0 grep 30411

In above example, nice value is set to 0 for give PID.

Renice process id :

Process id can be submitted to renice using -p option. In below example we will renice value to 2 for pid 30411.

# renice -n 2 -p 30411
30411: old priority 0, new priority 2
# ps -eo pid,user,nice,command | grep 30411
  747 root       0 grep 30411
30411 root       2 top

renice command itself shows old and new nice values in output. We also verified new nice value using ps command.

Renice user id :

If you want to change priorities of all processes of a particular user then you can submit UID of that user using -u option. This option is useful when you want all processes by the user to complete fast, so you can set the user to -20 to get things speedy!

# ps -eo pid,user,nice,command | grep top
 3859 user4   0 top
 3892 user4   0 top
 4588 root    0 grep top
# renice -n 2 -u user4
54323: old priority 0, new priority 2
# ps -eo pid,user,nice,command | grep top
 3859 user4   2 top
 3892 user4   2 top
 4966 root    0 grep top

In the above example, there are two processes owned by user4 with priority 0. We changed the priority of user4 to 2. So both processes had their priority changed to 2.

Normal users can change their own process priority too. But he can not override priority set by the administrator. -20 is the lowest minimum nice value one can set on the system. This is the speediest priority, once set that process gets service and all available resources on the system to get its task done.

5 different examples to send email through Linux terminal

Leave a reply

Five different examples to show you how to send an email using the Linux terminal. Learn how to send an email with the subject, attachment, and body from the terminal.

We have seen how to configure SMTP in Linux, in this post we will see different examples to send a mail through terminal. This post will elaborate on how to attach a file to mail, how to add a subject line, how to add body content to mail-in command-line interface.

Also read :

First of all, you need to check if SMTP is configured correctly and you are able to send mail from your Linux server. You can test it by sending test mail like below :

# echo test |sendmail -v info@xyz.com
[<-] 220 mailserver.xyz.com ESMTP Postfix
[->] HELO testsrv2
[<-] 250 mailserver.xyz.com
[->] MAIL FROM:<root@xyz.com>
[<-] 250 2.1.0 Ok
[->] RCPT TO:<info@xyz.com>
[<-] 250 2.1.5 Ok
[->] DATA
[<-] 354 End data with <CR><LF>.<CR><LF>
[->] Received: by testsrv2 (sSMTP sendmail emulation); Fri, 23 Dec 2016 02:29:07 +0800
[->] From: "root" <root@xyz.com>
[->] Date: Fri, 23 Dec 2016 02:29:07 +0800
[->] test
[->]
[->] .
[<-] 250 2.0.0 Ok: queued as 19F75822B8
[->] QUIT
[<-] 221 2.0.0 Bye

Once you receive test mail, it’s confirmed that your SMTP configurations are correct. Let’s see different examples to send mail.

1. An email with the subject line :

Sendmail is low-level utility hence it doesn’t support -s option for a subject line like other email utilities do. We have to stdin subject to sendmail like below :

# echo "Subject : test" |sendmail -v user4@xyz.com
user4@xyz.com... Connecting to mailserver.xyz.com via relay...
220 mailserver.xyz.com ESMTP Postfix
>>> EHLO server2.xyz.com
250-mailserver.xyz.com
250-PIPELINING
250-SIZE 25600000
---- output clipped -----

Text following “Subject:” will be treated as the subject line for mail.

You can also use mailx command with -s option to specify the subject line.

# mailx -s "test email subject line"  user4@xyz.com
Your email body goes here... type and hit ctrl+d
Cc:

2. An email with mail body :

If you have a long email body then you can save it in a file. Then you can pass that file to sendmail command. Sendmail will parse the file and the content of files will be seen in the body of the email.

# cat /tmp/mail_body.txt | sendmail -v user4@xyz.com
user4@xyz.com... Connecting to mailserver.xyz.com via relay...
220 mailserver.xyz.com ESMTP Postfix
>>> EHLO server2.xyz.com
250-mailserver.xyz.com
250-PIPELINING
250-SIZE 25600000
250-VRFY

The same format works with mailx command too. You can even source file using redirector for mail body like below:

# mailx user4.xyz.com < /tmp/mail_body.txt

3. Email with attachment

Sending attachment with sendmail/mail is a bit tricky. We need to encode files before sending as an attachment using uuencode.

# uuencode /tmp/attachement.txt | mail -s "Subject line" user4.xyz.com

4. An email with different FROM field

You can even change from the field of the email so that you can send an email with nicer names in the FROM field rather than ugly account names which mail utility picks up from your account details. There are few options when one is not available on your system to try another.

# mailx -r from_sender_id@xyz.com -s "subject" user4@xyz.com

# mailx -s "subject" user4.xyz.com -- -f from_sender_id@xyz.com 

# mailx -s "subject" -a "From: Sender <from_sender_id@xyz.com>" user4.xyz.com

5. Adding CC and BCC fields:

You can add BC and BCC fields too. Using options -c cc and -b for bcc:

# mail -s "test subject" -c abc@xyz.com -b pqr@xyz.com user4@xyz.com

How to add a subject line in sendmail command

You can add an email subject line in sendmail by sending “Subject: xxxxx” to sendmail command. See below examples –

(echo "Subject: Test email"; cat mailbody.txt) | sendmail -v info@kerneltalks.com

Errors and logfile for email in Linux

Once you send an email check /var/mail/maillog file for errors or success messages. If your configuration is correct and everything is working as expected you can see below message in there.

Sep 17 23:44:38 testsrv postfix/qmgr[10290]: 1086212AA0C: from=<root@testsrv>, size=470, nrcpt=1 (queue active)
Sep 17 23:44:38 testsrv postfix/smtp[17001]: 1086212AA0C: to=<info@kerneltalks.com>, relay=smtp.kerneltalks.com[10.10.1.1]:25, delay=4193, delays=4193/0.02/0.03/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 66AE24043)
Sep 17 23:44:38 testsrv postfix/qmgr[10290]: 1086212AA0C: removed

It means your message is successfully sent to the SMTP server. If you still did not receive the messages in your mailbox then you should check with the SMTP server owner that if they have valid email address restriction. Since if you don’t configure sender address then command by default sent an email with TO address as username@hostname

You can use -r option in mailx command to define TO email address as well which matches the policy of the SMTP server.

# echo test | mailx -s "Test from `hostname`" -r root@kerneltalks.com -S smtp="smtp.kerneltalks.com" info@kerneltalks.com

Like we have defined the sender address as root@kerneltalks.com in the above test command. Now, this email will be accepted by the SMTP server and will be delivered to the recipient.

Another error can be seen in maillog is as below –

Sep 17 23:07:33 testsrv postfix/smtp[9918]: 1086212AA0C: to=<info@kerneltalks.com>, relay=none, delay=1968, delays=1968/0.01/0.26/0, dsn=4.3.5, status=deferred (Host or domain name not found. Name service error for name=smtp.kerneltalks.com type=A: Host not found)

Solution: type A record, the host is not found means server is unable to resolve SMTP server name from DNS. Make sure you have DNS configured in /etc/resolv.conf and SMTP name is registered in DNS. You can verify DNS resolution and DNS server being used using nslookup smtp.kerneltalks.com command.

One more same error but for record type AAAA can be seen as below :

Sep 15 22:41:04 testsrv postfix/smtp[7833]: 97E9C12A7D9: to=<info@kerneltalks.com>, relay=none, delay=349010, delays=349010/0.01/0.26/0, dsn=4.3.5, status=deferred (Host or domain name not found. Name service error for name=smtp.kerneltalks.com type=AAAA: Host not found)

Solution: This means the server is looking for an IPv6 record for the SMTP server. Edit /etc/postfix/main.cf file and remove all protocols and define only ipv4 and you will be good.

# vi /etc/postfix/main.cf
inet_protocols = ipv4

3 tricks to get multiple commands output in the same row

Leave a reply

Three tricks to get two commands output in a single row. Normally two commands outputs will be always separated by a carriage return in Linux terminal. 

One of the major hurdles in scripting is to get your outputs formatted well according to requirements. In Linux/Unix each command stdout its output always in a new line. This is a hurdle in many situations where the coder wants two outputs, two terms, two variables in a single row for example CSV format. In this post, we will see how to present two or more commands output in a single row.

Read also : Scripting related posts

Normally when we execute more than one command, they will display outputs in different rows:

# date ; hostname
Sat Dec 24 01:35:50 EDT 2016
testsrv2

# echo text1; echo text2; echo text3
text1
text2
text3

In the above example, commands have outputs per row. We are going to accomplish all outputs in one row using the below tricks.

Trick 1:

Using the translate function. AS I said earlier each command output is accompanied by a carriage return. We will be translating carriage return with tab character hence getting two or more commands output in a single row.

# (date; hostname) |tr '\n' '\t'
Sat Dec 24 01:36:22 EDT 2016    testsrv2

# (echo text1; echo text2; echo text3) |tr '\n' '\t'
text1   text2   text3

Bypassing outputs to translate function we achieved single row output. Here we instructed tr function to translate all carriage returns (\n) to tab (\t).

If you want to output in CSV format (comma-separated value) then replace tab with a comma and you will get it.

# (date; hostname) |tr '\n' ','
Sat Dec 24 01:43:09 EDT 2016,testsrv2,

# (echo text1; echo text2; echo text3) |tr '\n' ','
text1,text2,text3,

You can observe one trailing comma which shows every command output ends with \n and hence it gets replaced by ,

Trick 2:

By defining each command output as a variable. Here we will store the output of each command in one variable and then we will echo those variables in a single line.

# mydate=`date`

# myname=`hostname`

# echo $myname $mydate
testsrv2 Sat Dec 24 01:46:04 EDT 2016

Here we stored date and hostname command’s output in mydate and myname variables. In the last command, we echoed both variables with space in between. Note that commands output can be stored in a variable by executing the command using backticks (in-line execution)!

Trick 3:

By echoing in-line execution outputs. In-line execution i.e. using backticks to execute commands within another command. We will be using this trick to echo outputs in a single row.

# echo "`date` `hostname`"
Sat Dec 24 01:50:36 EDT 2016 testsrv2

In the above example, first, we have executed commands and got outputs. But those outputs are fed into echo command. Hence echo command stdout them in a single row.

# echo "`echo text1` `echo text2` `echo text3`"
text1 text2 text3

Make a note that we executed each command in separate back tick quotes.

sar command (Part III) : Disk, Network reporting

Leave a reply

Learn System Activity Report sar command with real-world scenario examples. Understand how to do disk, network reporting using this command.

In the last two parts of the sar command, we have seen time formats to be used with command, its data files, CPU & Memory reporting. In this last part, we will be seeing disk, network reporting using sar command.

Read last two parts of this tutorial here :

Disk IO reporting

sar provides disk (block devices)  report with -d option.  Normally, it shows below parameters values (highlighted values are more commonly observed for performance monitoring) :

  • DEV: Block device name. It follows the dev m-n format. M is major and n is a minor number of the block devices.
  • tps: Transfers per second
  • rd_sec/s: Sector reads per second (sector is 512 byte)
  • wr_sec/s: Sector writes per second 
  • avgrq-sz:  average size (in sectors) of the requests that were issued to the device
  • avgqu-sz: average queue length of the requests that were issued to the device
  • await: The average time (in milliseconds) for I/O requests
  • svctm: The average service time (in milliseconds) for I/O requests
  • %util: Percentage  of  CPU  time  during  which  I/O requests were issued to the device
# sar -d 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/21/2016      _x86_64_        (4 CPU)

01:20:19 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
01:20:21 AM    dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:21 AM   dev8-64      3.00      2.00      1.00      1.00      0.00      0.50      0.50      0.15
01:20:21 AM   dev8-32      3.00      2.00      1.00      1.00      0.00      0.50      0.33      0.10
01:20:21 AM  dev252-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:21 AM  dev252-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:21 AM  dev252-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

In the above output, device names are not so user friendly. So to identify devices easily, -p option is available. This option prints pretty device names in the DEV column and it should always be used with -d option.

# sar -d -p 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/21/2016      _x86_64_        (4 CPU)

01:20:38 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
01:20:40 AM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:40 AM       sdb      4.98      0.00     91.54     18.40      0.00      0.80      0.20      0.10
01:20:40 AM       sdc      2.99      1.99      1.00      1.00      0.00      0.67      0.67      0.20
01:20:40 AM      dm-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:40 AM      dm-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:40 AM      dm-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Now see above output where device names are easily identifiable. sda means disk /dev/sda and so on.

Network utilization reporting

Option -n gives all network stats. There are a total of 18 different keywords (like NFS, IP, DEV, TCP, etc.) which can be used with -n option to get related parameters. Each keyword has almost 8-10 parameters to display. So in the call, if you are using ALL keyword, then the output will be a huge list of parameters which is difficult to understand.

To keep it short here we will see only one example of keyword DEV i.e. device. This will show the device i.e. network card’s parameter values. Most of the time NIC performance is checked hence we are using this keyword example.

# sar -n DEV 2 1
Linux 2.6.39-200.24.1.el6uek.x86_64 (textsrv2)         12/21/2016      _x86_64_        (4 CPU)

01:35:22 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
01:35:24 AM        lo      6.00      6.00      0.29      0.29      0.00      0.00      0.00
01:35:24 AM      eth0     15.50      0.50      0.91      0.04      0.00      0.00      0.00
01:35:24 AM      eth1      6.50      4.50      0.97      0.77      0.00      0.00      0.00
01:35:24 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo      6.00      6.00      0.29      0.29      0.00      0.00      0.00
Average:         eth0     15.50      0.50      0.91      0.04      0.00      0.00      0.00
Average:         eth1      6.50      4.50      0.97      0.77      0.00      0.00      0.00
Average:         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

In the above example, I used the DEV keyword along with -n option and took only one iteration in output. Parameters displayed for device keyword are :

  • IFACE: Its interface name. You can easily see eth0, eth1, loopback (lo) interfaces here.
  • rxpck/s: Packets received per seconds
  • txpck/s: packets transmitted per second
  • rxkB/s: kilobytes received per second
  • txkB/s : kilobytes transmitted per second
  • rxcmp/s: compressed packets received per second
  • txcmp/s: compressed packets transmitted per second
  • rxmcst/s: Number of multicast packets received per second

This concludes sar command tutorial’s part III about the disk, network reporting. This is a three-part tutorial with example outputs included. Put your queries, suggestions, feedback in the comments below. You can also reach us using our Contact form. 

sar command (Part II) : CPU, Memory reporting

Leave a reply

Learn System Activity Report sar command with real-world scenario examples. Understand CPU, Memory reporting using this command.

In the last post of sar command, we have seen its data file structure, how to extract data from it, and time formats to be used with this command.  In this post, let’s see how to get CPU, memory utilization reports from data files, or real-time using sar command.

Read other parts of sar tutorial :

sar command follows below format :

# sar [ options ] [ <interval> [ <count> ] ]

We have already seen what is interval and count in the last post. Now we will see different options that can be used to get different system resource utilization stats. Also, we have seen how to get historic data from sar data files, I will be using only real-time commands (i.e. without -f option) for all below examples.

Before we start with resource reporting here is a quick tip about start and end time of reports when you are extracting data from datafiles. Below two options can be used with sar command (in conjunction with -f) so that specific time window data can be extracted.

  • -s hh:mm:ss Start time of the report. Sar starts output records tagged to this time or very next available time-tagged record. Default start time is 08:00:00
  • -e hh:mm:ss The end time of the report. The default end time is 18:00:00

CPU utilization reporting using sar

For CPU statistics, sar command has -u option. Executing sar command with -u gives us below utilization matrices (highlighted values are more commonly observed for performance monitoring) :

  • %user: CPU % used by user processes
  • %nice: CPU % used by user processes with nice priority
  • %system:  CPU % used by system processes
  • %iowait: % of the time when CPU was idle (since processes were busy in IO)
  • %steal: % of time wait by virtual CPU while hypervisor servicing another CPU (virtualization aspect)
  • %idle: CPU % idle.
# sar -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/20/2016      _x86_64_        (16 CPU)

03:34:51 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:34:53 AM     all     33.00      0.00      9.99     32.22      0.00     24.80
03:34:55 AM     all     37.32      0.00     10.44     32.95      0.00     19.29
03:34:57 AM     all     36.04      0.00     11.90     29.83      0.00     22.24
Average:        all     35.46      0.00     10.78     31.66      0.00     22.11

See the above example to get the values of the parameters explained above. The output starts with a line that has OS kernel version details, hostname in brackets, date, architecture, and the total number of CPUs. Followed by a data with interval (2 sec) and count (3) specified in the command. Finally, it also gives us the average value for all parameters. Column CPU denoting value all indicates these are averaged out values of all 16 CPU for the given time instance.

If you are interested in seeing values for each processor then -P option (per processor reporting) can be used with CPU number of your choice or ‘ALL’. When ALL is specified each processor’s data is shown or only specified CPU’s data is processed.

# sar -P ALL -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsvr2)         12/20/2016      _x86_64_        (16 CPU)

04:08:33 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:08:35 AM     all     34.50      0.00      6.38      7.66      0.00     51.45
04:08:35 AM       0     32.66      0.00      9.55      4.52      0.00     53.27
04:08:35 AM       1     76.24      0.00      2.48      5.45      0.00     15.84
04:08:35 AM       2     24.62      0.00     10.05      6.53      0.00     58.79
04:08:35 AM       3     38.50      0.00     14.50      5.50      0.00     41.50
04:08:35 AM       4      3.05      0.00      4.06      0.51      0.00     92.39
04:08:35 AM       5      1.99      0.00      1.49      9.45      0.00     87.06
04:08:35 AM       6     99.00      0.00      1.00      0.00      0.00      0.00
04:08:35 AM       7      1.50      0.00      1.00      0.00      0.00     97.50
04:08:35 AM       8     62.00      0.00     13.00     13.50      0.00     11.50
04:08:35 AM       9     91.96      0.00      6.53      0.00      0.00      1.51
04:08:35 AM      10     34.67      0.00     10.55     18.09      0.00     36.68
04:08:35 AM      11     57.00      0.00      4.00      4.00      0.00     35.00
04:08:35 AM      12     11.50      0.00      5.50     20.50      0.00     62.50
04:08:35 AM      13      6.47      0.00      2.49     15.42      0.00     75.62
04:08:35 AM      14      3.00      0.00      2.00      3.50      0.00     91.50
04:08:35 AM      15      7.54      0.00     15.08     15.58      0.00     61.81
----- output clipped -----

See the above example where I mentioned ALL with -P option and got each processor’s utilization report. In the below example, only CPU number 2 data is extracted.

# sar -P 2 -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsvr2)         12/20/2016      _x86_64_        (16 CPU)

04:11:11 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:11:13 AM       2     49.75      0.00      3.98     26.87      0.00     19.40
04:11:15 AM       2     97.50      0.00      1.50      1.00      0.00      0.00
04:11:17 AM       2     97.50      0.00      1.50      0.50      0.00      0.50
Average:          2     81.53      0.00      2.33      9.48      0.00      6.66

Another small stats regarding processor is power stats. Here sar shows you processor clock frequency at given instance of time. This helps in calculating power being used by CPU. -m option gives this data and it can be used per -processor reporting option too.

# sar -m 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/20/2016      _x86_64_        (16 CPU)

04:15:39 AM     CPU       MHz
04:15:41 AM     all   1970.50
04:15:43 AM     all   1845.81
04:15:45 AM     all   1587.93
Average:        all   1801.41

# sar -P ALL 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/20/2016      _x86_64_        (16 CPU)

04:15:52 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:15:54 AM     all     15.67      0.00      7.59      7.02      0.00     69.72
04:15:54 AM       0     20.00      0.00      7.00     37.50      0.00     35.50
04:15:54 AM       1     27.86      0.00     19.40      6.97      0.00     45.77
04:15:54 AM       2     47.50      0.00     15.50      2.50      0.00     34.50
04:15:54 AM       3      2.49      0.00      1.99      1.99      0.00     93.53
04:15:54 AM       4      3.02      0.00      5.03      1.01      0.00     90.95
04:15:54 AM       5      1.00      0.00      7.00      1.00      0.00     91.00
04:15:54 AM       6      0.51      0.00      0.51      0.00      0.00     98.97
04:15:54 AM       7      1.00      0.00      0.50      0.00      0.00     98.50
04:15:54 AM       8     35.18      0.00     21.61     20.10      0.00     23.12
04:15:54 AM       9     51.24      0.00     22.89      4.98      0.00     20.90
04:15:54 AM      10     28.64      0.00      8.04     14.57      0.00     48.74
04:15:54 AM      11     12.94      0.00      5.97     11.94      0.00     69.15
04:15:54 AM      12      8.50      0.00      3.50      7.50      0.00     80.50
04:15:54 AM      13      7.04      0.00      2.51      1.51      0.00     88.94
04:15:54 AM      14      1.01      0.00      0.51      1.52      0.00     96.97
04:15:54 AM      15      1.49      0.00      0.99      0.00      0.00     97.52
----- output clipped -----

Memory utilization reporting using sar

Memory stats can be extracted with -r option. When sar runs with -r option, it presents below parameters (highlighted values are more commonly observed for performance monitoring) :

  • kbmemfree: Free memory available in kilobytes.
  • kbmemused: Memory used (excluding kernel usage)
  • %memused: Percentage of memory used
  • kbbuffers:  memory used as buffers by the kernel in kilobytes
  • kbcached: memory used to cache data by the kernel in kilobytes
  • kbcommit: memory in kilobytes needed for the current workload. (commitment!)
  • %commit: % of memory needed for the current workload in relation to the total memory (RAM+swap)
# sar -r 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/20/2016      _x86_64_        (16 CPU)

03:42:07 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
03:42:09 AM 119133212 145423112     54.97   1263568 109109560 106882060     32.23
03:42:11 AM 119073748 145482576     54.99   1263572 109135424 107032108     32.27
03:42:13 AM 119015480 145540844     55.01   1263572 109162404 106976556     32.25
Average:    119074147 145482177     54.99   1263571 109135796 106963575     32.25

The output above shows the parameter values. It sections the same as explained above (CPU report example); first-line details, last avg row, etc. Note that, %commit can be 100%+ too since kernel always over-commit to avoid out of memory situation.

Paging statistics can be obtained using -B option. Normally, parameters shown in this option’s output are not observed by sysadmin. But if in-depth troubleshooting or monitoring is required then only this option is used. It shows the below parameters:

  • pgpgin/s:  Number of kilobytes the system paged in from disk per second.
  • pgpgout/s: Number of kilobytes the system paged out to disk per second.
  • fault/s: Number of page faults per second.
  • majflt/s: Number of major page faults per second.
  • pgfree/s: Number of pages placed on the free list by the system per second.
  • pgscank/s: Number of pages scanned by the kswapd daemon per second.
  • pgscand/s: Number of pages scanned directly per second.
  • pgsteal/s: Number of pages the system has reclaimed from cache per second.
  • %vmeff: This is a metric of the efficiency of page reclaim.
# sar -B 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/21/2016      _x86_64_        (4 CPU)

12:59:41 AM  pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
12:59:43 AM      3.05     40.10 120411.68      0.00  99052.28      0.00      0.00      0.00      0.00
12:59:45 AM      2.99      7.46  42649.75      0.00  37486.57      0.00      0.00      0.00      0.00
12:59:47 AM      3.00     47.50     43.00      0.00     80.50      0.00      0.00      0.00      0.00
Average:         3.01     31.61  54017.22      0.00  45257.86      0.00      0.00      0.00      0.00

Swap statistics can be obtained with -S option. Swap is another aspect of memory hence its utilization monitoring is as important as memory. Swap utilization reports are shown with -S option. It shows below parameters (highlighted values are more commonly observed for performance monitoring) :

  • kbswpfree: Free swap in kilobytes
  • kbswpused: Used swap in kilobytes
  • %swpused: % of swap used
  • kbswpcad: Amount of cached swap memory in kilobytes
  • %swpcad: % of cached swap memory in relation to used swap.
# sar -S 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/21/2016      _x86_64_        (4 CPU)

01:01:45 AM kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
01:01:47 AM   8388604         0      0.00         0      0.00
01:01:49 AM   8388604         0      0.00         0      0.00
01:01:51 AM   8388604         0      0.00         0      0.00
Average:      8388604         0      0.00         0      0.00

In the above example, you can see a total of 8GB swap available on server and nothing of it used. The swap will get hits only if memory gets completely utilized.

This concludes the second part of the sar tutorial. We will be seeing network-related reporting in the next part.