Yearly Archives: 2016

How to install patch/software in HPUX

A how-to guide for the installation of patch or software on the HPUX operating environment. This includes steps to install, configure, and verify.

This post is a how-to guide that has steps to install, configure, and verify patch or software on HPUX operating system. Before we begin the patching process, one needs to finalize the downtime window for patching with the concerned team.  Also, if individual patches are planned to install then patch ID needs to be known to download from the HP portal. HP Passport ID required to sign-in on the HP portal.

Also read: HPUX patch naming conventions

HPUX patching enables a system with new features, enhancements and removes bugs if any. HP releases patch bundles for HPUX every 6 months i.e twice in a year. These bundles can be downloaded and applied to the system. Also, individual patches can be downloaded. For individual patches, one should know the patch ID first.

Prerequisite :

  • Log in to the HP portal and download required patches. HP provides the facility to download your required patch along with its all dependencies so you need not to again search for dependencies separately.  When prompted for download, select the gzip bundle format.
  • Upload the patch on the server using FTP. Unzip patch file which will have create_depot script inside. When executed,  this script will create $PWD/depot directory which includes all patches to be installed. Make sure depot is created properly using:
# /usr/sbin/swlist  -s /tmp/PACPTHP_00001.depot

# Initializing...
# Contacting target "testsrv2"...
#
# Target:  testsrv2:/tmp/PACPTHP_00001.depot
#

#
# No Bundle(s) on testsrv2:/tmp/PACPTHP_00001.depot
# Product(s):
#

  HPOvLcore     6.00.000       HP Software Core Functionality
  HPOvLcore     6.00.000       HP Software Core Functionality
  HPOvPerf      4.70.000       HP OpenView Performance
  HPOvPerf      4.70.000       HP OpenView Performance
  • A server backup is an important thing to be done before any activity. Take ignite backup over network or tape. Also collect nickel, sysinfo outputs from the server. Ensure the downtime window and other related ITIL processes are completed and approved.

Installation & configuration :

Before starting activity make sure all applications on the server are brought down by respective parties. Once confirmed, proceed with patch installation. Install the patch using the command:

# /usr/sbin/swinstall -s /tmp/PACPTHP_00001.depot

NOTE:    The interactive UI was invoked, since no software was
         specified.

Starting the terminal version of swinstall...

To move around in swinstall:

- use the "Tab" key to move between screen elements
- use the arrow keys to move within screen elements
- use "Ctrl-F" for context-sensitive help anywhere in swinstall

On screens with a menubar at the top like this:

        ------------------------------------------------------
       |File View Options Actions                         Help|
       | ---- ---- ------- ------------------------------- ---|

- use "Tab" to move from the list to the menubar
- use the arrow keys to move around
- use "Return" to pull down a menu or select a menu item
- use "Tab" to move from the menubar to the list without selecting a menu item
- use the spacebar to select an item in the list

On any screen, press "CTRL-K" for more information on how to use the keyboard.

This will start the text-based GUI. You can navigate in this GUI menu using, arrow keys and SPACE key. Select the all listed patches using arrow key and space bar to highlight them.

Once marked, select Install option from the Actions menu.

This will take time to install and configure the patch in the system depending upon the size of the depot. This configuration happens in the background with this command execution only.

If the required system will be rebooted post-installation. To check if the patch needs a reboot after install, you can run above swinstall command with -p preview option. Which will just check dependencies, depot health, and show you report. This report includes if a reboot is required or not. Even while downloading the patch from the HP portal, you can view its details which also includes information about reboot requirements.

Verification :

Once the installation is complete, or after reboot, check if all installed patches are configured in the kernel or not using the command :

# swlist -l fileset -a state | grep –v –e “#” –e “configured”

The output of the above command should be blank. If there are any file sets left in an unconfigured/installed state then the above command’s output will reveal it. So, if the output is not blank, you need to configure these unconfigured/installed state file sets manually. To configure all patches again use below command:

# /usr/sbin/swconfig \ *

=======  12/27/16 10:17:15 SST  BEGIN swconfig SESSION
         (non-interactive) (jobid=testsrv2-0396)

       * Session started for user "root@testsrv2".

       * Beginning Selection
       * Target connection succeeded for "testsrv2:/".

----- output clipped -----
 * Selection succeeded.


       * Beginning Analysis
       * Session selections have been saved in the file
         "/home/user3/.sw/sessions/swconfig.last".
       * "testsrv2:/":  2831 filesets have already been configured.
       * Analysis succeeded.


       * Beginning Execution
       * "testsrv2:/":  2831 software objects were determined to be
         skipped in the analysis phase.
       * Execution succeeded.


NOTE:    More information may be found in the agent logfile using the
         command "swjob -a log testsrv2-0396 @ testsrv2:/".

=======  12/27/16 10:17:59 SST  END swconfig SESSION (non-interactive)
         (jobid=testsrv2-0396)

This command will try to configure the remaining patches. After completion of this command, once again verify your installation.

5 different examples to send email through Linux terminal

Five different examples to show you how to send an email using the Linux terminal. Learn how to send an email with the subject, attachment, and body from the terminal.

We have seen how to configure SMTP in Linux, in this post we will see different examples to send a mail through terminal. This post will elaborate on how to attach a file to mail, how to add a subject line, how to add body content to mail-in command-line interface.

Also read :

First of all, you need to check if SMTP is configured correctly and you are able to send mail from your Linux server. You can test it by sending test mail like below :

# echo test |sendmail -v info@xyz.com
[<-] 220 mailserver.xyz.com ESMTP Postfix
[->] HELO testsrv2
[<-] 250 mailserver.xyz.com
[->] MAIL FROM:<root@xyz.com>
[<-] 250 2.1.0 Ok
[->] RCPT TO:<info@xyz.com>
[<-] 250 2.1.5 Ok
[->] DATA
[<-] 354 End data with <CR><LF>.<CR><LF>
[->] Received: by testsrv2 (sSMTP sendmail emulation); Fri, 23 Dec 2016 02:29:07 +0800
[->] From: "root" <root@xyz.com>
[->] Date: Fri, 23 Dec 2016 02:29:07 +0800
[->] test
[->]
[->] .
[<-] 250 2.0.0 Ok: queued as 19F75822B8
[->] QUIT
[<-] 221 2.0.0 Bye

Once you receive test mail, it’s confirmed that your SMTP configurations are correct. Let’s see different examples to send mail.

1. An email with the subject line :

Sendmail is low-level utility hence it doesn’t support -s option for a subject line like other email utilities do. We have to stdin subject to sendmail like below :

# echo "Subject : test" |sendmail -v user4@xyz.com
user4@xyz.com... Connecting to mailserver.xyz.com via relay...
220 mailserver.xyz.com ESMTP Postfix
>>> EHLO server2.xyz.com
250-mailserver.xyz.com
250-PIPELINING
250-SIZE 25600000
---- output clipped -----

Text following “Subject:” will be treated as the subject line for mail.

You can also use mailx command with -s option to specify the subject line.

# mailx -s "test email subject line"  user4@xyz.com
Your email body goes here... type and hit ctrl+d
Cc:

2. An email with mail body :

If you have a long email body then you can save it in a file. Then you can pass that file to sendmail command. Sendmail will parse the file and the content of files will be seen in the body of the email.

# cat /tmp/mail_body.txt | sendmail -v user4@xyz.com
user4@xyz.com... Connecting to mailserver.xyz.com via relay...
220 mailserver.xyz.com ESMTP Postfix
>>> EHLO server2.xyz.com
250-mailserver.xyz.com
250-PIPELINING
250-SIZE 25600000
250-VRFY

The same format works with mailx command too. You can even source file using redirector for mail body like below:

# mailx user4.xyz.com < /tmp/mail_body.txt

3. Email with attachment

Sending attachment with sendmail/mail is a bit tricky. We need to encode files before sending as an attachment using uuencode.

# uuencode /tmp/attachement.txt | mail -s "Subject line" user4.xyz.com

4. An email with different FROM field

You can even change from the field of the email so that you can send an email with nicer names in the FROM field rather than ugly account names which mail utility picks up from your account details. There are few options when one is not available on your system to try another.

# mailx -r from_sender_id@xyz.com -s "subject" user4@xyz.com

# mailx -s "subject" user4.xyz.com -- -f from_sender_id@xyz.com 

# mailx -s "subject" -a "From: Sender <from_sender_id@xyz.com>" user4.xyz.com

5. Adding CC and BCC fields:

You can add BC and BCC fields too. Using options -c cc and -b for bcc:

# mail -s "test subject" -c abc@xyz.com -b pqr@xyz.com user4@xyz.com

How to add a subject line in sendmail command

You can add an email subject line in sendmail by sending “Subject: xxxxx” to sendmail command. See below examples –

(echo "Subject: Test email"; cat mailbody.txt) | sendmail -v info@kerneltalks.com

Errors and logfile for email in Linux

Once you send an email check /var/mail/maillog file for errors or success messages. If your configuration is correct and everything is working as expected you can see below message in there.

Sep 17 23:44:38 testsrv postfix/qmgr[10290]: 1086212AA0C: from=<root@testsrv>, size=470, nrcpt=1 (queue active)
Sep 17 23:44:38 testsrv postfix/smtp[17001]: 1086212AA0C: to=<info@kerneltalks.com>, relay=smtp.kerneltalks.com[10.10.1.1]:25, delay=4193, delays=4193/0.02/0.03/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 66AE24043)
Sep 17 23:44:38 testsrv postfix/qmgr[10290]: 1086212AA0C: removed

It means your message is successfully sent to the SMTP server. If you still did not receive the messages in your mailbox then you should check with the SMTP server owner that if they have valid email address restriction. Since if you don’t configure sender address then command by default sent an email with TO address as username@hostname

You can use -r option in mailx command to define TO email address as well which matches the policy of the SMTP server.

# echo test | mailx -s "Test from `hostname`" -r root@kerneltalks.com -S smtp="smtp.kerneltalks.com" info@kerneltalks.com

Like we have defined the sender address as root@kerneltalks.com in the above test command. Now, this email will be accepted by the SMTP server and will be delivered to the recipient.

Another error can be seen in maillog is as below –

Sep 17 23:07:33 testsrv postfix/smtp[9918]: 1086212AA0C: to=<info@kerneltalks.com>, relay=none, delay=1968, delays=1968/0.01/0.26/0, dsn=4.3.5, status=deferred (Host or domain name not found. Name service error for name=smtp.kerneltalks.com type=A: Host not found)

Solution: type A record, the host is not found means server is unable to resolve SMTP server name from DNS. Make sure you have DNS configured in /etc/resolv.conf and SMTP name is registered in DNS. You can verify DNS resolution and DNS server being used using nslookup smtp.kerneltalks.com command.

One more same error but for record type AAAA can be seen as below :

Sep 15 22:41:04 testsrv postfix/smtp[7833]: 97E9C12A7D9: to=<info@kerneltalks.com>, relay=none, delay=349010, delays=349010/0.01/0.26/0, dsn=4.3.5, status=deferred (Host or domain name not found. Name service error for name=smtp.kerneltalks.com type=AAAA: Host not found)

Solution: This means the server is looking for an IPv6 record for the SMTP server. Edit /etc/postfix/main.cf file and remove all protocols and define only ipv4 and you will be good.

# vi /etc/postfix/main.cf
inet_protocols = ipv4

4 steps guide for SMTP configuration in HPUX

Learn how to configure SMTP in HPUX machines. Know configuration file locations, parameters in config files, and testing commands.

SMTP is the short name for the Simple Mail Transfer Protocol. SMTP is an Internet standard for electronic mail (e-mail) transmission. We have already seen the SMTP configuration in Linux. In this post, we will be seeing SMTP in HPUX.

Pre-requisite

  • Hostname and IP address of SMTP server should be known (e.g. we will use mailserver.xyz.com 10.10.2.5)
  • The client should be able to reach the SMTP server over the network

Configurations

Step 1.

Add SMTP server entry in /etc/hosts file using vi editor or concatenate using echo command below.

# echo "10.10.2.5   mailserver.xyz.com" >> /etc/hosts

# cat /etc/hosts |grep -i mail
10.10.2.5   mailserver.xyz.com

Step 2.

Update daemon configuration file /etc/rc.config.d/mailservs to start the SMTP services with server startup. Edit the file with vi editors and update below entry:

SENDMAIL_SERVER=1

Add the SMTP server detail into SMTP configuration file /etc/mail/sendmail.cf. Edit below two entities

DMxyz.com
Dj<hostname>.com <<Add clients hostname
 DSmailserver.xyz.com
#C{E}root <<hash this entry

Step 3.

Now restart the SMTP daemon to take up this new configuration:

# /sbin/init.d/sendmail stop
Shutting down sendmail          [Done]
Shutting down sm-client         [Done]

# /sbin/init.d/sendmail start
Starting sendmail               [Done]
Starting sm-client              [Done]

Now check if sendmail is running using below command:

# ps -ef |grep -i sendmail
    root  1185     1  0  Nov 11  ?        61:33 sendmail: accepting connections on port 25
    root 21945 21815  0 03:18:28 pts/0     0:00 grep -i sendmail

Step 4.

Send a test mail and verify it.

# echo test | /usr/sbin/sendmail -v info@xyz.com
info@xyz.com... Connecting to mailserver.xyz.com via relay...
220 mailserver.xyz.com  ESMTP Postfix
>>> EHLO xyz.com
250-mailserver.xyz.com 
250-PIPELINING
250-SIZE 25600000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
>>> MAIL From:<root@xyz.com> SIZE=5
250 2.1.0 Ok
>>> RCPT To:<info@xyz.com>
250 2.1.5 Ok
>>> DATA
354 End data with <CR><LF>.<CR><LF>
>>> .
250 2.0.0 Ok: queued as 715C48203C
info@xyz.com... Sent (2.0.0 Ok: queued as 715C48203C)
Closing connection to mailserver.xyz.com
>>> QUIT
221 2.0.0 Bye

If you don’t receive an email, check logs under /var/adm/syslog/mail.log for troubleshooting.

3 tricks to get multiple commands output in the same row

Three tricks to get two commands output in a single row. Normally two commands outputs will be always separated by a carriage return in Linux terminal. 

One of the major hurdles in scripting is to get your outputs formatted well according to requirements. In Linux/Unix each command stdout its output always in a new line. This is a hurdle in many situations where the coder wants two outputs, two terms, two variables in a single row for example CSV format. In this post, we will see how to present two or more commands output in a single row.

Read also : Scripting related posts

Normally when we execute more than one command, they will display outputs in different rows:

# date ; hostname
Sat Dec 24 01:35:50 EDT 2016
testsrv2

# echo text1; echo text2; echo text3
text1
text2
text3

In the above example, commands have outputs per row. We are going to accomplish all outputs in one row using the below tricks.

Trick 1:

Using the translate function. AS I said earlier each command output is accompanied by a carriage return. We will be translating carriage return with tab character hence getting two or more commands output in a single row.

# (date; hostname) |tr '\n' '\t'
Sat Dec 24 01:36:22 EDT 2016    testsrv2

# (echo text1; echo text2; echo text3) |tr '\n' '\t'
text1   text2   text3

Bypassing outputs to translate function we achieved single row output. Here we instructed tr function to translate all carriage returns (\n) to tab (\t).

If you want to output in CSV format (comma-separated value) then replace tab with a comma and you will get it.

# (date; hostname) |tr '\n' ','
Sat Dec 24 01:43:09 EDT 2016,testsrv2,

# (echo text1; echo text2; echo text3) |tr '\n' ','
text1,text2,text3,

You can observe one trailing comma which shows every command output ends with \n and hence it gets replaced by ,

Trick 2:

By defining each command output as a variable. Here we will store the output of each command in one variable and then we will echo those variables in a single line.

# mydate=`date`

# myname=`hostname`

# echo $myname $mydate
testsrv2 Sat Dec 24 01:46:04 EDT 2016

Here we stored date and hostname command’s output in mydate and myname variables. In the last command, we echoed both variables with space in between. Note that commands output can be stored in a variable by executing the command using backticks (in-line execution)!

Trick 3:

By echoing in-line execution outputs. In-line execution i.e. using backticks to execute commands within another command. We will be using this trick to echo outputs in a single row.

# echo "`date` `hostname`"
Sat Dec 24 01:50:36 EDT 2016 testsrv2

In the above example, first, we have executed commands and got outputs. But those outputs are fed into echo command. Hence echo command stdout them in a single row.

# echo "`echo text1` `echo text2` `echo text3`"
text1 text2 text3

Make a note that we executed each command in separate back tick quotes.

5 steps guide for SMTP configuration in Linux

Learn how to configure SMTP in Linux machines. Know configuration file locations, parameters in config files, and testing commands.

Simple Mail Transfer Protocol. SMTP is one of the important services in any production system. Nowadays, mail notifications became important to monitor systems in real-time. Hence, it becomes essential to know how to configure mail utilities on servers. To enable any mail program on the system, SMTP is a basic component. It’s a very easy and quick procedure to setup SMTP on servers.

Pre-requisite :

  • Hostname and IP address of SMTP server should be known (e.g. we will use mailserver.xyz.com 10.10.2.5)
  • The client should be able to reach SMTP server over a network

Configurations :

Its 5 steps procedure to setup SMTP client on Linux.

Step 1.

Add SMTP server entry in /etc/hosts file using vi editor or concatenate using echo command below.

# echo "10.10.2.5   mailserver.xyz.com" >> /etc/hosts
# cat /etc/hosts |grep -i mail
10.10.2.5   mailserver.xyz.com
Step 2.

We need to edit the macro configuration file /etc/mail/sendmail.mc. Search and edit below parameters in this file :

  • define(`SMART_HOST',`mailserver.xyz.com'): Smart relay server name
  • define(`confDOMAIN_NAME',`xyz.com')dnl: Valid domain name
  • FEATURE(always_add_domain)dnlalways masquerades email address
  • FEATURE(`allmasquerade')dnl: rewrite both from/to relative to the local machine.
  • MASQUERADE_DOMAIN(`xyz.com.')dnl: Domain for masquerading emails
  • MASQUERADE_AS(`xyz.com')dnl: All mails masquerades as came from the defined domain
Step 3.

Once the above changes are done, you need to compile the mc file so that all these changes of the macro (mc) file will be transferred to the config ( cf) file using m4 utility.

# m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf
Step 4.

Edit below parameters in /etc/mail/submit.cf so that mails will be submitted to defined domain.

D{MTAHost} mailserver.xyz.com 
Djxyz.com: Define this in case of sendmail won’t able to determine your domain.
DSmailserver.xyz.com: Smart relay host

Step 5.

Finally, restart mail services to take up this new configurations :

# service sendmail restart
# service postfix restart

You are done! Send a test email with the below command and check if you receive it.

# echo test |sendmail -v info@xyz.com
[<-] 220 mailserver.xyz.com ESMTP Postfix
[->] HELO testsrv2
[<-] 250 mailserver.xyz.com
[->] MAIL FROM:<root@xyz.com>
[<-] 250 2.1.0 Ok
[->] RCPT TO:<info@xyz.com>
[<-] 250 2.1.5 Ok
[->] DATA
[<-] 354 End data with <CR><LF>.<CR><LF>
[->] Received: by testsrv2 (sSMTP sendmail emulation); Fri, 23 Dec 2016 02:29:07 +0800
[->] From: "root" <root@xyz.com>
[->] Date: Fri, 23 Dec 2016 02:29:07 +0800
[->] test
[->]
[->] .
[<-] 250 2.0.0 Ok: queued as 19F75822B8
[->] QUIT
[<-] 221 2.0.0 Bye

If you don’t receive an email, check logs under /var/log/maillog for troubleshooting.

How to save top command output in file

A how-to guide to save top command output in a simple text file. A useful tip to share real-time, dynamic command output in a static log file.

TOP is the first command that came to mind whenever someone asks about system resource utilization monitoring. CPU, memory utilization can be easily tracked in this tool. This tool is available to almost every UNIX and Linux flavor.

The top command outputs in a real-time updating ASCII window. Data within this ASCII GUI screen updates every 2 seconds in real-time. So all bits in it change every time and its difficult to share its output to anyone who doesn’t have access to the system. Also, if you want to collect the current output and save for future use then that’s also not possible due to updating data.

For such a situation where you want to save data being shown in the output window, top command gives you options of iterations and output files. You can define how many iterations you need and the time difference between them. It works like iterations and intervals in SAR command. Use the below options to save the output in a file.

  • -b: batch operation mode
  • -n: number of iterations
  • -d: Delay i.e. intervals between iterations (not needed for single iteration)
  • -f file: Output file in which command saves data (supports in few versions only)

We are using a batch operation mode of the top. By defining, iterations you ask top command to stop execution after that many counts. Normally it runs continuously with a delay of 2 sec. This makes it easy to capture the output in a file using -o option. If -o is not supported then you can simply redirect output to file using redirection operator.

# top -n 1 -b >/tmp/top_output.txt
# cat  /tmp/top_output.txt
top - 02:58:22 up 278 days,  7:19,  1 user,  load average: 0.56, 0.52, 0.56
Tasks: 189 total,   1 running, 188 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.8%us,  4.9%sy,  0.0%ni, 84.0%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  12199956k total,  7774572k used,  4425384k free,   856212k buffers
Swap:  8388604k total,        0k used,  8388604k free,  4787268k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2994 root      RT   0  688m  84m  55m S  7.9  0.7   2998:17 osysmond.bin
 4227 grid      20   0  501m  29m  17m S  2.0  0.3 458:12.36 oracle
 4229 grid      -2   0  500m  29m  17m S  2.0  0.3   1009:43 oracle
 4251 grid      20   0  485m  27m  25m S  2.0  0.2 112:07.76 oracle
 4266 root      20   0 2206m  73m  25m S  2.0  0.6   2519:45 crsd.bin
    1 root      20   0 19400 1552 1228 S  0.0  0.0   0:03.23 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.41 kthreadd
    3 root      20   0     0    0    0 S  0.0  0.0  56:25.42 ksoftirqd/0
----- output clipped -----

In the above example, I ran the top command with a single iteration only. Since its single iteration, I didn’t use a delay option. Also -o option was not supported so I redirected output to a file. You can see the content of the file is the top command output screen!

If you use more than 1 iteration lets say 2. Then there will 2 top output screens concatenated one after another in a log file. So it will be like you have a snapshot of all values for a given period of time for future reference. See example below:

# top -n 2 -d 4 -b >/tmp/top_out.txt
# cat /tmp/top_out.txt
top - 03:07:39 up 278 days,  7:28,  1 user,  load average: 0.43, 0.51, 0.54
Tasks: 191 total,   1 running, 190 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.8%us,  4.9%sy,  0.0%ni, 84.0%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  12199956k total,  7762352k used,  4437604k free,   856212k buffers
Swap:  8388604k total,        0k used,  8388604k free,  4774904k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4213 grid      -2   0  485m  14m  12m S  2.0  0.1   5601:55 oracle
    1 root      20   0 19400 1552 1228 S  0.0  0.0   0:03.23 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.41 kthreadd
    3 root      20   0     0    0    0 S  0.0  0.0  56:25.58 ksoftirqd/0
    6 root      RT   0     0    0    0 S  0.0  0.0 399995:22 migration/0
    7 root      RT   0     0    0    0 S  0.0  0.0   1:17.98 watchdog/0
    8 root      RT   0     0    0    0 S  0.0  0.0 399911:47 migration/1
   10 root      20   0     0    0    0 S  0.0  0.0  31:40.41 ksoftirqd/1
   11 root      20   0     0    0    0 S  0.0  0.0  19:42.29 kworker/0:1
  
----- output clipped -----

top - 03:07:44 up 278 days,  7:28,  1 user,  load average: 0.48, 0.51, 0.55
Tasks: 193 total,   4 running, 189 sleeping,   0 stopped,   0 zombie
Cpu(s): 24.5%us, 10.1%sy,  0.0%ni, 65.1%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  12199956k total,  7803692k used,  4396264k free,   856212k buffers
Swap:  8388604k total,        0k used,  8388604k free,  4774968k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2466 root      20   0  129m  25m 1288 S  6.6  0.2   8235:08 OSWatcher.sh
15104 root      20   0  129m  24m  344 R  1.8  0.2   0:00.09 OSWatcher.sh
 4213 grid      -2   0  485m  14m  12m S  1.4  0.1   5601:55 oracle
 2994 root      RT   0  688m  84m  55m S  1.0  0.7   2998:22 osysmond.bin
 3038 grid      RT   0 1608m 116m  53m S  0.8  1.0   2379:27 ocssd.bin
 4385 root      20   0 2543m  42m  13m S  0.8  0.4   3013:44 orarootagent.bi
----- output clipped -----

You can even put up a cron to save the output of top commands to some log file. But this is not recommended since it will pile up logfile in no time and hence fill mount point rapidly. Anyway, we have in-depth utilization data being collected by OS which can be viewed using SAR utility.

-f option which is available in HPUX native top utility has few limitations. It stores only the first 16 lines of output to file and it supports only one iteration. 16 lines limitations make logfile quite short and it shows mainly header values CPU, memory, etc. Most of the processes related to data get lost. See example below :

$ top -f /tmp/test
$ cat /tmp/test
System: testsrv3 Wed Dec 21 03:18:19 2016
Load averages: 0.03, 0.04, 0.04
246 processes: 168 sleeping, 78 running
Cpu states:
CPU   LOAD   USER   NICE    SYS   IDLE  BLOCK  SWAIT   INTR   SSYS
 0    0.04   0.0%   0.0%   0.2%  99.8%   0.0%   0.0%   0.0%   0.0%
14    0.03   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0%
15    0.03   0.0%   0.0%   0.2%  99.8%   0.0%   0.0%   0.0%   0.0%
16    0.04   0.0%   0.0%   0.4%  99.6%   0.0%   0.0%   0.0%   0.0%
17    0.02   0.2%   0.0%   0.4%  99.4%   0.0%   0.0%   0.0%   0.0%
18    0.04   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0%
---   ----  -----  -----  -----  -----  -----  -----  -----  -----
avg   0.03   0.0%   0.0%   0.2%  99.8%   0.0%   0.0%   0.0%   0.0%

System Page Size: 4Kbytes
Memory: 1679640K (1411160K) real, 3170312K (2643480K) virtual, 30532328K free  P
CPU TTY  PID USERNAME PRI NI   SIZE    RES STATE    TIME %WCPU  %CPU COMMAND

sar command (Part III) : Disk, Network reporting

Learn System Activity Report sar command with real-world scenario examples. Understand how to do disk, network reporting using this command.

In the last two parts of the sar command, we have seen time formats to be used with command, its data files, CPU & Memory reporting. In this last part, we will be seeing disk, network reporting using sar command.

Read last two parts of this tutorial here :

Disk IO reporting

sar provides disk (block devices)  report with -d option.  Normally, it shows below parameters values (highlighted values are more commonly observed for performance monitoring) :

  • DEV: Block device name. It follows the dev m-n format. M is major and n is a minor number of the block devices.
  • tps: Transfers per second
  • rd_sec/s: Sector reads per second (sector is 512 byte)
  • wr_sec/s: Sector writes per second 
  • avgrq-sz:  average size (in sectors) of the requests that were issued to the device
  • avgqu-sz: average queue length of the requests that were issued to the device
  • await: The average time (in milliseconds) for I/O requests
  • svctm: The average service time (in milliseconds) for I/O requests
  • %util: Percentage  of  CPU  time  during  which  I/O requests were issued to the device
# sar -d 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/21/2016      _x86_64_        (4 CPU)

01:20:19 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
01:20:21 AM    dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:21 AM   dev8-64      3.00      2.00      1.00      1.00      0.00      0.50      0.50      0.15
01:20:21 AM   dev8-32      3.00      2.00      1.00      1.00      0.00      0.50      0.33      0.10
01:20:21 AM  dev252-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:21 AM  dev252-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:21 AM  dev252-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

In the above output, device names are not so user friendly. So to identify devices easily, -p option is available. This option prints pretty device names in the DEV column and it should always be used with -d option.

# sar -d -p 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/21/2016      _x86_64_        (4 CPU)

01:20:38 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
01:20:40 AM       sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:40 AM       sdb      4.98      0.00     91.54     18.40      0.00      0.80      0.20      0.10
01:20:40 AM       sdc      2.99      1.99      1.00      1.00      0.00      0.67      0.67      0.20
01:20:40 AM      dm-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:40 AM      dm-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:20:40 AM      dm-2      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Now see above output where device names are easily identifiable. sda means disk /dev/sda and so on.

Network utilization reporting

Option -n gives all network stats. There are a total of 18 different keywords (like NFS, IP, DEV, TCP, etc.) which can be used with -n option to get related parameters. Each keyword has almost 8-10 parameters to display. So in the call, if you are using ALL keyword, then the output will be a huge list of parameters which is difficult to understand.

To keep it short here we will see only one example of keyword DEV i.e. device. This will show the device i.e. network card’s parameter values. Most of the time NIC performance is checked hence we are using this keyword example.

# sar -n DEV 2 1
Linux 2.6.39-200.24.1.el6uek.x86_64 (textsrv2)         12/21/2016      _x86_64_        (4 CPU)

01:35:22 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
01:35:24 AM        lo      6.00      6.00      0.29      0.29      0.00      0.00      0.00
01:35:24 AM      eth0     15.50      0.50      0.91      0.04      0.00      0.00      0.00
01:35:24 AM      eth1      6.50      4.50      0.97      0.77      0.00      0.00      0.00
01:35:24 AM      eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo      6.00      6.00      0.29      0.29      0.00      0.00      0.00
Average:         eth0     15.50      0.50      0.91      0.04      0.00      0.00      0.00
Average:         eth1      6.50      4.50      0.97      0.77      0.00      0.00      0.00
Average:         eth3      0.00      0.00      0.00      0.00      0.00      0.00      0.00

In the above example, I used the DEV keyword along with -n option and took only one iteration in output. Parameters displayed for device keyword are :

  • IFACE: Its interface name. You can easily see eth0, eth1, loopback (lo) interfaces here.
  • rxpck/s: Packets received per seconds
  • txpck/s: packets transmitted per second
  • rxkB/s: kilobytes received per second
  • txkB/s : kilobytes transmitted per second
  • rxcmp/s: compressed packets received per second
  • txcmp/s: compressed packets transmitted per second
  • rxmcst/s: Number of multicast packets received per second

This concludes sar command tutorial’s part III about the disk, network reporting. This is a three-part tutorial with example outputs included. Put your queries, suggestions, feedback in the comments below. You can also reach us using our Contact form. 

sar command (Part II) : CPU, Memory reporting

Learn System Activity Report sar command with real-world scenario examples. Understand CPU, Memory reporting using this command.

In the last post of sar command, we have seen its data file structure, how to extract data from it, and time formats to be used with this command.  In this post, let’s see how to get CPU, memory utilization reports from data files, or real-time using sar command.

Read other parts of sar tutorial :

sar command follows below format :

# sar [ options ] [ <interval> [ <count> ] ]

We have already seen what is interval and count in the last post. Now we will see different options that can be used to get different system resource utilization stats. Also, we have seen how to get historic data from sar data files, I will be using only real-time commands (i.e. without -f option) for all below examples.

Before we start with resource reporting here is a quick tip about start and end time of reports when you are extracting data from datafiles. Below two options can be used with sar command (in conjunction with -f) so that specific time window data can be extracted.

  • -s hh:mm:ss Start time of the report. Sar starts output records tagged to this time or very next available time-tagged record. Default start time is 08:00:00
  • -e hh:mm:ss The end time of the report. The default end time is 18:00:00

CPU utilization reporting using sar

For CPU statistics, sar command has -u option. Executing sar command with -u gives us below utilization matrices (highlighted values are more commonly observed for performance monitoring) :

  • %user: CPU % used by user processes
  • %nice: CPU % used by user processes with nice priority
  • %system:  CPU % used by system processes
  • %iowait: % of the time when CPU was idle (since processes were busy in IO)
  • %steal: % of time wait by virtual CPU while hypervisor servicing another CPU (virtualization aspect)
  • %idle: CPU % idle.
# sar -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/20/2016      _x86_64_        (16 CPU)

03:34:51 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:34:53 AM     all     33.00      0.00      9.99     32.22      0.00     24.80
03:34:55 AM     all     37.32      0.00     10.44     32.95      0.00     19.29
03:34:57 AM     all     36.04      0.00     11.90     29.83      0.00     22.24
Average:        all     35.46      0.00     10.78     31.66      0.00     22.11

See the above example to get the values of the parameters explained above. The output starts with a line that has OS kernel version details, hostname in brackets, date, architecture, and the total number of CPUs. Followed by a data with interval (2 sec) and count (3) specified in the command. Finally, it also gives us the average value for all parameters. Column CPU denoting value all indicates these are averaged out values of all 16 CPU for the given time instance.

If you are interested in seeing values for each processor then -P option (per processor reporting) can be used with CPU number of your choice or ‘ALL’. When ALL is specified each processor’s data is shown or only specified CPU’s data is processed.

# sar -P ALL -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsvr2)         12/20/2016      _x86_64_        (16 CPU)

04:08:33 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:08:35 AM     all     34.50      0.00      6.38      7.66      0.00     51.45
04:08:35 AM       0     32.66      0.00      9.55      4.52      0.00     53.27
04:08:35 AM       1     76.24      0.00      2.48      5.45      0.00     15.84
04:08:35 AM       2     24.62      0.00     10.05      6.53      0.00     58.79
04:08:35 AM       3     38.50      0.00     14.50      5.50      0.00     41.50
04:08:35 AM       4      3.05      0.00      4.06      0.51      0.00     92.39
04:08:35 AM       5      1.99      0.00      1.49      9.45      0.00     87.06
04:08:35 AM       6     99.00      0.00      1.00      0.00      0.00      0.00
04:08:35 AM       7      1.50      0.00      1.00      0.00      0.00     97.50
04:08:35 AM       8     62.00      0.00     13.00     13.50      0.00     11.50
04:08:35 AM       9     91.96      0.00      6.53      0.00      0.00      1.51
04:08:35 AM      10     34.67      0.00     10.55     18.09      0.00     36.68
04:08:35 AM      11     57.00      0.00      4.00      4.00      0.00     35.00
04:08:35 AM      12     11.50      0.00      5.50     20.50      0.00     62.50
04:08:35 AM      13      6.47      0.00      2.49     15.42      0.00     75.62
04:08:35 AM      14      3.00      0.00      2.00      3.50      0.00     91.50
04:08:35 AM      15      7.54      0.00     15.08     15.58      0.00     61.81
----- output clipped -----

See the above example where I mentioned ALL with -P option and got each processor’s utilization report. In the below example, only CPU number 2 data is extracted.

# sar -P 2 -u 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsvr2)         12/20/2016      _x86_64_        (16 CPU)

04:11:11 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:11:13 AM       2     49.75      0.00      3.98     26.87      0.00     19.40
04:11:15 AM       2     97.50      0.00      1.50      1.00      0.00      0.00
04:11:17 AM       2     97.50      0.00      1.50      0.50      0.00      0.50
Average:          2     81.53      0.00      2.33      9.48      0.00      6.66

Another small stats regarding processor is power stats. Here sar shows you processor clock frequency at given instance of time. This helps in calculating power being used by CPU. -m option gives this data and it can be used per -processor reporting option too.

# sar -m 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/20/2016      _x86_64_        (16 CPU)

04:15:39 AM     CPU       MHz
04:15:41 AM     all   1970.50
04:15:43 AM     all   1845.81
04:15:45 AM     all   1587.93
Average:        all   1801.41

# sar -P ALL 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/20/2016      _x86_64_        (16 CPU)

04:15:52 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:15:54 AM     all     15.67      0.00      7.59      7.02      0.00     69.72
04:15:54 AM       0     20.00      0.00      7.00     37.50      0.00     35.50
04:15:54 AM       1     27.86      0.00     19.40      6.97      0.00     45.77
04:15:54 AM       2     47.50      0.00     15.50      2.50      0.00     34.50
04:15:54 AM       3      2.49      0.00      1.99      1.99      0.00     93.53
04:15:54 AM       4      3.02      0.00      5.03      1.01      0.00     90.95
04:15:54 AM       5      1.00      0.00      7.00      1.00      0.00     91.00
04:15:54 AM       6      0.51      0.00      0.51      0.00      0.00     98.97
04:15:54 AM       7      1.00      0.00      0.50      0.00      0.00     98.50
04:15:54 AM       8     35.18      0.00     21.61     20.10      0.00     23.12
04:15:54 AM       9     51.24      0.00     22.89      4.98      0.00     20.90
04:15:54 AM      10     28.64      0.00      8.04     14.57      0.00     48.74
04:15:54 AM      11     12.94      0.00      5.97     11.94      0.00     69.15
04:15:54 AM      12      8.50      0.00      3.50      7.50      0.00     80.50
04:15:54 AM      13      7.04      0.00      2.51      1.51      0.00     88.94
04:15:54 AM      14      1.01      0.00      0.51      1.52      0.00     96.97
04:15:54 AM      15      1.49      0.00      0.99      0.00      0.00     97.52
----- output clipped -----

Memory utilization reporting using sar

Memory stats can be extracted with -r option. When sar runs with -r option, it presents below parameters (highlighted values are more commonly observed for performance monitoring) :

  • kbmemfree: Free memory available in kilobytes.
  • kbmemused: Memory used (excluding kernel usage)
  • %memused: Percentage of memory used
  • kbbuffers:  memory used as buffers by the kernel in kilobytes
  • kbcached: memory used to cache data by the kernel in kilobytes
  • kbcommit: memory in kilobytes needed for the current workload. (commitment!)
  • %commit: % of memory needed for the current workload in relation to the total memory (RAM+swap)
# sar -r 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/20/2016      _x86_64_        (16 CPU)

03:42:07 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
03:42:09 AM 119133212 145423112     54.97   1263568 109109560 106882060     32.23
03:42:11 AM 119073748 145482576     54.99   1263572 109135424 107032108     32.27
03:42:13 AM 119015480 145540844     55.01   1263572 109162404 106976556     32.25
Average:    119074147 145482177     54.99   1263571 109135796 106963575     32.25

The output above shows the parameter values. It sections the same as explained above (CPU report example); first-line details, last avg row, etc. Note that, %commit can be 100%+ too since kernel always over-commit to avoid out of memory situation.

Paging statistics can be obtained using -B option. Normally, parameters shown in this option’s output are not observed by sysadmin. But if in-depth troubleshooting or monitoring is required then only this option is used. It shows the below parameters:

  • pgpgin/s:  Number of kilobytes the system paged in from disk per second.
  • pgpgout/s: Number of kilobytes the system paged out to disk per second.
  • fault/s: Number of page faults per second.
  • majflt/s: Number of major page faults per second.
  • pgfree/s: Number of pages placed on the free list by the system per second.
  • pgscank/s: Number of pages scanned by the kswapd daemon per second.
  • pgscand/s: Number of pages scanned directly per second.
  • pgsteal/s: Number of pages the system has reclaimed from cache per second.
  • %vmeff: This is a metric of the efficiency of page reclaim.
# sar -B 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/21/2016      _x86_64_        (4 CPU)

12:59:41 AM  pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
12:59:43 AM      3.05     40.10 120411.68      0.00  99052.28      0.00      0.00      0.00      0.00
12:59:45 AM      2.99      7.46  42649.75      0.00  37486.57      0.00      0.00      0.00      0.00
12:59:47 AM      3.00     47.50     43.00      0.00     80.50      0.00      0.00      0.00      0.00
Average:         3.01     31.61  54017.22      0.00  45257.86      0.00      0.00      0.00      0.00

Swap statistics can be obtained with -S option. Swap is another aspect of memory hence its utilization monitoring is as important as memory. Swap utilization reports are shown with -S option. It shows below parameters (highlighted values are more commonly observed for performance monitoring) :

  • kbswpfree: Free swap in kilobytes
  • kbswpused: Used swap in kilobytes
  • %swpused: % of swap used
  • kbswpcad: Amount of cached swap memory in kilobytes
  • %swpcad: % of cached swap memory in relation to used swap.
# sar -S 2 3
Linux 2.6.39-200.24.1.el6uek.x86_64 (testsrv2)         12/21/2016      _x86_64_        (4 CPU)

01:01:45 AM kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
01:01:47 AM   8388604         0      0.00         0      0.00
01:01:49 AM   8388604         0      0.00         0      0.00
01:01:51 AM   8388604         0      0.00         0      0.00
Average:      8388604         0      0.00         0      0.00

In the above example, you can see a total of 8GB swap available on server and nothing of it used. The swap will get hits only if memory gets completely utilized.

This concludes the second part of the sar tutorial. We will be seeing network-related reporting in the next part.