Monthly Archives: March 2016

Processes using high memory, cpu using unix95

Learn how to get a list of processes consuming high memory or CPU on the kernel in sorted order using unix95. Helpful for troubleshooting high system utilization.


Listing of processes who are using high memory :

# UNIX95=1 ps -ef -o vsz= -o pid= -o comm= | sort -rnk1 | awk '{ print $1/1024" MB "$2" "$3; }'
749.156 MB 2025 cimprovagt
60.1875 MB 2842 midaemon
56.8672 MB 3616 opcmona
49.3633 MB 6905 cmcld
43.8906 MB 3485 coda
42.9375 MB 2035 cimprovagt
41.043 MB 2372 java
22.875 MB 2039 cimprovagt
21.2578 MB 3414 nsrexecd
18.2617 MB 2617 vxpal
17.2188 MB 2431 vxsvc
16.9297 MB 3264 postgres:

----- output truncated -----

or

# UNIX95= ps -eo vsz,comm,args | sed 1d | sort -rn | more
 767136 cimprovagt      /opt/wbem/lbin/cimprovagt 0 4 9 root SFMProviderModule
  61632 midaemon        /opt/perf/bin/midaemon
  58232 opcmona         /opt/OV/lbin/eaagt/opcmona
  50548 cmcld           /usr/lbin/cmcld -j
  44944 coda            /opt/OV/lbin/perf/coda
  43968 cimprovagt      /opt/wbem/lbin/cimprovagt 0 4 9 root HPUXFCIndicationProviderModule
  42028 java            /opt/hpservices/jre/opt/java1.4/jre/bin/IA64N/java -Xmx512m -DRunnerValuePairFile=/var/opt/runner/data/valuePairs -cp /opt/runn
  23424 cimprovagt      /opt/wbem/lbin/cimprovagt 0 4 9 root HPUXStorageIndicationProviderModule
  21768 nsrexecd        /opt/networker/bin/nsrexecd
  18700 vxpal           /opt/VRTSobc/pal33/bin/vxpal -a StorageAgent -x
  17632 vxsvc           /opt/VRTSob/bin/vxsvc -r /etc/vx/isis/Registry -e

----- output truncated -----





Listing of processes who are using high CPU:

# UNIX95= ps -e -o "vsz pcpu ruser pid stime time state args" | sort -rn |head -10
 767136  0.00 root      2025  Nov 15  1-04:47:22 R /opt/wbem/lbin/cimprovagt 0 4 9 root SFMProviderModule
  61632  0.07 root      2842  Nov 15  11:29:30 R /opt/perf/bin/midaemon
  58232  0.00 root      3616  Nov 15  05:52:05 R /opt/OV/lbin/eaagt/opcmona
  50548  0.14 root      6905  Nov 15  1-05:23:35 R /usr/lbin/cmcld -j
  44944  0.00 root      3485  Nov 15  04:46:17 R /opt/OV/lbin/perf/coda
  43968  0.00 root      2035  Nov 15  05:53:58 R /opt/wbem/lbin/cimprovagt 0 4 9 root HPUXFCIndicationProviderModule
  42028  0.09 root      2372  Nov 15  22:59:12 R /opt/hpservices/jre/opt/java1.4/jre/bin/IA64N/java -Xmx512m -DRunnerValuePairFile=/var/opt/runner/data/valuePairs -cp /opt/runn
  23424  0.00 root      2039  Nov 15  02:26:51 R /opt/wbem/lbin/cimprovagt 0 4 9 root HPUXStorageIndicationProviderModule
  21768  0.00 root      3414  Nov 15  01:22:45 R /opt/networker/bin/nsrexecd
  18700  0.00 root      2617  Nov 15  01:04:36 R /opt/VRTSobc/pal33/bin/vxpal -a StorageAgent -x

How to restore nagios configuration from backup

Learn to restore Nagios configuration backup. If you messed up Nagios configuration, you can restore last known good config using these steps.

Restore Nagios configuration

Requirement

While troubleshooting or doing some configuration changes (R n D), sometimes you mess up the configuration in Nagios. This leads to no data dashboard. You need to revert back to the last known good configuration so that the tool can resume its work.

Also read: How to install & configure checkmk in Linux

Solution

Normally you should have daily (or frequency of your choice i.e. weekly, hourly, etc.) backups scheduled for tool configuration using services like cron. We will see how to configure this configuration backup in another article.

Now, navigate to the directory where configuration backups are kept. Normally they should be in gunzip format.

Once inside that directory run below restore command.

# check_mk --restore check_mk.11-Mar-2016.tar.gz

After restore to make sure, ownership is well in place, run below command

# chown -R apache:nagcmd /etc/check_mk/conf.d/wato/

Lastly, restart Nagios to take up restored configuration.

# check_mk -R --restart nagios-check_mk

Now go back to the dashboard and check its populated with values!

Highest size files in mount point

Search for the highest sized file in the mount point. Useful while dealing with full mount points and cleaning up files to free up space.


There are many instances in the life of sysadmin when his system starts throwing filesystem full events and its headache to search culprit. Here we are going to see a command to search and display files with high utilization in a mount point. This is very useful to dig down and target exactly huge/big files rather than wasting time on trimming/deleting small files.

For example, if you are facing /tmp is full. Below is the command to search for large files in the given mount point. Here we are using du i.e. disk usage command then sorting the output list in reverse order so that the highest size files come to the top. Then getting top 10 to display as a result.

# du -a /tmp | sort -nr  | head -n 10
3396448 /tmp
1801984 /tmp/Acasrro411704
994848  /tmp/software.depot
81088   /tmp/PACPTHP_00001.depot
77088   /tmp/OraInstall2008-03-19_09-39-20AM
76912   /tmp/OraInstall2008-03-19_10-15-28AM
76848   /tmp/OraInstall2008-03-19_09-39-20AM/jre/1.4.2
76848   /tmp/OraInstall2008-03-19_09-39-20AM/jre
76656   /tmp/OraInstall2008-03-19_10-15-28AM/jre/1.4.2
76656   /tmp/OraInstall2008-03-19_10-15-28AM/jre

Replace /tmp with a mount point of your choice while running it in your system.

There are also different ways you can trace the highest size hogger on a mount point. Below are few

To find the large file size above 5000

# find /var -xdev -size +5000
/var/opt/perf/datafiles/logglob
/var/opt/perf/datafiles/logproc
/var/opt/perf/datafiles/logdev
/var/opt/perf/datafiles/logtran
/var/opt/OV/log/SPISvcDisc/DBSPI_ORACLE_UNIX_discovery.trc
/var/opt/OV/log/osspi/osspi.log
----- output truncated -----





Top 10 large directories in the mount point 

# du -kx /var | sort -rn -k1 | head -n 10
6726048	/var
3962830	/var/opt
3471340	/var/opt/OV
2494362	/var/adm
2465390	/var/opt/OV/tmp
1512210	/var/opt/OV/tmp/OpC
1438642	/var/adm/openv/logs
1438642	/var/adm/openv
921030	/var/adm/sa
822672	/var/adm/openv/logs/bpdbm

Top 10 large files in the mount point

# find /var -type f -xdev -print | xargs -e ll | sort -rn -k5 | head -n 10
-rw-rw----   1 root       bin        1463238440 Nov  1 14:45 /var/opt/OV/tmp/OpC/msgagtdf
-rw-rw-r--   1 root       bin        944541567 Nov  1 14:52 /var/opt/OV/tmp/sqlnet.log
-rw-rw-rw-   1 oracle92   dba        307369984 Mar  5  2014 /var/opt/omni/tmp/rcvcat21945.exp
-rw-------   1 root       mail       187848937 Nov  1 14:01 /var/tmp/dead.letter
-rw-rw----   1 root       bin        84680704 Feb 16  2013 /var/opt/OV/tmp/OpC/msgagtdf-11975.gc
-rw-r--r--   1 root       bin        80937879 Nov  1 14:54 /var/opt/OV/log/osspi/osspi.log
-rwxrwxrwx   1 root       users      58566204 Nov  1 14:55 /var/adm/rrdtool/apcrss32.rrd
-rw-rw-r--   1 root       bin        55159884 May 25  2009 /var/opt/OV/installation/temp/bbc/fx/upload/DeplM22799
-rw-rw-r--   1 root       bin        55159884 May 18  2009 /var/opt/OV/installation/temp/bbc/fx/upload/DeplM22011
-rw-rw-r--   1 root       bin        55159884 Jun 30  2010 /var/opt/OV/installation/temp/bbc/fx/upload/DeplM10143