Tag Archives: troubleshooting basics in linux

11 log files you should see on your Linux system

2 Replies

Listing of important Linux log files and their formats. These logs play a vital role in troubleshooting and every sysadmin should be aware of them.

Important log files in Linux

Troubleshooting any issues with your system needs proper knowledge of log file structure and their locations. As a sysadmin you should know for which log file should be checked for a particular service issue. In this article we will walk through several hand-picked log files which are pretty much helpful for conducting preliminary analysis during troubleshooting. 10 log files we selected to discuss here are :

  1. /var/log/messages : General message and system related stuff
  2. /var/log/boot.log : Services success/failures at boot
  3. /var/log/secure or /var/log/auth.log : Authentication log
  4. /var/log/utmp or /var/log/wtmp : Login records
  5. /var/log/btmp  : Failed login records
  6. /var/log/cron : Scheduler log file
  7. /var/log/maillog : Mail logs
  8. /var/log/xferlog : File transfer logs
  9. /var/log/lastlog : Last login details
  10. dmesg : Device driver messages
  11. /var/crash logs : System crash dump

Lets see each log file one by one.

System consolidated log file : /var/log/messages

All system services which do not have their own special log file, normally write to this log file. Most of the system activity can be seen here hence its also called Syslog (system’s log). Every sysadmin first opens up this log when he starts troubleshooting! Sample log looks like below :

May 22 02:00:29 server1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="999" x-info="http://www.rsyslog.com"] exiting on signal 15.
May 22 02:00:29 server1 kernel: imklog 5.8.10, log source = /proc/kmsg started.
May 22 02:00:29 server1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1698" x-info="http://www.rsyslog.com"] start
May 22 02:17:43 server1 dhclient[916]: DHCPREQUEST on eth0 to 172.31.0.1 port 67 (xid=0x445faedb)

File can be read from left as:

  1. Date
  2. Time
  3. System hostname
  4. Service name (and sometimes PID as well)
  5. Message text

System hostname stands value when you have a centralized Syslog server. Message text and service name help you narrow your search. You can directly grep on this file to find specific service-related logs omitting other clutter. All file operations like cat, more, less, etc can be done on this file since its plain text file.

Since this log fills very fast on a busy system (as it logs almost everything), you should consider configuring logrotate for it so that it won’t full your mount point.

Service success/failures at boot : /var/log/boot.log

At the time of booting Linux server, you can see services being started and their success or failure status is displayed on local console. The same logs can be obtained from the boot log post-boot. This file lists all service’s success/failure status at boot time so that it can be referred later to troubleshoot any service-related issues.

$ cat /var/log/boot.log
growroot: FAILED: GPT partition found but no sgdisk
                Welcome to Red Hat Enterprise Linux Server
Starting udev: udevd[404]: can not read 'https://z5.kerneltalks.com/etc/udev/rules.d/75-persistent-net-generator.rules'
udevd[404]: can not read 'https://z5.kerneltalks.com/etc/udev/rules.d/75-persistent-net-generator.rules'

                                                           [  OK  ]
Setting hostname ip-172-31-1-120.ap-south-1.compute.interna[  OK  ]
Setting up Logical Volume Management:                      [  OK  ]
Checking filesystems
/dev/xvda1: clean, 77980/393216 files, 811583/1572864 blocks
                                                           [  OK  ]
Remounting root filesystem in read-write mode:             [  OK  ]
Mounting local filesystems:                                [  OK  ]
Enabling local filesystem quotas:                          [  OK  ]
Enabling /etc/fstab swaps:                                 [  OK  ]
Entering non-interactive startup

The above sample, file shows services, their status (on right), and any error/warning messages written to console (by service daemons).

Authentication logs : /var/log/secure, /var/log/auth.log

This log file is crucial to check user access logs. Log files stores information on user logins along with authentication used. It also stores sudo logs.

Jun 14 23:41:00 server1 sshd[1586]: Accepted publickey for ec2-user from 59.184.130.135 port 51265 ssh2
Jun 14 23:41:01 server1 sshd[1586]: pam_unix(sshd:session): session opened for user ec2-user by (uid=0)
Jun 14 23:41:04 server1 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=/bin/su -
Jun 14 23:41:04 server1 su: pam_unix(su-l:session): session opened for user root by ec2-user(uid=0)
Jun 14 23:43:45 server1 su: pam_unix(su-l:session): session closed for user root

Log file can be read from left to right as :

  1. Date
  2. Time
  3. Server hostname
  4. Authentication service or daemon (sometimes along with PID)
  5. Message

Login records : /var/log/utmp, /var/log/wtmp

These wtmp or utmp file stores user login details. utmp stores login details like id, time, duration, the terminal used, system reboot details, etc. wtmp provides historical utmp data.  These files are not plain text files and hence need to be parsed to last command to read data within. You can use last -f <path> to read file.

ec2-user pts/1        59.184.170.243   Mon Jun 19 07:24   still logged in
ec2-user pts/0        59.184.170.243   Mon Jun 19 07:21 - 07:24  (00:02)
reboot   system boot  2.6.32-696.el6.x Mon Jun 19 07:20 - 07:39  (00:18)
ec2-user pts/0        59.184.130.135   Wed Jun 14 23:41 - 00:09  (00:28)
reboot   system boot  2.6.32-696.el6.x Wed Jun 14 23:40 - 00:09  (00:29)

In above output of last -f /var/log/wtmp you can see from left to right :

  1. User or event (reboot)
  2. Terminal used
  3. IP from which user was connected
  4. Date/time of login
  5. Date /time of log out
  6. Duration of session

Failed login records : /var/log/btmp

Its file dedicated to logging only failed login attempts. This too can be read using last command as mentioned above.

user1    ssh:notty    31.207.47.50     Tue Jun 13 01:18    gone - no logout
user     ssh:notty    31.207.47.50     Tue Jun 13 01:18 - 01:18  (00:00)
ubnt     ssh:notty    31.207.47.50     Tue Jun 13 01:18 - 01:18  (00:00)

It also shows user id, terminal, source IP, and time who tried to log into the system but failed to do so.

Scheduler logs : /var/log/cron

Cron (Linux scheduler) logs are saved under /var/log/cron. Job run status, daemon logs are saved here. It helps in understanding job execution and troubleshooting of scheduled jobs. Its plain text file and supports all normal file operations.

Jun 15 00:09:51 server1 crond[1473]: (CRON) INFO (Shutting down)
Jun 19 07:21:18 server1 crond[1474]: (CRON) STARTUP (1.4.4)
Jun 19 07:21:18 server1 crond[1474]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 26% if used.)
Jun 19 07:21:20 server1 crond[1474]: (CRON) INFO (running with inotify support)
Jun 19 07:30:01 server1 CROND[1676]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jun 19 07:40:01 server1 CROND[1702]: (root) CMD (/usr/lib64/sa/sa1 1 1)

Since the server was shut on 15 Jun and started on 19 Jun you can see cron daemon wrote down and up logs in this log file. For job executed, log code CMD is used. Daemon messages are tagged with INFO.

Mail logs : /var/log/maillog

Mail related logs saved here. It logs date-time, hostname, mail protocol (with PID sometimes), and messages.

Jun 15 00:09:51 server1 postfix/postfix-script[1939]: stopping the Postfix mail system
Jun 15 00:09:51 server1 postfix/master[1432]: terminating on signal 15
Jun 19 07:21:17 server1 postfix/postfix-script[1432]: starting the Postfix mail system
Jun 19 07:21:17 server1 postfix/master[1433]: daemon started -- version 2.6.6, configuration /etc/postfix

Since our server doesn’t have an SMTP configuration, there is nothing much in our file shown above.

File transfer logs : /var/log/xferlog

This log contains information from file transfer program/service daemons.

Mon Jun 19 04:44:43 2017 1 10.4.1.22 20 /CSV/test1.csv a _ o r testuser ftp 0 * c
Mon Jun 19 04:51:33 2017 1 10.4.1.22 432 /CSV/test4.csv a _ o r testuser ftp 0 * c
Mon Jun 19 04:57:15 2017 1 10.4.1.22 110 /CSV/test14.csv a _ o r testuser ftp 0 * c
Mon Jun 19 04:57:19 2017 1 10.4.1.22 2505 /CSV/master.csv a _ o r testuser ftp 0 * c

Sample log files hows date, time, source IP, file path copied, options used, the user who copied files, the protocol used details.

Last login details : /var/log/lastlog

System’s all user’s recent login details are saved to this log file. Its not plain text file and hence you need to use lastlog command which reads data from this log file and print on the terminal in a human-readable format. It sorts users with their order in /etc/passwd file.

# lastlog
Username         Port     From             Latest
root                                       **Never logged in**
bin                                        **Never logged in**
oprofile                                   **Never logged in**
tcpdump                                    **Never logged in**
ec2-user         pts/1    59.184.170.243   Mon Jun 19 07:24:14 -0400 2017
apache                                     **Never logged in**

Above output is pretty self explanatory.

Device driver messages : dmesg

Boot time device/driver writes messages on the console. Those messages can be viewed as post boot as well using dmesg command. These messages help in troubleshooting devices or driver initialization issues.

# dmesg
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-696.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC) ) #1 SMP Tue Feb 21 00:53:17 EST 2017
Command line: console=ttyS0 ro root=UUID=1b7ea291-67b4-48da-802a-be4b2bcb64d5 rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 xen_blkfront.sda_is_xvda=1 console=ttyS0,115200n8 console=tty0 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009e000 (usable)
 BIOS-e820: 000000000009e000 - 00000000000a0000 (reserved)

----- output trimmed -----

Those are all boot time messages just before /var/log/boot.log

System crash dump : /var/crash

This is the directory under which the core-dump is saved in case of a system crash. Obviously you need to configure dump configs on the system for it. This information is very important to get the root cause of the system crash. This is almost the image of the current state of the system at the time of the crash. Most of the vendor asks this dump files to analyze when we log a case with them for a system crash.