Troubleshooting check_mk agent in Linux

Different steps to troubleshoot check_mk agent in Linux

Before we start you may want to check our tutorial about setting up check_mk monitoring in Linux. You might experience issues when adding a server into check_mk monitoring. Below are a few steps, you can check to validate your configuration and communication between check_mk server and client works well.

Port 6556 communication

check_mk communicate over port 6556 by default. You need to check if port 6556 is open between client and server.

Also, you need to check if port 6556 is listening on your check_mk client so that server can fetch details by querying the client on port 6556.

root@kerneltalks#  netstat -lpn | grep 6556
tcp6       0      0 :::6556                 :::*                    LISTEN      24303/xinetd

You can even check using telnet on client and from server to client.

root@kerneltalks # telnet localhost 6556
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.

How to restart check_mk agent

check_mk agent runs on top of xinetd service in Linux. So if you make any changes to the config file then you need to reload configuration or restart the agent.

Reload check_mk configuration using below command –

root@kerneltalks # service xinetd reload
Redirecting to /bin/systemctl reload xinetd.service

Restart check_mk agent using below command –

root@kerneltalks # service xinetd stop
root@kerneltalks # service xinetd start

check_mk configuration file

Make sure you have a proper configuration file in place. check_mk configuration file is located at /etc/xinetd.d/check_mk_agent. The sample configuration file is as below –

# Created by Check_MK Agent Bakery.
# This file is managed via WATO, do not edit manually or you
# lose your changes next time when you update the agent.


service check_mk_agent
{
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/bin/check_mk_agent
        log_on_success =
        disable        = no
        only_from      = 10.10.1.2 10.10.1.3
}

only_from field denotes check_mk server IPs to which your agent will communicate. It will be populated by RPM you used to install. If it doesn’t reflect proper values you can edit it and reload check_mk configuration by using commands mentioned above.

check_mk agent updater

Check update for check_mk agent and download from the server using check_mk updater.

root@kerneltalks #  cmk-update-agent -v -v
+-------------------------------------------------------------------+
|                                                                   |
|  Check_MK Agent Updater - Update                                  |
|                                                                   |
+-------------------------------------------------------------------+
Read /etc/cmk-update-agent.state.
Getting target agent configuration from deployment server
Fetching URL: http://kerneltalks1/master/check_mk/deploy_agent.py?...
Response from deployment server:
AgentAvailable: False

Read /etc/cmk-update-agent.state.
Saved deployment status to /etc/cmk-update-agent.state.
No agent available for us.

Using this command you can even see to which server client is registered. If the client is not registered with check_mk server then you will see below error –

root@kerneltalks # cmk-update-agent -v -v
+-------------------------------------------------------------------+
|                                                                   |
|  Check_MK Agent Updater - Update                                  |
|                                                                   |
+-------------------------------------------------------------------+
Read /etc/cmk-update-agent.state.
Getting target agent configuration from deployment server
Fetching URL: http://kerneltalks1/master/check_mk/deploy_agent.py?...
ERROR This host is not registered for deployment

cmk-update-agent can not execute binary error

If you come across below error while executing cmk-update-agent command then it’s probably you are running binary of architecture it is not meant to run. e.g. you are running x86 binary on ARM server

/usr/bin/cmk-update-agent: line 8: /usr/lib/check_mk_agent/plugins/21600/cmk-update-agent: cannot execute binary file

Solution: Make sure you install the proper architecture package on your server which matches your server architecture. Generally x86 check_mk package is marked as noarch package but it will land you up in the above issue when you install it on the ARM arch server.

check_mk xinetd error

---------------------------------------------
WARNING
This package needs xinetd to be installed.
Currently you do not have installed xinetd.
Please install and start xinetd or install
and setup another inetd manually.
It's also possible to monitor via SSH without
an inetd.
---------------------------------------------

Solution: Install xinetd package. Start and enable xinetd service. If you are running a newer kernel/OS like Suse 15 SP1, xinetd is not available in it. Its sockets being served by systemd. You need to install a higher version of check_mk agent RPM which identifies systemd sockets and it won’t show you above error. e.g. version 1.4 packages were showing above error to me while when I used version 1.5 package, it got away and installation succeeded.

check_mk python error

---------------------------------------------
ERROR
Failed to install agent: Could not find a
'python' interpreter.
---------------------------------------------

Solution: It means you have a higher version of python like python3 and agent rpm is looking for python. Use the higher version of check_mk agent which understands the latest python binaries (which has nomenclature with like python3 and not python)

How to add new disk in Linux

Leave a reply

Step by step procedure to add disk in Linux machine

In this article, we will walk you through steps to add a new disk in the Linux machine. Adding a raw disk to the Linux machine may vary depending upon the type of server you have but once the disk is presented to the machine, the procedure of getting it to mount points is almost the same.

Objective: Add a new 10GB disk to the server and create a 5GB mount point out of it using LVM and newly created volume group.

Adding raw disk to Linux machine

If you are using the AWS EC2 Linux server, you may follow these steps to add raw disk. If you are on VMware Linux VM you will have a different set of steps to follow to add disk. If you are running a physical rack-mount/blade server then adding disk will be a physical task.

Now once the disk is attached to the Linux machine physically/virtually, it will be identified by the kernel and then our rally starts.

Identifying newly added disk in Linux

After the attachment of the raw disk, you need to ask the kernel to scan a new disk. Mostly it’s done now automatically by the kernel in new versions.

First thing is to identify the newly added disk and its name in the kernel. There are numerous ways to achieve this. I will list a few –

You can observer lsblk output before and after adding/scanning disk to get a new disk name.
Check newly created disk files in /dev filesystem. Match timestamp of file and disk addition time.
Observer fdisk -l output before and after adding/scanning disk to get a new disk name.

For our example, I am using the AWS EC2 server and I added 5GB disk to my server. here is my lsblk output –

[root@kerneltalks ~]# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  10G  0 disk
├─xvda1 202:1    0   1M  0 part
└─xvda2 202:2    0  10G  0 part /
xvdf    202:80   0  10G  0 disk

You can see xvdf is our newly added disk. Full path for disk is /dev/xvdf.

Add new disk in LVM

We are using LVM here since its widely used and flexible volume manager on the Linux platform. Make sure you have lvm or lvm2 package installed on your system. If not, install lvm/lvm2 package.

Now, we are going to add this RAW disk in Logical Volume Manager and create 10GB of mount point out of it. List of commands you need to follow are –

If you are willing to add a disk to the existing mount point and use its space to extend mount point then vgcreate should be replaced by vgextend.

Sample outputs from my session –

[root@kerneltalks ~]# pvcreate /dev/xvdf
  Physical volume "/dev/xvdf" successfully created.
[root@kerneltalks ~]# vgcreate vgdata /dev/xvdf
  Volume group "vgdata" successfully created
[root@kerneltalks ~]# lvcreate -L 5G -n lvdata vgdata
  Logical volume "lvdata" created.

Now, you have a logical volume created. You need to format it with the filesystem on your choice and mount it. We are choosing ext4 filesystem here and formatting using mkfs.ext4 .

[root@kerneltalks ~]# mkfs.ext4 /dev/vgdata/lvdata
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1310720 blocks
65536 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Mounting volume from new disk on mount point

Lets mount the logical volume of 5GB which we created and formatted on /data mount point using mount command.

[root@kerneltalks ~]# mount /dev/vgdata/lvdata /data
[root@kerneltalks ~]# df -Ph /data
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/vgdata-lvdata  4.8G   20M  4.6G   1% /data

Verify your mount point with df command as above and you are all done! You can always add an entry in /etc/fstab to make this mount persistent over reboots.

You have attached a 10GB disk to the Linux machine and created a 5GB mount point out of it!

How to use sudo access in winSCP

4 Replies

Learn how to use sudo access in winSCP with screenshots.

Before you move into configurations, make sure that the user is having NOPASSWD access to sudo to target user or root account. This means /etc/sudoers file must have entry something like below –

user1 ALL=(root) NOPASSWD: ALL

Once you have proper sudo configuration you can go ahead with WinSCP config. This is to ensure that once shell tries sudo it should be non-interactive.

First of all, you need to check where is your SFTP server binary located on the server you are trying to connect with WinSCP.

You can check SFTP server binary location with below command –

[root@kerneltalks ~]# cat /etc/ssh/sshd_config |grep -i sftp-server
Subsystem sftp  /usr/libexec/openssh/sftp-server

Here you can see sftp server binary is located at /usr/libexec/openssh/sftp-server

Now open winSCP and click Advanced button to open up advanced settings.

It will open up an advanced setting window like one below. Here select SFTP under Environment on the left-hand side panel. You will be presented with an option on the right hand side.

Now, add SFTP server value here with the command sudo su -c here as displayed in the screenshot below –

So we added sudo su -c /usr/libexec/openssh/sftp-server in settings here. Now click Ok and connect to the server as you normally do.

After connection, you will be able to transfer files from the directory where you normally need sudo permission to access.

That’s it! You logged to server using WinSCP and sudo access.

How to redirect port in Linux using iptables

Leave a reply

Short post to explain how to redirect port in Linux using iptables.

In this short tutorial, we will walk you through the process to redirect port using iptables. How to check port redirection in Linux and how to save iptables rules.

Here are few iptables tutorials for your basics :

Our requirement is to redirect port 80 to port 8080 in the same server. This can be done by adding rules in PREROUTING chain. So run below command –

[root@kerneltalks ~]# iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080

If you have an interface name other than eth0 then you need to edit your command accordingly. You can even add your source and destinations as well in same command using --src and --dst options. Without them, it’s assumed to any source and any destination.

How to check port redirection in iptable

Verify port redirect rule in iptables using below command –

[root@kerneltalks ~]# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REDIRECT   tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 redir ports 8080
..............

You can see port 80 is being redirected to port 8080 on the server. Note here target is REDIRECT. Do not get confused with port redirection with port forwarding.

How to save iptables rules

To save iptables rules and make them persistent over reboots use below command –

[root@kerneltalks ~]# iptables-save

How to forward port using iptables in Linux

2 Replies

Quick article to demonstrate how to configure port forwarding in Linux using iptables.

In this article, we will walk you through port forwarding using iptables in Linux. First of all, you need to check if port forwarding is enabled or not on your server. For better understanding, we will be using eth0 as a reference interface and all our command executions will be related to eth0 in this article.

How to check if port forwarding is enabled in Linux

Either you can use sysctl to check if forwarding is enabled or not. Use below command to check –

[root@kerneltalks ~]#  sysctl -a |grep -i eth0.forwarding
net.ipv4.conf.eth0.forwarding = 0
net.ipv6.conf.eth0.forwarding = 0

Since both values are zero, port forwarding is disabled for ipv4 and ipv6 on interface eth0.

Or you can use the process filesystem to check if port forwarding is enabled or not.

[root@kerneltalks ~]# cat /proc/sys/net/ipv4/conf/eth0/forwarding
0
[root@kerneltalks ~]# cat /proc/sys/net/ipv6/conf/eth0/forwarding
0

Again here process FS with zero values confirms port forwarding is disabled on our system. Now we need to first enable port forwarding on our system then we will configure port forwarding rules in iptables.

How to enable port forwarding in Linux

As we checked above, using the same methods you can enable port forwarding in Linux. But its recommended using sysctl command rather than replacing 0 by 1 in proc files.

Enable port forwarding in Linux using sysctl command –

[root@kerneltalks ~]# sysctl net.ipv4.conf.eth0.forwarding=1
net.ipv4.conf.eth0.forwarding = 1
[root@kerneltalks ~]# sysctl net.ipv6.conf.eth0.forwarding=1
net.ipv6.conf.eth0.forwarding = 1

To make it persistent over reboots, add parameters in /etc/sysctl.conf

[root@kerneltalks ~]# echo "net.ipv4.conf.eth0.forwarding = 1">>/etc/sysctl.conf
[root@kerneltalks ~]# echo "net.ipv6.conf.eth0.forwarding = 1">>/etc/sysctl.conf
[root@kerneltalks ~]# sysctl -p
net.ipv4.conf.eth0.forwarding = 1
net.ipv6.conf.eth0.forwarding = 1

Now, we have port forwarding enabled on our server, we can go ahead with configuring port forwarding rules using iptables.

How to forward port in Linux

Here we will forward port 80 to port 8080 on 172.31.40.29. Do not get confused port forwarding with port redirection.

We need to insert an entry in PREROUTING chain of iptables with DNAT target. Command will be as follows –

# iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j DNAT --to 172.31.40.29:8080
# iptables -A FORWARD -p tcp -d 172.31.40.29 --dport 8080 -j ACCEPT

Change interface, IP and ports as per your requirement. The first command tells us to redirect packets coming to port 80 to IP 172.31.40.29 on port 8080. Now packet also needs to go through FORWARD chain so we are allowing in in the second command.

Now rules have been applied. You need to verify them.

How to check port forwarding iptables rules

Command to verify port forwarding rules is –

[root@kerneltalks ~]# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.31.40.29:8080

Here REDIRECT target means its a redirection rule. Since we have configured forwarding rule we see the target as DNAT

How to save iptables rules

To save iptables rules and make them persistent over reboots use below command –

[root@kerneltalks ~]# iptables-save

Script to create mount points in LVM

2 Replies

Here is a little script to create a mount point using CSV file which has a mount point name, size, and VG name.

Caution : Use script on your own risk!
Do not use it on production servers. Test it and use it on newly built/dev/testing servers.

Below is the script code. Save it under /tmp/lvm_script.sh and also save your CSV file under the same directory with the name list.csv

CSV file format is mount point name,size in GB,VG name. For example : /data,10,data_vg

Script code :

#Script to create mount point using CSV file
#Author : Shrikant Lavhate (kerneltalks.com)
#Save CSV file as list.csv in current working directory with format mount point name,size in GB,VG name

chckfail()
{
        if [ $? -ne 0 ];then
                echo "Check error above. Halting..."
                exit 1
        fi
}

for i in `cat list.csv`
do
        kt_mountname=`echo $i | cut -d, -f1`
        kt_lvname=`echo $i |cut -d, -f1|cut -c 2-|tr / _`
        kt_vgname=`echo $i | cut -d, -f3`
        kt_lvsize=`echo $i | cut -d, -f2`
        kt_lvsize="${kt_lvsize}G"
        lvcreate -n $kt_lvname -L $kt_lvsize $kt_vgname >/dev/null
        chckfail
        mkfs.ext4 /dev/$kt_vgname/$kt_lvname >/dev/null
        chckfail
        mkdir -p $kt_mountname >/dev/null
        chckfail
        mount /dev/$kt_vgname/$kt_lvname $kt_mountname>/dev/null
        chckfail
        echo "/dev/$kt_vgname/$kt_lvname $kt_mountname ext4 defaults 0 0">>/etc/fstab
        chckfail
done

Breaking the code :

Quick walk through above code.

Part one is chckfail function which used to check if the command ran is successful or not. If the command failed, it will stop the execution of the script and exits.
Variable part extracts mount point name, size, VG to be used details from CSV file. It also creates LV names out of mount point name in CSV
Standard LVM commands to create LV, format it with EXT4, create mount point directory, and mount LV on it.
Finally, it adds an entry to /etc/fstab for the persistent mount.

Modifying script for your requirement :

If you are using size in MB then remove line kt_lvsize="${kt_lvsize}G"
If you are using size in TB then replace G with T in above mentioned line.
If you are using filesystem other than ext4 then change mkfs.ext4 & /etc/fstab command accordingly.

One liner scripts to ease your Linux tasks

Leave a reply

An assorted collection of one-liner scripts that are helpful in Linux sysadmin’s day to day tasks.

In this article, I am consolidating many one-liner scripts that I used or came across which will help you to perform Linux day to day tasks. Great way to save your time in repetitive work ensuring zero human errors!

Setting up hostname in SUSE (older versions)

I always prefer hostnamectl to set hostname in systems running on newer kernels.

# echo myserver.mydomain.com > /etc/HOSTNAME
# sed --in-place 's/preserve_hostname: false/preserve_hostname: true/' /etc/cloud/cloud.cfg #For Cloud servers
# sed --in-place 's/DHCLIENT_SET_HOSTNAME="yes"/DHCLIENT_SET_HOSTNAME="no"/' /etc/sysconfig/network/dhcp
# hostname myserver

Add your hostname instead of myserver and your FQDN domain instead of mydomain.

Setting up nameservers in Linux

# echo "nameserver 10.8.14.33 #Lab nameserver1
nameserver 10.8.17.33 #Lab nameserver2
search lab.kerneltalks.com">>/etc/resolv.conf

Add your own nameserver IPs and search domain in above code.

Add FQDN in hostfile

This applies to server with single IP allocated only.

# echo "`hostname -I` `hostname`.labs.kerneltalks.com `hostname`">>/etc/hosts

Add your own domain instead of labs.kerneltalks.com

Configure sudo so that it asks user’s password when user tries to execute sudo

# sed --in-place 's/Defaults targetpw/#Defaults targetpw/' /etc/sudoers

Remove existing NTP servers and add new in /etc/ntp.conf

# sed -e '/^server/s/^/#/g' -i /etc/ntp.conf
# echo "server 10.8.14.8 #Lab NTP1
server 10.8.14.9 #Lab NTP2">>/etc/ntp.conf

Commands to enable root access in Linux server

Below is a list of the commands you can execute to enable root access on the Cloud server or AWS Linux server.

# sed --in-place 's/PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config

# sed --in-place 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
# service sshd restart
# passwd root

If you are doing it on a public cloud server make sure that you reset the root account password since cloud server spin up with key-based authentication and their root does not carry a password initially.

Test port connectivity using telnet and exit in single command

# echo 'exit' | telnet 10.10.0.1 7657
Trying 10.10.0.1...
Connected to 10.10.0.1.
Escape character is '^]'.
Connection closed by foreign host.

How to upgrade SUSE 12 SP1 to SP3 or SP4

2 Replies

Short article to demonstrate how to upgrade SUSE 12 SP1 to SP3 and SP4

First, you need to install zypper-migration-plugin . This plugin helps you in the migration from a lower service pack to the higher service pack.

kerneltalks:~ # zypper in zypper-migration-plugin
Refreshing service 'SMT-http_smt-ec2_susecloud_net'.
Refreshing service 'cloud_update'.
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW package is going to be installed:
  zypper-migration-plugin

1 new package to install.
Overall download size: 10.5 KiB. Already cached: 0 B. After the operation, additional 16.7 KiB will be used.
Continue? [y/n/? shows all options] (y): y
Retrieving package zypper-migration-plugin-0.10-9.1.noarch                                                                         (1/1),  10.5 KiB ( 16.7 KiB unpacked)
Retrieving: zypper-migration-plugin-0.10-9.1.noarch.rpm ..........................................................................................................[done]
Checking for file conflicts: .....................................................................................................................................[done]
(1/1) Installing: zypper-migration-plugin-0.10-9.1.noarch ........................................................................................................[done]

Then make sure your system is patched to the current patch level. You can use the below command to install all the latest patches.

kerneltalks:~ #  zypper patch

Now, once you are ready with the backup of the current system, proceed to migrate from SP1 to SP2. Use command zypper migration and you can see a list of service pack upgrades for your system. Although, we see that we can skip SP and upgrade to higher service packs, its

kerneltalks:~ # zypper migration

Executing 'zypper  refresh'

Refreshing service 'cloud_update'.
........
All repositories have been refreshed.

Executing 'zypper  --no-refresh patch-check --updatestack-only'

Loading repository data...
Reading installed packages...
0 patches needed (0 security patches)

Unavailable migrations (product is not mirrored):

        SUSE Linux Enterprise High Performance Computing 12 SP3 x86_64 (not available)
        SUSE Linux Enterprise Software Development Kit 12 SP3 x86_64
        Advanced Systems Management Module 12 x86_64 (already installed)
        Containers Module 12 x86_64 (already installed)
        Public Cloud Module 12 x86_64 (already installed)
        Legacy Module 12 x86_64 (already installed)
        Web and Scripting Module 12 x86_64 (already installed)
        Toolchain Module 12 x86_64 (already installed)

        SUSE Linux Enterprise High Performance Computing 12 SP2 x86_64 (not available)
        SUSE Linux Enterprise Software Development Kit 12 SP2 x86_64
        Advanced Systems Management Module 12 x86_64 (already installed)
        Containers Module 12 x86_64 (already installed)
        Public Cloud Module 12 x86_64 (already installed)
        Legacy Module 12 x86_64 (already installed)
        Web and Scripting Module 12 x86_64 (already installed)
        Toolchain Module 12 x86_64 (already installed)


Available migrations:

    1 | SUSE Linux Enterprise Server 12 SP4 x86_64
        SUSE Linux Enterprise Software Development Kit 12 SP4 x86_64
        Advanced Systems Management Module 12 x86_64 (already installed)
        Containers Module 12 x86_64 (already installed)
        Public Cloud Module 12 x86_64 (already installed)
        Legacy Module 12 x86_64 (already installed)
        Web and Scripting Module 12 x86_64 (already installed)
        Toolchain Module 12 x86_64 (already installed)

    2 | SUSE Linux Enterprise Server 12 SP3 x86_64
        SUSE Linux Enterprise Software Development Kit 12 SP3 x86_64
        Advanced Systems Management Module 12 x86_64 (already installed)
        Containers Module 12 x86_64 (already installed)
        Public Cloud Module 12 x86_64 (already installed)
        Legacy Module 12 x86_64 (already installed)
        Web and Scripting Module 12 x86_64 (already installed)
        Toolchain Module 12 x86_64 (already installed)

    3 | SUSE Linux Enterprise Server 12 SP2 x86_64
        SUSE Linux Enterprise Software Development Kit 12 SP2 x86_64
        Advanced Systems Management Module 12 x86_64 (already installed)
        Containers Module 12 x86_64 (already installed)
        Public Cloud Module 12 x86_64 (already installed)
        Legacy Module 12 x86_64 (already installed)
        Web and Scripting Module 12 x86_64 (already installed)
        Toolchain Module 12 x86_64 (already installed)


[num/q]:

You can see the migration plugin gave us the choice to jump from SP1 to SP2 or SP3 or SP4. Enter numeric against your choice and then it will upgrade related packages on your system. Here we select to go from SP1 to SP2 by tying 3.

[num/q]: 3

Executing 'snapper create --type pre --cleanup-algorithm=number --print-number --userdata important=yes --description 'before online migration''

sh: snapper: command not found
Upgrading product SUSE Linux Enterprise Server 12 SP2 x86_64.
Upgrading product SUSE Linux Enterprise Software Development Kit 12 SP2 x86_64.
Upgrading product Advanced Systems Management Module 12 x86_64.
Upgrading product Containers Module 12 x86_64.
Upgrading product Public Cloud Module 12 x86_64.
Upgrading product Legacy Module 12 x86_64.
Upgrading product Web and Scripting Module 12 x86_64.
Upgrading product Toolchain Module 12 x86_64.

Executing 'zypper --releasever 12.2 ref -f'
...................................

Once completed reboot system. Check OS version and you can see we are upgraded from SP1 to SP2

kerneltalks:~ # cat /etc/os-release
NAME="SLES"
VERSION="12-SP2"
VERSION_ID="12.2"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP2"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp2"

Now, repeat process to upgrade OS from SP2 to SP3 and SP4.

Troubleshooting errors seen in Linux

Leave a reply

I am consolidating errors I came across and their solution in quick words for easy reference to me and you as well!

Error saw while starting the MariaDB server process on RHEL 6

# service mysql start
mysql: unrecognized service

Solution: You do not have MariaDB installed on your server. Install MariaDB

Error while starting MariaDB server process n RHEL 7

# systemctl start mariadb
Failed to issue method call: Unit mariadb.service failed to load: No such file or directory.

Solution: You do not have MariaDB installed. Install mariadb-server package

Error while installing Symantec Antivirus

which: no uudecode in (/usr/sbin:/usr/bin:/bin)
ERROR: Required utility missing: uudecode. Please install this
utility before using this Intelligent Updater package.

Solution : uudecode is provided by sharutils package. Install sharutils package.

Error while exporting a filesystem

# exportfs -ra
exportfs: 34.89.123.45:/data: Function not implemented

Solution: Check and start the nfs-server process.

Error while listing directory files

# ls -lrt
ls: cannot open directory '.': Permission denied

Solution: Your directory does not have read permission to the owner. Sometimes due to windows to Linux file copy etc. Set permission and you are good to go. Use the command in the same directory # chmod -R +r .

Error while querying NTP

# ntpq -p
localhost: timed out, nothing received
***Request timed out

Solution : Edit /etc/ntp.conf and replace restrict 127.0.0.1 to restrict localhost then restart ntpd service with systemctl restart ntpd

Error during mounting of the file system

# mount /dev/vg01/lvol0 /dump
mount: unknown filesystem type '(null)'

Solution: You are trying to mount a file system which is not formatted yet. Format filesystem and then try mounting.

Error while mounting other system’s disk

I was trying to mount a disk from another server in AWS and it was not mounting. I checked dmesg and got below error :

[  792.138218] XFS (xvdh2): Filesystem has duplicate UUID d295b18a-2a70-4260-9f59-60e51432ea92 - can't mount

Solution: Since I was doing some research I temporarily mounted it without UUID. using below command –

root@kerneltalks # mount -t xfs -o nouuid /dev/xvdh2 /disk1

But ideally, you should have unique UUID to all disks on the system and you can generate UUID in such a case using XFS utility.

keytool command not found

keytool is used to generate key or CSR for SSL certificate.

# keytool -genkey -alias server -keyalg RSA -keystore kerneltalks.jks                                                             -keysize 2048
If 'keytool' is not a typo you can use command-not-found to lookup the package that contains it, like this:
    cnf keytool

Solution: Make sure you have JRE installed (Java Runtime Environment). Goto JRE binary directory and then run this command.

java version typo

# /usr/bin/java version
Error: Could not find or load main class version

Its java trying to load the program named version. You missed hyphen there!

Solution: Try below command

# java -version
java version "1.7.0_211"
OpenJDK Runtime Environment (rhel-2.6.17.1.0.1.el7_6-x86_64 u211-b02)
OpenJDK 64-Bit Server VM (build 24.211-b02, mixed mode)

Bad magic number in super-block

Error below seen while trying to resize filesystem in RHEL7

# resize2fs /dev/mapper/vg01-data
 resize2fs 1.42.9 (28-Dec-2013)
 resize2fs: Bad magic number in super-block while trying to open /dev/mapper/vg01-data
 Couldn't find valid filesystem superblock.

Solution: This is because RHEL7 has the XFS filesystem by default so you need to use xfs_growfs command to resize the filesystem.

# xfs_growfs  /dev/vg01/data
meta-data=/dev/mapper/vg01-root  isize=256    agcount=4, agsize=851968 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=3407872, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 3407872 to 7863296

How to change DocumentRoot in Apache2 to different directory than /srv/www/htdocs

Apache2 has by default DocumentRoot set to /srv/www/html. If you want to change it to some different directory you need to change it in a couple of configuration files.

Easy way to search all those files is searched in the directory –

# grep -R "DocumentRoot" /etc/apache2
# grep -R srv /etc/apche2

Here are few files and the lines within them you need to edit.

# vi /etc/apache2/default-server.conf 
ScriptAlias /cgi-bin/ "/srv/www/cgi-bin/"
<Directory "/srv/www/cgi-bin">
DocumentRoot "/srv/www/htdocs"
<Directory "/srv/www/htdocs">

# vi /etc/apache2/vhosts.d/vhost-ssl.conf 
DocumentRoot "/srv/www/htdocs"

You need to edit /srv/www/htdocs to directory of your choice. Also, you need to change relative directories to /srv as well. Once you are done with editing, you need to restart the apache2 service and you are good to go.

server_id_usr_crc warning in Suse Manger

Repeatedly below warning is being logged in /var/log/messages in Suse Manager server 4.0

2019-08-07T20:38:02.832696+08:00 susemgr-test salt-master[12485]: [WARNING ] /usr/lib/python3.6/site-packages/salt/grains/core.py:2815: DeprecationWarning: This server_id is computed nor by Adler32 neither by CRC32. Please use "server_id_use_crc" option and define algorithm youprefer (default "Adler32"). The server_id will be computed withAdler32 by default.

Solution : Add server_id_use_crc: adler32 entry at end of the file /etc/salt/master.d/susemanager.conf and then restart the Suse Manager process.

smdba backup fails to run in cron on SUSE Manager

smdba is a DB backup tool by SUSE to be used on Suse Manager which runs on postgres database. smdba tool to be run by root and in the backend it switches to DB user to connect with database and execute database stuff. It runs manually well but when scheduled in cron it exits with the below error.

Backend error:
        Access denied to UID 'postgres' via sudo.

You can see this error in root mail or you need to redirect stderr of cron command to file and you can see it in there.

Solution: This is because the root is not able to sudo to postgres user since cron spawned process don’t have tty attached to it and your sudo most likely have Defaults requiretty active in /etc/sudoers. If you want you can disable it system-wide by putting # in front of it or add a dedicated entry for root Defaults:root !requiretty to move out of this restriction. Once done try running smdba commands via cron and they will run successfully.

/etc/resolv.conf resetting to default after reboot

Issue: My /etc/resolv.conf entries gets wiped out after reboot. Manual entries added in /etc/resolv.conf are getting deleted after reboot.

Solution: This is probably because your /etc/resolv.conf is being auto-generated by netconfig. It will be symlink to /var/run/netconfig/resolv.conf. You can disable this by setting NETCONFIG_DNS_POLICY='' in /etc/sysconfig/network/config file. It will be defined as auto, you set it to blank. Or you can edit below parameters in the same file if you want to keep the above policy parameter untouched.

NETCONFIG_DNS_STATIC_SEARCHLIST
NETCONFIG_DNS_STATIC_SERVERS
NETCONFIG_DNS_FORWARDER

Once done adjust /etc/resolv.conf by running command netconfig update -f. If after this your /etc/resolv.conf remains as it is then you are good otherwise you need to review the above settings again carefully.

If it is being reloaded by DHCP you will see below line in /etc/resolv.conf

; generated by /usr/sbin/dhclient-script

In that case you need to perform below actions.

# vi /etc/dhcp/dhclient-enter-hooks
#!/bin/sh
make_resolv_conf(){
    :
}
#chmod +x /etc/dhcp/dhclient-enter-hooks

yum command giving metadata errors

yum command showing below error :

Error while executing packages action: failed to retrieve repodata/filelists.xml.gz from Oraclelinux7-x86_64 error was [Errno -1] Metadata file does not match checksum

Solution :

Run below commands and you are good to go.

# yum clean all
# yum makecache

PAM module error

PAM unable to dlopen(https://z5.kerneltalks.com/lib64/security/pam_gnome_keyring.so): /lib64/security/pam_gnome_keyring.so: cannot open shared object file: No such file or directory
PAM adding faulty module: /lib64/security/pam_gnome_keyring.so

Solution :

Update pam packages and/or install gnome-keyring package.

Account login error with LDAP

pam_sss(sudo:auth): received for user shrikant: 10 (User not known to the underlying authentication module)

Solution :

This is because account shrikant does not exists in LDAP server account list. If this is local user to that perticular client then you can add it to ignore list in LDAP config file /etc/sssd/sssdconf in below parameters.

[nss]
filter_users = root,shrikant
filter_groups = root,dba

NFS Timeout error

# mount -v -t nfs 10.10.1.2:/data /mnt/data
mount.nfs: timeout set for Wed Jan 29 08:29:01 2020
mount.nfs: trying text-based options 'vers=4,addr=10.10.1.2,clientaddr=10.10.1.3'
mount.nfs: mount(2): Connection timed out
mount.nfs: Connection timed out

Solution :

This is because client is not able to reach NFS server. There are couple of things you should check.

TCP and UDP port 2049 and 111 should be open between client and server. Use nc -v -u <nfs_server> port
NFS server service should be running on the server
NFS client service should be running on the client
If you have SEP 14 (Symantec Endpoint Protection) antivirus running on your machines then un-install and reboot both client and server.

Warning in xclock command

# xclock
Warning: Missing charsets in String to FontSet conversion

Solution:

This is just a warning about improper environment variables. You can avoid it by exporting –

export LC_ALL=C

You can add this in the user profile file as well so that it will be exported at login and no need to exporting manually.

sssd service is not starting up

After patching or system migration like activities your sssd dont start up. When you try to start sssd service you get below errors in systemd status sssd :

sssd[16866]: Exiting the SSSD. Could not restart critical service [kerneltalks.com].
systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start System Security Services Daemon.
systemd[1]: sssd.service: Unit entered failed state.
systemd[1]: sssd.service: Failed with result 'exit-code'.

In such cases the best way to check actual errors is to check the log file located in /var/log/sssd/sssd*.log. You can see sssd logs as well as domain logs here. You need to check both.

In my case I got errors in domain log file –

[sssd[be[kerneltalks.com]]] [dp_target_init] (0x0010): Unable to load module krb5
[sssd[be[kerneltalks.com]]] [be_process_init] (0x0010): Unable to setup data provider [1432158209]: Internal Error
[sssd[be[kerneltalks.com]]] [main] (0x0010): Could not initialize backend [1432158209]
[sssd[be[kerneltalks.com]]] [dp_module_open_lib] (0x0010): Unable to load module [krb5] with path [/usr/lib64/sssd/libsss_krb5.so]: /usr/lib64/sssd/libsss_krb5.so: cannot open shared object file: No such file or directory

For this missing file, I installed sssd-krb5 package and my issue got resolved.

sssd service is running but user can not login

sssd service was running fine but showing below error in systemctl status sssd and the user was not able to log in –

 sssd_be[2338]: GSSAPI Error: An invalid name was supplied (Success)

Solution :

Add below line under section [libdefaults] in /etc/krb5.conf

rdns = false

then restart sssd service using systemctl restart sssd

Install MariaDB 5.5 in RHEL 6

Leave a reply

Learn how to install MariaDB 5.5 in RHEL 6 along with how to secure it and how to connect MariaDB from shell prompt.

MariaDB is a MySQL based database management system. It is one of the components of the famous LAMP stack. In this article, we will walk you through how to install MariaDB, how to secure MariaDB installation, and how to connect MariaDB from shell prompt.

Without any further delay lets jump in to mariadb installation.

Create repo for downloading MariaDB packages

Mostly, package managers don’t come with MariaDB packages listed in RHEL 6 or lower. So, you have to add below the repo file on your server so that you can download and install the MariaDB server using yum.

root@kerneltalks # cat /etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/5.5/rhel6-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

Install MariaDB

Now install MariaDB packages MariaDB-server and MariaDB-client

root@kerneltalks # yum install MariaDB-server MariaDB-client

Start MariaDB server proces

Run below commands to start MariaDB with the boot. chkconfig manages processes at boot so use it here.

root@kerneltalks # chkconfig --add mysql
root@kerneltalks # chkconfig --level 345 mysql on
root@kerneltalks # chkconfig |grep mysql
mysql           0:off   1:off   2:on    3:on    4:on    5:on    6:off

Now, start MariaDB server process

root@kerneltalks # service mysql start
Starting MariaDB.190110 07:15:32 mysqld_safe Logging to 'https://z5.kerneltalks.com/var/lib/mysql/kerneltalks.test.err'.
190110 07:15:32 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
.                                                          [  OK  ]

See the brief article about how to start, stop & restart MariaDB server process in Linux

Secure MariaDB installation

Use the below command to secure your MariaDB installation. Answer queries asked according to your requirement.

root@kerneltalks # mysql_secure_installation
/usr/bin/mysql_secure_installation: line 393: find_mysql_client: command not found

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

Test MariaDB server connection

Test connection to MariaDB server using below command and root account.

root@kerneltalks # mysql -u root -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 10
Server version: 5.5.62-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SELECT VERSION();
+----------------+
| VERSION()      |
+----------------+
| 5.5.62-MariaDB |
+----------------+
1 row in set (0.00 sec)

MariaDB [(none)]>