Tag Archives: clnt_create rpc port mapper failure – timed out ubuntu

How to troubleshoot RPC: Port mapper failure – Timed out error

Learn how to troubleshoot RPC: Port mapper failure – Timed out error on NFS client. This will help you to resolve NFS mounting being timed out issue.

RPC: Port mapper failure – Timed out

In this article, we are going to discuss the troubleshooting of one of the NFS errors you see on NFS clients. This error can be seen while trying commands related to NFS like below :

root@kerneltalks # showmount -e mynfsserver
clnt_create: RPC: Port mapper failure - Timed out

root@kerneltalks # rpcinfo -p  mynfsserver
mynfsserver: RPC: Port mapper failure - Timed out

Normally when you see this error you are not able to mount NFS share as well. You will see mount.nfs: Connection timed out error when you try to mount NFS share.

root@kerneltalks # mount mynfsserver:/data /nfs_data
mount.nfs: Connection timed out

Troubleshooting steps

Follow below troubleshooting steps to fix RPC: Port mapper failure - Timed out error.

Check NFS services on NFS server

First, check if NFS server services are running smoothly on the NFS server.

root@mynfsserver # service nfs-server status
nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled)
  Drop-In: /usr/lib/systemd/system/nfs-server.service.d
           └─nfsserver.conf
        /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Tue 2017-11-07 15:58:08 BRST; 6 days ago
 Main PID: 1586 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

The above output is from the Suse Linux server. The output may look different n different Linux distros. If it’s not running or hung, you may need to restart NFS services.

Check connectivity between NFS server and client

Make sure you are able to reach the NFS server from your client. Check using ping and telnet to NFS ports like 111 and 2049 over both protocols TCP and UDP.

root@kerneltalks #  ping mynfsserver
PING lasnfsp01v.la.holcim.net (10.186.1.22) 56(84) bytes of data.
64 bytes from 10.186.1.22: icmp_seq=1 ttl=56 time=3.92 ms
64 bytes from 10.186.1.22: icmp_seq=2 ttl=56 time=3.74 ms
64 bytes from 10.186.1.22: icmp_seq=3 ttl=56 time=3.82 ms
^C
--- mynfsserver ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.748/3.830/3.920/0.086 ms

root@kerneltalks # telnet 10.186.1.22 2049
Trying 10.186.1.22...
Connected to 10.186.1.22.
Escape character is '^]'.
root@kerneltalks # nc -v -u mynfsserver 111
Connection to mynfsserver 111 port [udp/sunrpc] succeeded!
^C
root@kerneltalks # nc -v -u mynfsserver 2049
Connection to mynfsserver 2049 port [udp/nfs] succeeded!
^C
root@kerneltalks # nc -v mynfsserver 111
Connection to mynfsserver 111 port [tcp/sunrpc] succeeded!
^C
root@kerneltalks # nc -v mynfsserver 2049
Connection to mynfsserver 2049 port [tcp/nfs] succeeded!
^C
Check if RPC info is reachable from client

Run below command to check if you can read RPC information of the NFS server from the client machine.

root@kerneltalks # rpcinfo -p 10.186.1.22
   program vers proto   port  service
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    2   udp  20048  mountd
    100005    2   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100024    1   udp   4000  status
    100024    1   tcp   4000  status
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049  nfs_acl
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    3   udp   2049  nfs_acl
    100021    1   udp   4001  nlockmgr
    100021    3   udp   4001  nlockmgr
    100021    4   udp   4001  nlockmgr
    100021    1   tcp   4001  nlockmgr
    100021    3   tcp   4001  nlockmgr
    100021    4   tcp   4001  nlockmgr

If you have connectivity issue then you will see mynfsserver: RPC: Port mapper failure - Timed out error here.

Check if you can read exported share info from client

Use the below command to check if you can read exported share info from the client.

root@kerneltalks # showmount -e  10.186.1.22
Export list for  10.186.1.22:
/data *(rw,sync,no_root_squash)
Check if antivirus kernel modules are blocking the NFS

Lastly, if you have SEP 14 (Symantec Endpoint Protection) antivirus installed on your server then you need to uninstall it. For some mysterious reason, SEP 14 holds on nfsd and crashes everything to NFS. You may see below messages in dmesg of NFS server to verify if SEP kernel modules are messing up with NFS

symev_custom_4_12_14_95_48_default_x86_64: loading out-of-tree module taints kernel.
symev_custom_4_12_14_95_48_default_x86_64: module verification failed: signature and/or required key missing - tainting kernel
symap_custom_4_12_14_95_48_default_x86_64: module license 'Proprietary' taints kernel.
symev: hold nfsd module
BUG: unable to handle kernel paging request at ffffffffc068f8b4
IP: svc_tcp_accept+0x1a6/0x320 [sunrpc]

You need a reboot after uninstalling antivirus since its kernel modules loaded in kernel won’t get freed with uninstall. For that, you need to reboot the server. You cant even remove it from the kernel as you normally remove the module from the running kernel. So reboot is a way to go after uninstalling antivirus.

root@kerneltalks # lsmod |grep sym
symev_custom_4_12_14_95_48_default_x86_64    98304  1

root@kerneltalks # modprobe -r symev_custom_4_12_14_95_48_default_x86_64
modprobe: FATAL: Module symev_custom_4_12_14_95_48_default_x86_64 is in use.

root@kerneltalks # rmmod  symev_custom_4_12_14_95_48_default_x86_64
rmmod: ERROR: Module symev_custom_4_12_14_95_48_default_x86_64 is in use

root@kerneltalks # lsmod | grep sym
symev_custom_4_12_14_95_48_default_x86_64    98304  1

root@kerneltalks # modinfo symev_custom_4_12_14_95_48_default_x86_64
filename:       /lib/modules/4.12.14-95.48-default/kernel/drivers/char/symev-custom-4-12-14-95-48-default-x86-64.ko
modinfo: ERROR: could not get modinfo from 'symev_custom_4_12_14_95_48_default_x86_64': No such file or directory

Once all the above commands are able to provide you expected output, you will then be ready to mount your share without any issues and the issue will be resolved.


How to resolve connectivity issue

To resolve connectivity between two servers first you need to check on network ends that two servers are able to communicate over a network. If you are running it on AWS Linux EC2 instances then you might need to check security groups to allow proper traffic.

On the OS front, you may need to check iptables settings and allow NFS ports. SELinux is also an area where you need to explore settings if you have customized SELinux running on your server. Normally by default SELinux allows NFS traffic.