Learn how to troubleshoot RPC: Port mapper failure – Timed out error on NFS client. This will help you to resolve NFS mounting being timed out issue.
In this article, we are going to discuss the troubleshooting of one of the NFS errors you see on NFS clients. This error can be seen while trying commands related to NFS like below :
root@kerneltalks # showmount -e mynfsserver
clnt_create: RPC: Port mapper failure - Timed out
root@kerneltalks # rpcinfo -p mynfsserver
mynfsserver: RPC: Port mapper failure - Timed out
Normally when you see this error you are not able to mount NFS share as well. You will see mount.nfs: Connection timed out error when you try to mount NFS share.
root@kerneltalks # mount mynfsserver:/data /nfs_data
mount.nfs: Connection timed out
Troubleshooting steps
Follow below troubleshooting steps to fix RPC: Port mapper failure - Timed out error.
Check NFS services on NFS server
First, check if NFS server services are running smoothly on the NFS server.
root@mynfsserver # service nfs-server status
nfs-server.service - NFS server and services
Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled)
Drop-In: /usr/lib/systemd/system/nfs-server.service.d
└─nfsserver.conf
/run/systemd/generator/nfs-server.service.d
└─order-with-mounts.conf
Active: active (exited) since Tue 2017-11-07 15:58:08 BRST; 6 days ago
Main PID: 1586 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/nfs-server.service
The above output is from the Suse Linux server. The output may look different n different Linux distros. If it’s not running or hung, you may need to restart NFS services.
Check connectivity between NFS server and client
Make sure you are able to reach the NFS server from your client. Check using ping and telnet to NFS ports like 111 and 2049 over both protocols TCP and UDP.
root@kerneltalks # ping mynfsserver
PING lasnfsp01v.la.holcim.net (10.186.1.22) 56(84) bytes of data.
64 bytes from 10.186.1.22: icmp_seq=1 ttl=56 time=3.92 ms
64 bytes from 10.186.1.22: icmp_seq=2 ttl=56 time=3.74 ms
64 bytes from 10.186.1.22: icmp_seq=3 ttl=56 time=3.82 ms
^C
--- mynfsserver ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.748/3.830/3.920/0.086 ms
root@kerneltalks # telnet 10.186.1.22 2049
Trying 10.186.1.22...
Connected to 10.186.1.22.
Escape character is '^]'.
root@kerneltalks # nc -v -u mynfsserver 111
Connection to mynfsserver 111 port [udp/sunrpc] succeeded!
^C
root@kerneltalks # nc -v -u mynfsserver 2049
Connection to mynfsserver 2049 port [udp/nfs] succeeded!
^C
root@kerneltalks # nc -v mynfsserver 111
Connection to mynfsserver 111 port [tcp/sunrpc] succeeded!
^C
root@kerneltalks # nc -v mynfsserver 2049
Connection to mynfsserver 2049 port [tcp/nfs] succeeded!
^C
Check if RPC info is reachable from client
Run below command to check if you can read RPC information of the NFS server from the client machine.
root@kerneltalks # rpcinfo -p 10.186.1.22
program vers proto port service
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100005 1 udp 20048 mountd
100005 1 tcp 20048 mountd
100005 2 udp 20048 mountd
100005 2 tcp 20048 mountd
100005 3 udp 20048 mountd
100005 3 tcp 20048 mountd
100024 1 udp 4000 status
100024 1 tcp 4000 status
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 3 tcp 2049 nfs_acl
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100227 3 udp 2049 nfs_acl
100021 1 udp 4001 nlockmgr
100021 3 udp 4001 nlockmgr
100021 4 udp 4001 nlockmgr
100021 1 tcp 4001 nlockmgr
100021 3 tcp 4001 nlockmgr
100021 4 tcp 4001 nlockmgr
If you have connectivity issue then you will see mynfsserver: RPC: Port mapper failure - Timed out error here.
Check if you can read exported share info from client
Use the below command to check if you can read exported share info from the client.
root@kerneltalks # showmount -e 10.186.1.22
Export list for 10.186.1.22:
/data *(rw,sync,no_root_squash)
Check if antivirus kernel modules are blocking the NFS
Lastly, if you have SEP 14 (Symantec Endpoint Protection) antivirus installed on your server then you need to uninstall it. For some mysterious reason, SEP 14 holds on nfsd and crashes everything to NFS. You may see below messages in dmesg of NFS server to verify if SEP kernel modules are messing up with NFS
symev_custom_4_12_14_95_48_default_x86_64: loading out-of-tree module taints kernel.
symev_custom_4_12_14_95_48_default_x86_64: module verification failed: signature and/or required key missing - tainting kernel
symap_custom_4_12_14_95_48_default_x86_64: module license 'Proprietary' taints kernel.
symev: hold nfsd module
BUG: unable to handle kernel paging request at ffffffffc068f8b4
IP: svc_tcp_accept+0x1a6/0x320 [sunrpc]
You need a reboot after uninstalling antivirus since its kernel modules loaded in kernel won’t get freed with uninstall. For that, you need to reboot the server. You cant even remove it from the kernel as you normally remove the module from the running kernel. So reboot is a way to go after uninstalling antivirus.
root@kerneltalks # lsmod |grep sym
symev_custom_4_12_14_95_48_default_x86_64 98304 1
root@kerneltalks # modprobe -r symev_custom_4_12_14_95_48_default_x86_64
modprobe: FATAL: Module symev_custom_4_12_14_95_48_default_x86_64 is in use.
root@kerneltalks # rmmod symev_custom_4_12_14_95_48_default_x86_64
rmmod: ERROR: Module symev_custom_4_12_14_95_48_default_x86_64 is in use
root@kerneltalks # lsmod | grep sym
symev_custom_4_12_14_95_48_default_x86_64 98304 1
root@kerneltalks # modinfo symev_custom_4_12_14_95_48_default_x86_64
filename: /lib/modules/4.12.14-95.48-default/kernel/drivers/char/symev-custom-4-12-14-95-48-default-x86-64.ko
modinfo: ERROR: could not get modinfo from 'symev_custom_4_12_14_95_48_default_x86_64': No such file or directory
Once all the above commands are able to provide you expected output, you will then be ready to mount your share without any issues and the issue will be resolved.
How to resolve connectivity issue
To resolve connectivity between two servers first you need to check on network ends that two servers are able to communicate over a network. If you are running it on AWS Linux EC2 instances then you might need to check security groups to allow proper traffic.
On the OS front, you may need to check iptables settings and allow NFS ports. SELinux is also an area where you need to explore settings if you have customized SELinux running on your server. Normally by default SELinux allows NFS traffic.