Tag Archives: upload file from linux to s3 bucket

How to mount S3 bucket in Linux server

Leave a reply

Learn to mount an S3 bucket in RHEL, Ubuntu, CentOS Linux server. Understand how to debug issues while mounting an S3 bucket.

Mount S3 bucket in Linux

In this article we will walk through the process by which you can mount an S3 bucket on the Linux server. S3 bucket is a storage container of S3 (simple storage service) AWS service. As all traditional data centers are moving to cloud computing, it is necessary to know how to interconnect cloud and traditional services. Let’s dive into the process to mount an S3 bucket in RHEL, Ubuntu, CentOS Linux. The complete process can be done in below 3 steps:

  1. Install fuse and s3fs packages
  2. Configure access keys of your AWS account
  3. Mount S3 bucket

For this tutorial, we are assuming you have an S3 bucket ready in your AWS account with a proper permission setup. If not, follow this tutorial to create an S3 bucket in AWS.

Install fuse and s3fs packages

These packages can be found here: fuse & s3fs. You have to download them on the Linux server with tools like wget and compile them. Make sure you have their dependencies installed before you try to compile them.

Dependencies are :

for RedHat based : automake gcc gcc-c++ git libcurl-devel libxml2-devel make openssl-devel mailcap curl-devel libstdc++-devel

for Debian based : automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

Install all these packages and follow the below steps to configure fuse and s3fs. Make sure there is no package named fuse of s3fs exist in the server before you proceed. This is to avoid conflicts in installation.

Fuse installation –

Use the below commands. Use the latest Fuse download link used in wget command. You can obtain it from Github here.

# cd /usr/src/
# wget https://github.com/libfuse/libfuse/releases/download/fuse-3.1.0/fuse-3.1.0.tar.gz
# tar -zxf fuse-3.1.0.tar.gz
# cd fuse-3.1.0
# ./configure -prefix=/usr/local
# make && make install
# export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
# ldconfig
# modprobe fuse

The fuse installation output is below for your reference. Click and expand if you want to view it.

# cd /usr/src/
# wget https://github.com/libfuse/libfuse/releases/download/fuse-3.1.0/fuse-3.1.0.tar.gz
--2017-07-11 05:19:36--  https://github.com/libfuse/libfuse/releases/download/fuse-3.1.0/fuse-3.1.0.tar.gz
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/48296177/00b4f6e4-63dc-11e7-938f-c32c894af199 [following]
--2017-07-11 05:19:37--  https://github-production-release-asset-2e65be.s3.amazonaws.com/48296177/00b4f6e4-63dc-11e7-938f-c32c894af199
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.230.243
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.230.243|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 809470 (790K) [application/octet-stream]
Saving to: ‘fuse-3.1.0.tar.gz’

fuse-3.1.0.tar.gz                       100%[============================================================================>] 790.50K   479KB/s    in 1.7s

2017-07-11 05:19:40 (479 KB/s) - ‘fuse-3.1.0.tar.gz’ saved [809470/809470]

# tar -zxf fuse-3.1.0.tar.gz
# cd fuse-3.1.0

# ./configure -prefix=/usr/local
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking how to print strings... printf
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... mt
checking if mt is a manifest tool... no
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for gcc option to accept ISO C99... none needed
checking for gcc option to accept ISO Standard C... (cached) none needed
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for fork... yes
checking for setxattr... yes
checking for fdatasync... yes
checking for splice... yes
checking for vmsplice... yes
checking for utimensat... yes
checking for pipe2... yes
checking for posix_fallocate... yes
checking for fstatat... yes
checking for openat... yes
checking for readlinkat... yes
checking for struct stat.st_atim... yes
checking for struct stat.st_atimespec... no
checking for library containing dlopen... -ldl
checking for library containing clock_gettime... none required
checking for ulockmgr_op in -lulockmgr... no
checking for ld used by gcc... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking for shared library run path origin... done
checking for iconv... yes
checking for working iconv... yes
checking for iconv declaration...
         extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft);
configure: MOUNT_FUSE_PATH env var not set, using default ${sbindir}
configure: UDEV_RULES_PATH env var not set, using default ${libdir}/udev/rules.d
configure: INIT_D_PATH env var not set, using default ${sysconfdir}/init.d
checking if umount supports --fake --no-canonicalize... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating fuse3.pc
config.status: creating Makefile
config.status: creating lib/Makefile
config.status: creating util/Makefile
config.status: creating example/Makefile
config.status: creating include/Makefile
config.status: creating doc/Makefile
config.status: creating test/Makefile
config.status: creating include/config.h
config.status: executing depfiles commands
config.status: executing libtool commands

# make && make install
Making all in include
make[1]: Entering directory '/usr/src/fuse-3.1.0/include'
make  all-am
make[2]: Entering directory '/usr/src/fuse-3.1.0/include'
make[2]: Nothing to be done for 'all-am'.
make[2]: Leaving directory '/usr/src/fuse-3.1.0/include'
make[1]: Leaving directory '/usr/src/fuse-3.1.0/include'
Making all in lib
make[1]: Entering directory '/usr/src/fuse-3.1.0/lib'
  CC       fuse.lo
  CC       fuse_loop.lo
  CC       fuse_loop_mt.lo
  CC       fuse_lowlevel.lo
  CC       fuse_opt.lo
  CC       fuse_signals.lo
  CC       buffer.lo
  CC       cuse_lowlevel.lo
  CC       helper.lo
helper.c: In function ‘fuse_daemonize’:
helper.c:226:4: warning: ignoring return value of ‘read’, declared with attribute warn_unused_result [-Wunused-result]
    (void) read(waiter[0], &completed, sizeof(completed));
    ^
helper.c:235:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   (void) chdir("/");
   ^
helper.c:248:3: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result]
   (void) write(waiter[1], &completed, sizeof(completed));
   ^
helper.c:252:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result]
   (void) chdir("/");
   ^
  CC       modules/subdir.lo
  CC       modules/iconv.lo
  CC       mount.lo
  CC       mount_util.lo
mount_util.c: In function ‘mtab_needs_update’:
mount_util.c:68:4: warning: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Wunused-result]
    setreuid(0, -1);
    ^
mount_util.c:73:4: warning: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Wunused-result]
    setreuid(ruid, -1);
    ^
  CCLD     libfuse3.la
ar: `u' modifier ignored since `D' is the default (see `U')
make[1]: Leaving directory '/usr/src/fuse-3.1.0/lib'
Making all in util
make[1]: Entering directory '/usr/src/fuse-3.1.0/util'
make  all-am
make[2]: Entering directory '/usr/src/fuse-3.1.0/util'
  CC       fusermount3-fusermount.o
  CC       fusermount3-mount_util.o
mount_util.c: In function ‘mtab_needs_update’:
mount_util.c:68:4: warning: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Wunused-result]
    setreuid(0, -1);
    ^
mount_util.c:73:4: warning: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Wunused-result]
    setreuid(ruid, -1);
    ^
  CCLD     fusermount3
  CC       mount.fuse.o
  CCLD     mount.fuse3
make[2]: Leaving directory '/usr/src/fuse-3.1.0/util'
make[1]: Leaving directory '/usr/src/fuse-3.1.0/util'
Making all in example
make[1]: Entering directory '/usr/src/fuse-3.1.0/example'
  CC       passthrough.o
  CCLD     passthrough
  CC       passthrough_fh.o
  CCLD     passthrough_fh
  CC       null.o
  CCLD     null
  CC       hello.o
  CCLD     hello
  CC       hello_ll.o
  CCLD     hello_ll
  CC       ioctl.o
  CCLD     ioctl
  CC       ioctl_client-ioctl_client.o
  CCLD     ioctl_client
  CC       poll.o
  CCLD     poll
  CC       poll_client-poll_client.o
  CCLD     poll_client
  CC       passthrough_ll.o
  CCLD     passthrough_ll
  CC       notify_inval_inode.o
  CCLD     notify_inval_inode
  CC       notify_store_retrieve.o
  CCLD     notify_store_retrieve
  CC       notify_inval_entry.o
  CCLD     notify_inval_entry
  CC       cuse.o
  CCLD     cuse
  CC       cuse_client.o
  CCLD     cuse_client
make[1]: Leaving directory '/usr/src/fuse-3.1.0/example'
Making all in test
make[1]: Entering directory '/usr/src/fuse-3.1.0/test'
  CC       test_syscalls.o
  CCLD     test_syscalls
  CC       test_write_cache.o
  CCLD     test_write_cache
  CC       test_setattr.o
  CCLD     test_setattr
make[1]: Leaving directory '/usr/src/fuse-3.1.0/test'
Making all in doc
make[1]: Entering directory '/usr/src/fuse-3.1.0/doc'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/usr/src/fuse-3.1.0/doc'
make[1]: Entering directory 'https://z5.kerneltalks.com/usr/src/fuse-3.1.0'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory 'https://z5.kerneltalks.com/usr/src/fuse-3.1.0'
Making install in include
make[1]: Entering directory '/usr/src/fuse-3.1.0/include'
make[2]: Entering directory '/usr/src/fuse-3.1.0/include'
make[2]: Nothing to be done for 'install-exec-am'.
 /bin/mkdir -p '/usr/local/include/fuse3'
 /usr/bin/install -c -m 644 fuse.h fuse_common.h fuse_lowlevel.h fuse_opt.h cuse_lowlevel.h '/usr/local/include/fuse3'
make[2]: Leaving directory '/usr/src/fuse-3.1.0/include'
make[1]: Leaving directory '/usr/src/fuse-3.1.0/include'
Making install in lib
make[1]: Entering directory '/usr/src/fuse-3.1.0/lib'
make[2]: Entering directory '/usr/src/fuse-3.1.0/lib'
 /bin/mkdir -p '/usr/local/lib'
 /bin/bash ../libtool   --mode=install /usr/bin/install -c   libfuse3.la '/usr/local/lib'
libtool: install: /usr/bin/install -c .libs/libfuse3.so.3.1.0 /usr/local/lib/libfuse3.so.3.1.0
libtool: install: (cd /usr/local/lib && { ln -s -f libfuse3.so.3.1.0 libfuse3.so.3 || { rm -f libfuse3.so.3 && ln -s libfuse3.so.3.1.0 libfuse3.so.3; }; })
libtool: install: (cd /usr/local/lib && { ln -s -f libfuse3.so.3.1.0 libfuse3.so || { rm -f libfuse3.so && ln -s libfuse3.so.3.1.0 libfuse3.so; }; })
libtool: install: /usr/bin/install -c .libs/libfuse3.lai /usr/local/lib/libfuse3.la
libtool: install: /usr/bin/install -c .libs/libfuse3.a /usr/local/lib/libfuse3.a
libtool: install: chmod 644 /usr/local/lib/libfuse3.a
libtool: install: ranlib /usr/local/lib/libfuse3.a
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin" ldconfig -n /usr/local/lib
----------------------------------------------------------------------
Libraries have been installed in:
   /usr/local/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to 'https://z5.kerneltalks.com/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/usr/src/fuse-3.1.0/lib'
make[1]: Leaving directory '/usr/src/fuse-3.1.0/lib'
Making install in util
make[1]: Entering directory '/usr/src/fuse-3.1.0/util'
make  install-am
make[2]: Entering directory '/usr/src/fuse-3.1.0/util'
make[3]: Entering directory '/usr/src/fuse-3.1.0/util'
 /bin/mkdir -p '/usr/local/bin'
  /bin/bash ../libtool   --mode=install /usr/bin/install -c fusermount3 '/usr/local/bin'
libtool: install: /usr/bin/install -c fusermount3 /usr/local/bin/fusermount3
/bin/mkdir -p /usr/local/sbin
/usr/bin/install -c ./mount.fuse3 /usr/local/sbin/mount.fuse3
/bin/mkdir -p /usr/local/etc/init.d
/usr/bin/install -c ./init_script /usr/local/etc/init.d/fuse3
/usr/sbin/update-rc.d fuse start 34 S . start 41 0 6 . || true
update-rc.d: error: unable to read /etc/init.d/fuse
make  install-exec-hook
make[4]: Entering directory '/usr/src/fuse-3.1.0/util'
chmod u+s /usr/local/bin/fusermount3
make[4]: Leaving directory '/usr/src/fuse-3.1.0/util'
/bin/mkdir -p /usr/local/lib/udev/rules.d
/usr/bin/install -c -m 644 ./udev.rules /usr/local/lib/udev/rules.d/99-fuse3.rules
make[3]: Leaving directory '/usr/src/fuse-3.1.0/util'
make[2]: Leaving directory '/usr/src/fuse-3.1.0/util'
make[1]: Leaving directory '/usr/src/fuse-3.1.0/util'
Making install in example
make[1]: Entering directory '/usr/src/fuse-3.1.0/example'
make[2]: Entering directory '/usr/src/fuse-3.1.0/example'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/usr/src/fuse-3.1.0/example'
make[1]: Leaving directory '/usr/src/fuse-3.1.0/example'
Making install in test
make[1]: Entering directory '/usr/src/fuse-3.1.0/test'
make[2]: Entering directory '/usr/src/fuse-3.1.0/test'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/usr/src/fuse-3.1.0/test'
make[1]: Leaving directory '/usr/src/fuse-3.1.0/test'
Making install in doc
make[1]: Entering directory '/usr/src/fuse-3.1.0/doc'
make[2]: Entering directory '/usr/src/fuse-3.1.0/doc'
make[2]: Nothing to be done for 'install-exec-am'.
 /bin/mkdir -p '/usr/local/share/man/man1'
 /usr/bin/install -c -m 644 fusermount3.1 '/usr/local/share/man/man1'
 /bin/mkdir -p '/usr/local/share/man/man8'
 /usr/bin/install -c -m 644 mount.fuse.8 '/usr/local/share/man/man8'
make[2]: Leaving directory '/usr/src/fuse-3.1.0/doc'
make[1]: Leaving directory '/usr/src/fuse-3.1.0/doc'
make[1]: Entering directory 'https://z5.kerneltalks.com/usr/src/fuse-3.1.0'
make[2]: Entering directory 'https://z5.kerneltalks.com/usr/src/fuse-3.1.0'
make[2]: Nothing to be done for 'install-exec-am'.
 /bin/mkdir -p '/usr/local/lib/pkgconfig'
 /usr/bin/install -c -m 644 fuse3.pc '/usr/local/lib/pkgconfig'
make[2]: Leaving directory 'https://z5.kerneltalks.com/usr/src/fuse-3.1.0'
make[1]: Leaving directory 'https://z5.kerneltalks.com/usr/src/fuse-3.1.0'

# export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
# ldconfig
# modprobe fuse

s3fs installation-

Use the below commands. We are cloning git here so no need to check the latest release from GitHub website.

# git clone https://github.com/s3fs-fuse/s3fs-fuse.git
# cd s3fs-fuse
# ./autogen.sh
# ./configure
# make
# make install

s3fs installation output is below for your reference. Click and expand if you want to refer it.

# git clone https://github.com/s3fs-fuse/s3fs-fuse.git
Cloning into 's3fs-fuse'...
remote: Counting objects: 3552, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 3552 (delta 0), reused 2 (delta 0), pack-reused 3547
Receiving objects: 100% (3552/3552), 1.80 MiB | 1.15 MiB/s, done.
Resolving deltas: 100% (2426/2426), done.
Checking connectivity... done.

# cd s3fs-fuse
# ./autogen.sh
--- Make commit hash file -------
--- Finished commit hash file ---
--- Start autotools -------------
configure.ac:30: installing './compile'
configure.ac:26: installing './config.guess'
configure.ac:26: installing './config.sub'
configure.ac:27: installing './install-sh'
configure.ac:27: installing './missing'
src/Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'
--- Finished autotools ----------

# ./configure
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... gcc3
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking sys/xattr.h usability... yes
checking sys/xattr.h presence... yes
checking for sys/xattr.h... yes
checking attr/xattr.h usability... no
checking attr/xattr.h presence... no
checking for attr/xattr.h... no
checking sys/extattr.h usability... no
checking sys/extattr.h presence... no
checking for sys/extattr.h... no
checking s3fs build with nettle(GnuTLS)... no
checking s3fs build with OpenSSL... no
checking s3fs build with GnuTLS... no
checking s3fs build with NSS... no
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for common_lib_checking... yes
checking compile s3fs with... OpenSSL
checking for DEPS... yes
checking for malloc_trim... yes
checking for library containing clock_gettime... none required
checking for clock_gettime... yes
checking pthread mutex recursive... PTHREAD_MUTEX_RECURSIVE
checking for git... yes
checking for .git... yes
checking github short commit hash... b1fe419
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating test/Makefile
config.status: creating doc/Makefile
config.status: creating config.h
config.status: executing depfiles commands

# make
make  all-recursive
make[1]: Entering directory '/usr/src/s3fs-fuse'
Making all in src
make[2]: Entering directory '/usr/src/s3fs-fuse/src'
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT s3fs.o -MD -MP -MF .deps/s3fs.Tpo -c -o s3fs.o s3fs.cpp
mv -f .deps/s3fs.Tpo .deps/s3fs.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT curl.o -MD -MP -MF .deps/curl.Tpo -c -o curl.o curl.cpp
mv -f .deps/curl.Tpo .deps/curl.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT cache.o -MD -MP -MF .deps/cache.Tpo -c -o cache.o cache.cpp
mv -f .deps/cache.Tpo .deps/cache.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT string_util.o -MD -MP -MF .deps/string_util.Tpo -c -o string_util.o string_util.cpp
mv -f .deps/string_util.Tpo .deps/string_util.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT s3fs_util.o -MD -MP -MF .deps/s3fs_util.Tpo -c -o s3fs_util.o s3fs_util.cpp
mv -f .deps/s3fs_util.Tpo .deps/s3fs_util.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT fdcache.o -MD -MP -MF .deps/fdcache.Tpo -c -o fdcache.o fdcache.cpp
mv -f .deps/fdcache.Tpo .deps/fdcache.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT common_auth.o -MD -MP -MF .deps/common_auth.Tpo -c -o common_auth.o common_auth.cpp
mv -f .deps/common_auth.Tpo .deps/common_auth.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT addhead.o -MD -MP -MF .deps/addhead.Tpo -c -o addhead.o addhead.cpp
mv -f .deps/addhead.Tpo .deps/addhead.Po
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT openssl_auth.o -MD -MP -MF .deps/openssl_auth.Tpo -c -o openssl_auth.o openssl_auth.cpp
mv -f .deps/openssl_auth.Tpo .deps/openssl_auth.Po
g++  -g -O2 -Wall -D_FILE_OFFSET_BITS=64   -o s3fs s3fs.o curl.o cache.o string_util.o s3fs_util.o fdcache.o common_auth.o addhead.o openssl_auth.o   -lfuse -pthread -lcurl -lxml2 -lcrypto
fdcache.o: In function `FdEntity::OpenMirrorFile()':
/usr/src/s3fs-fuse/src/fdcache.cpp:761: warning: the use of `tmpnam' is dangerous, better use `mkstemp'
g++ -DHAVE_CONFIG_H -I. -I..  -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2    -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT test_string_util.o -MD -MP -MF .deps/test_string_util.Tpo -c -o test_string_util.o test_string_util.cpp
mv -f .deps/test_string_util.Tpo .deps/test_string_util.Po
g++  -g -O2 -Wall -D_FILE_OFFSET_BITS=64   -o test_string_util string_util.o test_string_util.o
make[2]: Leaving directory '/usr/src/s3fs-fuse/src'
Making all in test
make[2]: Entering directory '/usr/src/s3fs-fuse/test'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/usr/src/s3fs-fuse/test'
Making all in doc
make[2]: Entering directory '/usr/src/s3fs-fuse/doc'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/usr/src/s3fs-fuse/doc'
make[2]: Entering directory '/usr/src/s3fs-fuse'
make[2]: Nothing to be done for 'all-am'.
make[2]: Leaving directory '/usr/src/s3fs-fuse'
make[1]: Leaving directory '/usr/src/s3fs-fuse'

# make install
Making install in src
make[1]: Entering directory '/usr/src/s3fs-fuse/src'
make[2]: Entering directory '/usr/src/s3fs-fuse/src'
 /bin/mkdir -p '/usr/local/bin'
  /usr/bin/install -c s3fs '/usr/local/bin'
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/usr/src/s3fs-fuse/src'
make[1]: Leaving directory '/usr/src/s3fs-fuse/src'
Making install in test
make[1]: Entering directory '/usr/src/s3fs-fuse/test'
make[2]: Entering directory '/usr/src/s3fs-fuse/test'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/usr/src/s3fs-fuse/test'
make[1]: Leaving directory '/usr/src/s3fs-fuse/test'
Making install in doc
make[1]: Entering directory '/usr/src/s3fs-fuse/doc'
make[2]: Entering directory '/usr/src/s3fs-fuse/doc'
make[2]: Nothing to be done for 'install-exec-am'.
 /bin/mkdir -p '/usr/local/share/man/man1'
 /usr/bin/install -c -m 644 man/s3fs.1 '/usr/local/share/man/man1'
make[2]: Leaving directory '/usr/src/s3fs-fuse/doc'
make[1]: Leaving directory '/usr/src/s3fs-fuse/doc'
make[1]: Entering directory '/usr/src/s3fs-fuse'
make[2]: Entering directory '/usr/src/s3fs-fuse'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/usr/src/s3fs-fuse'
make[1]: Leaving directory '/usr/src/s3fs-fuse'

Configure access keys of your AWS account

Now you need to configure your AWS account security keys in s3fs utility. For that you need to visit your AWS account’s IAM (Identity and Access Management) page and get those keys under ‘Access Keys (Access Key ID and Secret Access Key)‘. If you haven’t created any till the time, then you can create a new key pair and use it.

There are two keys: Access Key ID and Secret Access Key. Add those keys in file /etc/s3fs-keys separated by a colon. You can keep them in any file name and path of your choice. I prefer to keep it in /etc where other OS imp files reside.

# cat /etc/s3fs-keys
AKIAJORAC6V66ML22F6Q:gsR2eul5mpvRYUQu4r15YhuxWKdKZQVeBbGwoOUw
# chmod 600 /etc/s3fs-keys

Remove other permission to read this key file. If you don’t set permission 0 to others then s3fs command utility will warn you about it like below. For extra security you can make this file hidden by adding . in the filename.

s3fs: credentials file /etc/s3fs-keys should not have others permissions.

Now you are ready to mount your bucket.

Mount S3 bucket

Now run s3fs utility with bucket name you want to mount followed by directory on which you want to mount it. -o switch to be used to specify a key file path. If you used another filename and path than /etc/s3fs-keys then use it accordingly. There are many other options that can be supplied with this command to control cache and permission on the Linux server (OS side) which I haven’t mentioned here.

# s3fs kerneltalks /my_s3_bucket/ -o passwd_file=/etc/passwd-s3fs

Here –

  • kerneltalks.bucket is my bucket name in S3
  • /my_s3_bucket is a directory on the server on which I mounted S3 bucket
  • passwd_file is the path where I kept my AWS account keys

Your bucket is mounted on your server! You can check it in df output. You will see the filesystem as s3fs.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M  3.1M   97M   4% /run
/dev/xvda1      7.7G  1.3G  6.5G  17% /
s3fs            256T     0  256T   0% /my_s3_bucket

# ll /my_s3_bucket
total 5
drwxrwxrwx 1 root root 0 Jan 1 1970 ./
drwxr-xr-x 25 root root 4096 Jul 11 06:19 ../
d--------- 1 root root 0 Jul 11 06:33 test/


You can perform all file and directory operations as you do on normal mount point. Observe size it showing – 256T! That’s huge, that’s S3 – almost unlimited storage!

You can add an entry in /etc/fstab to mount your bucket on boot too. Use below entry :

s3fs#bucket_name /mountpoint fuse _netdev,allow_other 0 0

Bonus tip :

If your bucket name includes . then s3fs may fail to mount your bucket. Curl will exist the mount operation as SSL on Amazon won’t be matched as certificate wild card matching fails due to extra . in your bucket name. To understand, if I have bucket name kerneltalks.bucket then SSL wildcard mismatch will happen as indicated by below error –

SSL: certificate subject name (*.s3.amazonaws.com) does not match target host name ‘kerneltalks.bucket.s3.amazonaws.com’

To see errors, warning during mount operation you can run s3fs command with debugging on. Append below switches to the end of your s3fs mount command:

-d -d -f -o f2 -o curldbg

This will help you troubleshoot any issues you faced during the S3 bucket mount.

This is how we mount S3 bucket in Linux servers like Redhat, Ubuntu, Debian, etc. Let us know if you have any feedback/comments/suggestions in below comment box.