Learn to mount S3 bucket in RHEL, ubuntu, CentOS Linux server. Understand how to debug issues while mounting S3 bucket.
In this article we will walk through process by which you can mount S3 bucket on Linux server. S3 bucket is storage container of S3 (simple storage service) AWS service. As all traditional datacenters are moving to cloud computing, it is necessary to know how to interconnect cloud and traditional services. Lets dive into process to mount S3 bucket in RHEL, ubuntu, CentOS Linux. Complete process can be done in below 3 steps:
- Install fuse and s3fs packages
- Configure access keys of your AWS account
- Mount S3 bucket
For this tutorial, we are assuming you have S3 bucket ready in your AWS account with proper permission setup. If not, follow this tutorial to create S3 bucket in AWS.
Install fuse and s3fs packages
These packages can be found here : fuse & s3fs. You have to download them on Linux server with tools like wget and compile them. Make sure you have their dependencies installed before you try to compile them.
Dependencies are :
for RedHat based : automake gcc gcc-c++ git libcurl-devel libxml2-devel make openssl-devel mailcap curl-devel libstdc++-devel
for Debian based : automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
Install all these packages and follow below steps to configure fuse and s3fs. Make sure there is no package named fuse of s3fs exist in server before you proceed. This is to avoid conflicts in installation.
Fuse installation –
Use below commands. Use latest Fuse download link used in wget command. You can obtain it from Github here.
1 2 3 4 5 6 7 8 9 10 11 | # cd /usr/src/ # wget https://github.com/libfuse/libfuse/releases/download/fuse-3.1.0/fuse-3.1.0.tar.gz # tar -zxf fuse-3.1.0.tar.gz # cd fuse-3.1.0 # ./configure -prefix=/usr/local # make && make install # export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig # ldconfig # modprobe fuse |
Fuse installation output is below for your reference. Click and expand if you want to view it.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 | # cd /usr/src/ # wget https://github.com/libfuse/libfuse/releases/download/fuse-3.1.0/fuse-3.1.0.tar.gz --2017-07-11 05:19:36-- https://github.com/libfuse/libfuse/releases/download/fuse-3.1.0/fuse-3.1.0.tar.gz Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113 Connecting to github.com (github.com)|192.30.253.112|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/48296177/00b4f6e4-63dc-11e7-938f-c32c894af199?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20170711%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170711T051937Z&X-Amz-Expires=300&X-Amz-Signature=aeab5db1ef80c13276b06d3e481e8d394cf08b41da6bdd0034c287c38b3c0413&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dfuse-3.1.0.tar.gz&response-content-type=application%2Foctet-stream [following] --2017-07-11 05:19:37-- https://github-production-release-asset-2e65be.s3.amazonaws.com/48296177/00b4f6e4-63dc-11e7-938f-c32c894af199?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20170711%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170711T051937Z&X-Amz-Expires=300&X-Amz-Signature=aeab5db1ef80c13276b06d3e481e8d394cf08b41da6bdd0034c287c38b3c0413&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dfuse-3.1.0.tar.gz&response-content-type=application%2Foctet-stream Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.230.243 Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.230.243|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 809470 (790K) [application/octet-stream] Saving to: ‘fuse-3.1.0.tar.gz’ fuse-3.1.0.tar.gz 100%[============================================================================>] 790.50K 479KB/s in 1.7s 2017-07-11 05:19:40 (479 KB/s) - ‘fuse-3.1.0.tar.gz’ saved [809470/809470] # tar -zxf fuse-3.1.0.tar.gz # cd fuse-3.1.0 # ./configure -prefix=/usr/local checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-pc-linux-gnu checking target system type... x86_64-pc-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether make supports nested variables... (cached) yes checking how to print strings... printf checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking dependency style of gcc... gcc3 checking for a sed that does not truncate output... /bin/sed checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for fgrep... /bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1572864 checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop checking for /usr/bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for ar... ar checking for archiver @FILE support... @ checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm -B output from gcc object... ok checking for sysroot... no checking for a working dd... /bin/dd checking how to truncate binary pipes... /bin/dd bs=4096 count=1 checking for mt... mt checking if mt is a manifest tool... no checking how to run the C preprocessor... gcc -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... yes checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking for gcc option to accept ISO C99... none needed checking for gcc option to accept ISO Standard C... (cached) none needed checking for special C compiler options needed for large files... no checking for _FILE_OFFSET_BITS value needed for large files... no checking for fork... yes checking for setxattr... yes checking for fdatasync... yes checking for splice... yes checking for vmsplice... yes checking for utimensat... yes checking for pipe2... yes checking for posix_fallocate... yes checking for fstatat... yes checking for openat... yes checking for readlinkat... yes checking for struct stat.st_atim... yes checking for struct stat.st_atimespec... no checking for library containing dlopen... -ldl checking for library containing clock_gettime... none required checking for ulockmgr_op in -lulockmgr... no checking for ld used by gcc... /usr/bin/ld -m elf_x86_64 checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes checking for shared library run path origin... done checking for iconv... yes checking for working iconv... yes checking for iconv declaration... extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft); configure: MOUNT_FUSE_PATH env var not set, using default ${sbindir} configure: UDEV_RULES_PATH env var not set, using default ${libdir}/udev/rules.d configure: INIT_D_PATH env var not set, using default ${sysconfdir}/init.d checking if umount supports --fake --no-canonicalize... yes checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating fuse3.pc config.status: creating Makefile config.status: creating lib/Makefile config.status: creating util/Makefile config.status: creating example/Makefile config.status: creating include/Makefile config.status: creating doc/Makefile config.status: creating test/Makefile config.status: creating include/config.h config.status: executing depfiles commands config.status: executing libtool commands # make && make install Making all in include make[1]: Entering directory '/usr/src/fuse-3.1.0/include' make all-am make[2]: Entering directory '/usr/src/fuse-3.1.0/include' make[2]: Nothing to be done for 'all-am'. make[2]: Leaving directory '/usr/src/fuse-3.1.0/include' make[1]: Leaving directory '/usr/src/fuse-3.1.0/include' Making all in lib make[1]: Entering directory '/usr/src/fuse-3.1.0/lib' CC fuse.lo CC fuse_loop.lo CC fuse_loop_mt.lo CC fuse_lowlevel.lo CC fuse_opt.lo CC fuse_signals.lo CC buffer.lo CC cuse_lowlevel.lo CC helper.lo helper.c: In function ‘fuse_daemonize’: helper.c:226:4: warning: ignoring return value of ‘read’, declared with attribute warn_unused_result [-Wunused-result] (void) read(waiter[0], &completed, sizeof(completed)); ^ helper.c:235:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result] (void) chdir("/"); ^ helper.c:248:3: warning: ignoring return value of ‘write’, declared with attribute warn_unused_result [-Wunused-result] (void) write(waiter[1], &completed, sizeof(completed)); ^ helper.c:252:3: warning: ignoring return value of ‘chdir’, declared with attribute warn_unused_result [-Wunused-result] (void) chdir("/"); ^ CC modules/subdir.lo CC modules/iconv.lo CC mount.lo CC mount_util.lo mount_util.c: In function ‘mtab_needs_update’: mount_util.c:68:4: warning: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Wunused-result] setreuid(0, -1); ^ mount_util.c:73:4: warning: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Wunused-result] setreuid(ruid, -1); ^ CCLD libfuse3.la ar: `u' modifier ignored since `D' is the default (see `U') make[1]: Leaving directory '/usr/src/fuse-3.1.0/lib' Making all in util make[1]: Entering directory '/usr/src/fuse-3.1.0/util' make all-am make[2]: Entering directory '/usr/src/fuse-3.1.0/util' CC fusermount3-fusermount.o CC fusermount3-mount_util.o mount_util.c: In function ‘mtab_needs_update’: mount_util.c:68:4: warning: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Wunused-result] setreuid(0, -1); ^ mount_util.c:73:4: warning: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Wunused-result] setreuid(ruid, -1); ^ CCLD fusermount3 CC mount.fuse.o CCLD mount.fuse3 make[2]: Leaving directory '/usr/src/fuse-3.1.0/util' make[1]: Leaving directory '/usr/src/fuse-3.1.0/util' Making all in example make[1]: Entering directory '/usr/src/fuse-3.1.0/example' CC passthrough.o CCLD passthrough CC passthrough_fh.o CCLD passthrough_fh CC null.o CCLD null CC hello.o CCLD hello CC hello_ll.o CCLD hello_ll CC ioctl.o CCLD ioctl CC ioctl_client-ioctl_client.o CCLD ioctl_client CC poll.o CCLD poll CC poll_client-poll_client.o CCLD poll_client CC passthrough_ll.o CCLD passthrough_ll CC notify_inval_inode.o CCLD notify_inval_inode CC notify_store_retrieve.o CCLD notify_store_retrieve CC notify_inval_entry.o CCLD notify_inval_entry CC cuse.o CCLD cuse CC cuse_client.o CCLD cuse_client make[1]: Leaving directory '/usr/src/fuse-3.1.0/example' Making all in test make[1]: Entering directory '/usr/src/fuse-3.1.0/test' CC test_syscalls.o CCLD test_syscalls CC test_write_cache.o CCLD test_write_cache CC test_setattr.o CCLD test_setattr make[1]: Leaving directory '/usr/src/fuse-3.1.0/test' Making all in doc make[1]: Entering directory '/usr/src/fuse-3.1.0/doc' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/usr/src/fuse-3.1.0/doc' make[1]: Entering directory '/usr/src/fuse-3.1.0' make[1]: Nothing to be done for 'all-am'. make[1]: Leaving directory '/usr/src/fuse-3.1.0' Making install in include make[1]: Entering directory '/usr/src/fuse-3.1.0/include' make[2]: Entering directory '/usr/src/fuse-3.1.0/include' make[2]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/local/include/fuse3' /usr/bin/install -c -m 644 fuse.h fuse_common.h fuse_lowlevel.h fuse_opt.h cuse_lowlevel.h '/usr/local/include/fuse3' make[2]: Leaving directory '/usr/src/fuse-3.1.0/include' make[1]: Leaving directory '/usr/src/fuse-3.1.0/include' Making install in lib make[1]: Entering directory '/usr/src/fuse-3.1.0/lib' make[2]: Entering directory '/usr/src/fuse-3.1.0/lib' /bin/mkdir -p '/usr/local/lib' /bin/bash ../libtool --mode=install /usr/bin/install -c libfuse3.la '/usr/local/lib' libtool: install: /usr/bin/install -c .libs/libfuse3.so.3.1.0 /usr/local/lib/libfuse3.so.3.1.0 libtool: install: (cd /usr/local/lib && { ln -s -f libfuse3.so.3.1.0 libfuse3.so.3 || { rm -f libfuse3.so.3 && ln -s libfuse3.so.3.1.0 libfuse3.so.3; }; }) libtool: install: (cd /usr/local/lib && { ln -s -f libfuse3.so.3.1.0 libfuse3.so || { rm -f libfuse3.so && ln -s libfuse3.so.3.1.0 libfuse3.so; }; }) libtool: install: /usr/bin/install -c .libs/libfuse3.lai /usr/local/lib/libfuse3.la libtool: install: /usr/bin/install -c .libs/libfuse3.a /usr/local/lib/libfuse3.a libtool: install: chmod 644 /usr/local/lib/libfuse3.a libtool: install: ranlib /usr/local/lib/libfuse3.a libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/sbin" ldconfig -n /usr/local/lib ---------------------------------------------------------------------- Libraries have been installed in: /usr/local/lib If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the '-LLIBDIR' flag during linking and do at least one of the following: - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable during execution - add LIBDIR to the 'LD_RUN_PATH' environment variable during linking - use the '-Wl,-rpath -Wl,LIBDIR' linker flag - have your system administrator add LIBDIR to '/etc/ld.so.conf' See any operating system documentation about shared libraries for more information, such as the ld(1) and ld.so(8) manual pages. ---------------------------------------------------------------------- make[2]: Nothing to be done for 'install-data-am'. make[2]: Leaving directory '/usr/src/fuse-3.1.0/lib' make[1]: Leaving directory '/usr/src/fuse-3.1.0/lib' Making install in util make[1]: Entering directory '/usr/src/fuse-3.1.0/util' make install-am make[2]: Entering directory '/usr/src/fuse-3.1.0/util' make[3]: Entering directory '/usr/src/fuse-3.1.0/util' /bin/mkdir -p '/usr/local/bin' /bin/bash ../libtool --mode=install /usr/bin/install -c fusermount3 '/usr/local/bin' libtool: install: /usr/bin/install -c fusermount3 /usr/local/bin/fusermount3 /bin/mkdir -p /usr/local/sbin /usr/bin/install -c ./mount.fuse3 /usr/local/sbin/mount.fuse3 /bin/mkdir -p /usr/local/etc/init.d /usr/bin/install -c ./init_script /usr/local/etc/init.d/fuse3 /usr/sbin/update-rc.d fuse start 34 S . start 41 0 6 . || true update-rc.d: error: unable to read /etc/init.d/fuse make install-exec-hook make[4]: Entering directory '/usr/src/fuse-3.1.0/util' chmod u+s /usr/local/bin/fusermount3 make[4]: Leaving directory '/usr/src/fuse-3.1.0/util' /bin/mkdir -p /usr/local/lib/udev/rules.d /usr/bin/install -c -m 644 ./udev.rules /usr/local/lib/udev/rules.d/99-fuse3.rules make[3]: Leaving directory '/usr/src/fuse-3.1.0/util' make[2]: Leaving directory '/usr/src/fuse-3.1.0/util' make[1]: Leaving directory '/usr/src/fuse-3.1.0/util' Making install in example make[1]: Entering directory '/usr/src/fuse-3.1.0/example' make[2]: Entering directory '/usr/src/fuse-3.1.0/example' make[2]: Nothing to be done for 'install-exec-am'. make[2]: Nothing to be done for 'install-data-am'. make[2]: Leaving directory '/usr/src/fuse-3.1.0/example' make[1]: Leaving directory '/usr/src/fuse-3.1.0/example' Making install in test make[1]: Entering directory '/usr/src/fuse-3.1.0/test' make[2]: Entering directory '/usr/src/fuse-3.1.0/test' make[2]: Nothing to be done for 'install-exec-am'. make[2]: Nothing to be done for 'install-data-am'. make[2]: Leaving directory '/usr/src/fuse-3.1.0/test' make[1]: Leaving directory '/usr/src/fuse-3.1.0/test' Making install in doc make[1]: Entering directory '/usr/src/fuse-3.1.0/doc' make[2]: Entering directory '/usr/src/fuse-3.1.0/doc' make[2]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/local/share/man/man1' /usr/bin/install -c -m 644 fusermount3.1 '/usr/local/share/man/man1' /bin/mkdir -p '/usr/local/share/man/man8' /usr/bin/install -c -m 644 mount.fuse.8 '/usr/local/share/man/man8' make[2]: Leaving directory '/usr/src/fuse-3.1.0/doc' make[1]: Leaving directory '/usr/src/fuse-3.1.0/doc' make[1]: Entering directory '/usr/src/fuse-3.1.0' make[2]: Entering directory '/usr/src/fuse-3.1.0' make[2]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/local/lib/pkgconfig' /usr/bin/install -c -m 644 fuse3.pc '/usr/local/lib/pkgconfig' make[2]: Leaving directory '/usr/src/fuse-3.1.0' make[1]: Leaving directory '/usr/src/fuse-3.1.0' # export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig # ldconfig # modprobe fuse |
s3fs installation-
Use below commands. We are cloning git here so no need to check latest release from GitHub website.
1 2 3 4 5 6 7 8 | # git clone https://github.com/s3fs-fuse/s3fs-fuse.git # cd s3fs-fuse # ./autogen.sh # ./configure # make # make install |
s3fs installation output is below for your reference. Click and expand if you want to view it.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 | # git clone https://github.com/s3fs-fuse/s3fs-fuse.git Cloning into 's3fs-fuse'... remote: Counting objects: 3552, done. remote: Compressing objects: 100% (5/5), done. remote: Total 3552 (delta 0), reused 2 (delta 0), pack-reused 3547 Receiving objects: 100% (3552/3552), 1.80 MiB | 1.15 MiB/s, done. Resolving deltas: 100% (2426/2426), done. Checking connectivity... done. # cd s3fs-fuse # ./autogen.sh --- Make commit hash file ------- --- Finished commit hash file --- --- Start autotools ------------- configure.ac:30: installing './compile' configure.ac:26: installing './config.guess' configure.ac:26: installing './config.sub' configure.ac:27: installing './install-sh' configure.ac:27: installing './missing' src/Makefile.am: installing './depcomp' parallel-tests: installing './test-driver' --- Finished autotools ---------- # ./configure checking build system type... x86_64-pc-linux-gnu checking host system type... x86_64-pc-linux-gnu checking target system type... x86_64-pc-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking for g++... g++ checking whether the C++ compiler works... yes checking for C++ compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C++ compiler... yes checking whether g++ accepts -g... yes checking for style of include used by make... GNU checking dependency style of g++... gcc3 checking for gcc... gcc checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking dependency style of gcc... gcc3 checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /bin/grep checking for egrep... /bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking sys/xattr.h usability... yes checking sys/xattr.h presence... yes checking for sys/xattr.h... yes checking attr/xattr.h usability... no checking attr/xattr.h presence... no checking for attr/xattr.h... no checking sys/extattr.h usability... no checking sys/extattr.h presence... no checking for sys/extattr.h... no checking s3fs build with nettle(GnuTLS)... no checking s3fs build with OpenSSL... no checking s3fs build with GnuTLS... no checking s3fs build with NSS... no checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for common_lib_checking... yes checking compile s3fs with... OpenSSL checking for DEPS... yes checking for malloc_trim... yes checking for library containing clock_gettime... none required checking for clock_gettime... yes checking pthread mutex recursive... PTHREAD_MUTEX_RECURSIVE checking for git... yes checking for .git... yes checking github short commit hash... b1fe419 checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating test/Makefile config.status: creating doc/Makefile config.status: creating config.h config.status: executing depfiles commands # make make all-recursive make[1]: Entering directory '/usr/src/s3fs-fuse' Making all in src make[2]: Entering directory '/usr/src/s3fs-fuse/src' g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT s3fs.o -MD -MP -MF .deps/s3fs.Tpo -c -o s3fs.o s3fs.cpp mv -f .deps/s3fs.Tpo .deps/s3fs.Po g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT curl.o -MD -MP -MF .deps/curl.Tpo -c -o curl.o curl.cpp mv -f .deps/curl.Tpo .deps/curl.Po g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT cache.o -MD -MP -MF .deps/cache.Tpo -c -o cache.o cache.cpp mv -f .deps/cache.Tpo .deps/cache.Po g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT string_util.o -MD -MP -MF .deps/string_util.Tpo -c -o string_util.o string_util.cpp mv -f .deps/string_util.Tpo .deps/string_util.Po g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT s3fs_util.o -MD -MP -MF .deps/s3fs_util.Tpo -c -o s3fs_util.o s3fs_util.cpp mv -f .deps/s3fs_util.Tpo .deps/s3fs_util.Po g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT fdcache.o -MD -MP -MF .deps/fdcache.Tpo -c -o fdcache.o fdcache.cpp mv -f .deps/fdcache.Tpo .deps/fdcache.Po g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT common_auth.o -MD -MP -MF .deps/common_auth.Tpo -c -o common_auth.o common_auth.cpp mv -f .deps/common_auth.Tpo .deps/common_auth.Po g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT addhead.o -MD -MP -MF .deps/addhead.Tpo -c -o addhead.o addhead.cpp mv -f .deps/addhead.Tpo .deps/addhead.Po g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT openssl_auth.o -MD -MP -MF .deps/openssl_auth.Tpo -c -o openssl_auth.o openssl_auth.cpp mv -f .deps/openssl_auth.Tpo .deps/openssl_auth.Po g++ -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -o s3fs s3fs.o curl.o cache.o string_util.o s3fs_util.o fdcache.o common_auth.o addhead.o openssl_auth.o -lfuse -pthread -lcurl -lxml2 -lcrypto fdcache.o: In function `FdEntity::OpenMirrorFile()': /usr/src/s3fs-fuse/src/fdcache.cpp:761: warning: the use of `tmpnam' is dangerous, better use `mkstemp' g++ -DHAVE_CONFIG_H -I. -I.. -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -I/usr/include/libxml2 -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -MT test_string_util.o -MD -MP -MF .deps/test_string_util.Tpo -c -o test_string_util.o test_string_util.cpp mv -f .deps/test_string_util.Tpo .deps/test_string_util.Po g++ -g -O2 -Wall -D_FILE_OFFSET_BITS=64 -o test_string_util string_util.o test_string_util.o make[2]: Leaving directory '/usr/src/s3fs-fuse/src' Making all in test make[2]: Entering directory '/usr/src/s3fs-fuse/test' make[2]: Nothing to be done for 'all'. make[2]: Leaving directory '/usr/src/s3fs-fuse/test' Making all in doc make[2]: Entering directory '/usr/src/s3fs-fuse/doc' make[2]: Nothing to be done for 'all'. make[2]: Leaving directory '/usr/src/s3fs-fuse/doc' make[2]: Entering directory '/usr/src/s3fs-fuse' make[2]: Nothing to be done for 'all-am'. make[2]: Leaving directory '/usr/src/s3fs-fuse' make[1]: Leaving directory '/usr/src/s3fs-fuse' # make install Making install in src make[1]: Entering directory '/usr/src/s3fs-fuse/src' make[2]: Entering directory '/usr/src/s3fs-fuse/src' /bin/mkdir -p '/usr/local/bin' /usr/bin/install -c s3fs '/usr/local/bin' make[2]: Nothing to be done for 'install-data-am'. make[2]: Leaving directory '/usr/src/s3fs-fuse/src' make[1]: Leaving directory '/usr/src/s3fs-fuse/src' Making install in test make[1]: Entering directory '/usr/src/s3fs-fuse/test' make[2]: Entering directory '/usr/src/s3fs-fuse/test' make[2]: Nothing to be done for 'install-exec-am'. make[2]: Nothing to be done for 'install-data-am'. make[2]: Leaving directory '/usr/src/s3fs-fuse/test' make[1]: Leaving directory '/usr/src/s3fs-fuse/test' Making install in doc make[1]: Entering directory '/usr/src/s3fs-fuse/doc' make[2]: Entering directory '/usr/src/s3fs-fuse/doc' make[2]: Nothing to be done for 'install-exec-am'. /bin/mkdir -p '/usr/local/share/man/man1' /usr/bin/install -c -m 644 man/s3fs.1 '/usr/local/share/man/man1' make[2]: Leaving directory '/usr/src/s3fs-fuse/doc' make[1]: Leaving directory '/usr/src/s3fs-fuse/doc' make[1]: Entering directory '/usr/src/s3fs-fuse' make[2]: Entering directory '/usr/src/s3fs-fuse' make[2]: Nothing to be done for 'install-exec-am'. make[2]: Nothing to be done for 'install-data-am'. make[2]: Leaving directory '/usr/src/s3fs-fuse' make[1]: Leaving directory '/usr/src/s3fs-fuse' |
Configure access keys of your AWS account
Now you need to configure your AWS account security keys in s3fs utility. For that you need to visit your AWS account’s IAM (Identity and Access Management) page and get those keys under ‘Access Keys (Access Key ID and Secret Access Key)‘. If you havnt created any till time, then you can create new key pair and use it.
There are two keys : Access Key ID and Secret Access Key. Add those keys in file /etc/s3fs-keys separated by colon. You can keep them in any file name and path of your choice. I prefer to keep it in /etc where other OS imp files resides.
1 2 3 4 5 | # cat /etc/s3fs-keys AKIAJORAC6V66ML22F6Q:gsR2eul5mpvRYUQu4r15YhuxWKdKZQVeBbGwoOUw # chmod 600 /etc/s3fs-keys |
Remove others permission to read this key file. If you dont set permission 0 to others then s3fs command utility will warn you about it like below . For extra security you can make this file hidden by adding . in filename.
1 2 3 | s3fs: credentials file /etc/s3fs-keys should not have others permissions. |
Now you are ready to mount your bucket.
Mount S3 bucket
Now run s3fs utility with bucket name you want to mount followed by directory on which you want to mount it. -o switch to be used to specify key file path. If you used another filename and path than /etc/s3fs-keys then use it accordingly. There are many other options which can be supplied with this command to control cache and permission on Linux server (OS side) which I havnt mentioned here.
1 2 3 | # s3fs kerneltalks /my_s3_bucket/ -o passwd_file=/etc/passwd-s3fs |
Here –
- kerneltalks.bucket is my bucket name in S3
- /my_s3_bucket is directory on server on which I mounted S3 bucket
- passwd_file is path where I kept my AWS account keys
Your bucket is mounted on your server! You can check it in df output. You will see filesystem as s3fs.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # df -h Filesystem Size Used Avail Use% Mounted on udev 488M 0 488M 0% /dev tmpfs 100M 3.1M 97M 4% /run /dev/xvda1 7.7G 1.3G 6.5G 17% / s3fs 256T 0 256T 0% /my_s3_bucket # ll /my_s3_bucket total 5 drwxrwxrwx 1 root root 0 Jan 1 1970 ./ drwxr-xr-x 25 root root 4096 Jul 11 06:19 ../ d--------- 1 root root 0 Jul 11 06:33 test/ |
You can perform all file and directory operations like you do on normal mount point. Observe size it showing – 256T ! Thats huge, thats S3 – almost unlimited storage!
You can add entry in /etc/fstab to mount your bucket on boot too. Use below entry :
1 2 3 | s3fs#bucket_name /mountpoint fuse _netdev,allow_other 0 0 |
Bonus tip :
If your bucket name includes . then s3fs may fail to mount your bucket. Curl will exist the mount operation as SSL on Amazon wont be matched as certificate wild card matching fails due to extra . in your bucket name. To understand, if I have bucket name kerneltalks.bucket then SSL wildcard mismatch will happen as indicated by below error –
SSL: certificate subject name (*.s3.amazonaws.com) does not match target host name ‘kerneltalks.bucket.s3.amazonaws.com’
To see errors, warning during mount operation you can run s3fs command with debug on. Append below switches to the end of your s3fs mount command:
1 2 3 | -d -d -f -o f2 -o curldbg |
This will help you troubleshooting any issues you faced during S3 bucket mount.
This is how we mount S3 bucket in Linux servers like Redhat, ubuntu, Debian etc. Let us know if you have any feedback/comments/suggestions in below comment box.
Got any feedback? Comment here!