https://github.com/BVLC/caffe
http://caffe.berkeleyvision.org/installation.html#prerequisites
http://caffe.berkeleyvision.org/install_yum.html
설치에 앞서, bvlc-caffe의 prerequisite이 존재합니다. 필요한 패키지들을 미리 설치, 버전을 맞춰주어야 합니다. 현재 bvlc-caffe를 빌드하는 시스템 환경은 POWER9 AC922 서버, RHEL 7.2(ppc64le), CUDA-9.1 환경입니다. => 향후 업데이트할 환경의 내용입니다. 우선은 GPU가 없는 Power8 RHEL 7.4 환경에서 CPU-ONLY caffe로 빌드하였습니다.
[root@sys-90994 ~]# yum install git protobuf-devel leveldb-devel snappy-devel opencv-devel boost-devel hdf5-devel gcc gcc-c++ python-devel python-enum34 numpy cmake
[root@sys-90994 usr]# yum install blas-devel blas64-devel atlas-devel
[root@sys-90994 ~]# cd /home/imsi
Prerequisites 중에 하나인 Boost를 빌드해보겠습니다.
* Boost >= 1.55
[root@sys-90994 imsi]# wget https://dl.bintray.com/boostorg/release/1.66.0/source/boost_1_66_0.tar.gz
[root@sys-90994 imsi]# tar -xvzf boost_1_66_0.tar.gz
[root@sys-90994 imsi]# cd boost_1_66_0
[root@sys-90994 boost_1_66_0]# ls
boost boostcpp.jam boost.png bootstrap.sh index.htm INSTALL libs more status
boost-build.jam boost.css bootstrap.bat doc index.html Jamroot LICENSE_1_0.txt rst.css tools
[root@sys-90994 boost_1_66_0]# ./bootstrap.sh --prefix=/usr/local --with-toolset=gcc => prefix를 /usr/local/boost로 설정하면, /usr/local/boost/include, /usr/local/boost/lib 가 각각 생성됩니다. 이를 나중에 옮겨주어야 하므로, 처음부터 /usr/local로 설정하면, /usr/local/include, /usr/local/lib에 boost header, library가 각각 생성되어 들어가게 됩니다.
[root@sys-90994 boost_1_66_0]# ./b2
...updated 60 targets...
The Boost C++ Libraries were successfully built!
The following directory should be added to compiler include paths:
/home/imsi/boost_1_66_0
The following directory should be added to linker library paths:
/home/imsi/boost_1_66_0/stage/lib
Build 과정은 잘 완료되었고, PDP VM 환경에서 약 30분 정도 소요되었습니다. 이어서 Boost 를 설치합니다.
[root@sys-90994 boost_1_66_0]# ./b2 install
...updated 60 targets...
The Boost C++ Libraries were successfully built!
The following directory should be added to compiler include paths:
/home/imsi/boost_1_66_0
The following directory should be added to linker library paths:
/home/imsi/boost_1_66_0/stage/lib
Build 과정은 잘 완료되었고, PDP VM 환경에서 약 30분 정도 소요되었습니다. 이어서 Boost 를 설치합니다.
[root@sys-90994 boost_1_66_0]# ./b2 install
(생략)
ommon.copy /usr/local/boost/lib/libboost_exception.a
common.copy /usr/local/boost/lib/libboost_system.a
common.copy /usr/local/boost/lib/libboost_chrono.a
common.copy /usr/local/boost/lib/libboost_timer.a
common.copy /usr/local/boost/lib/libboost_test_exec_monitor.a
...updated 14124 targets...
위 명령이 완료되면, 이전에 지정한 prefix 경로(/usr/local/boost)에 lib/ , include/를 생성합니다.
Boost 구성이 완료되었으면, 또 다른 prerequisites 중에 하나인 glog 를 설치합니다.
[root@sys-90994 imsi]# wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/google-glog/glog-0.3.3.tar.gz
[root@sys-90994 imsi]# tar zxvf glog-0.3.3.tar.gz
[root@sys-90994 imsi]# cd glog-0.3.3
[root@sys-90994 glog-0.3.3]# rpm --eval %{_host}
powerpc64le-redhat-linux-gnu
[root@sys-90994 glog-0.3.3]# ./configure --build=powerpc64le-redhat-linux-gnu
[root@sys-90994 glog-0.3.3]# make && make install
(생략)
----------------------------------------------------------------------
Libraries have been installed in:
/usr/local/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH' environment variable
during linking
- use the `-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to `/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
test -z "/usr/local/share/doc/glog-0.3.3" || /bin/mkdir -p "/usr/local/share/doc/glog-0.3.3"
/bin/install -c -m 644 AUTHORS COPYING ChangeLog INSTALL NEWS README README.windows doc/designstyle.css doc/glog.html '/usr/local/share/doc/glog-0.3.3'
test -z "/usr/local/include/glog" || /bin/mkdir -p "/usr/local/include/glog"
/bin/install -c -m 644 src/glog/log_severity.h '/usr/local/include/glog'
test -z "/usr/local/include/glog" || /bin/mkdir -p "/usr/local/include/glog"
/bin/install -c -m 644 src/glog/logging.h src/glog/raw_logging.h src/glog/vlog_is_on.h src/glog/stl_logging.h '/usr/local/include/glog'
test -z "/usr/local/lib/pkgconfig" || /bin/mkdir -p "/usr/local/lib/pkgconfig"
/bin/install -c -m 644 libglog.pc '/usr/local/lib/pkgconfig'
make[1]: Leaving directory `/home/imsi/glog-0.3.3'
다음은 gflags 를 설치합니다.
[root@sys-90994 imsi]# wget https://github.com/schuhschuh/gflags/archive/master.zip
[root@sys-90994 imsi]# tar zxvf glog-0.3.3.tar.gz
[root@sys-90994 imsi]# cd glog-0.3.3
[root@sys-90994 glog-0.3.3]# rpm --eval %{_host}
powerpc64le-redhat-linux-gnu
[root@sys-90994 glog-0.3.3]# ./configure --build=powerpc64le-redhat-linux-gnu
[root@sys-90994 glog-0.3.3]# make && make install
(생략)
----------------------------------------------------------------------
Libraries have been installed in:
/usr/local/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH' environment variable
during linking
- use the `-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to `/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
test -z "/usr/local/share/doc/glog-0.3.3" || /bin/mkdir -p "/usr/local/share/doc/glog-0.3.3"
/bin/install -c -m 644 AUTHORS COPYING ChangeLog INSTALL NEWS README README.windows doc/designstyle.css doc/glog.html '/usr/local/share/doc/glog-0.3.3'
test -z "/usr/local/include/glog" || /bin/mkdir -p "/usr/local/include/glog"
/bin/install -c -m 644 src/glog/log_severity.h '/usr/local/include/glog'
test -z "/usr/local/include/glog" || /bin/mkdir -p "/usr/local/include/glog"
/bin/install -c -m 644 src/glog/logging.h src/glog/raw_logging.h src/glog/vlog_is_on.h src/glog/stl_logging.h '/usr/local/include/glog'
test -z "/usr/local/lib/pkgconfig" || /bin/mkdir -p "/usr/local/lib/pkgconfig"
/bin/install -c -m 644 libglog.pc '/usr/local/lib/pkgconfig'
make[1]: Leaving directory `/home/imsi/glog-0.3.3'
다음은 gflags 를 설치합니다.
[root@sys-90994 imsi]# wget https://github.com/schuhschuh/gflags/archive/master.zip
[root@sys-90994 imsi]# unzip master.zip
[root@sys-90994 imsi]# cd gflags-master/
[root@sys-90994 gflags-master]# mkdir build && cd build
[root@sys-90994 build]# export CXXFLAGS="-fPIC" && cmake .. && make VERBOSE=1
[root@sys-90994 build]# make && make install
[ 50%] Built target gflags_nothreads_static
[100%] Built target gflags_static
[ 50%] Built target gflags_nothreads_static
[100%] Built target gflags_static
Install the project...
-- Install configuration: ""
-- Installing: /usr/local/lib/libgflags.a
-- Installing: /usr/local/lib/libgflags_nothreads.a
-- Installing: /usr/local/include/gflags/gflags.h
-- Installing: /usr/local/include/gflags/gflags_declare.h
-- Installing: /usr/local/include/gflags/gflags_completions.h
-- Installing: /usr/local/include/gflags/gflags_gflags.h
-- Installing: /usr/local/lib/cmake/gflags/gflags-config.cmake
-- Installing: /usr/local/lib/cmake/gflags/gflags-config-version.cmake
-- Installing: /usr/local/lib/cmake/gflags/gflags-targets.cmake
-- Installing: /usr/local/lib/cmake/gflags/gflags-targets-noconfig.cmake
-- Installing: /usr/local/bin/gflags_completions.sh
-- Installing: /usr/local/lib/pkgconfig/gflags.pc
-- Installing: /root/.cmake/packages/gflags/e5f7ce61772240490d3164df06f58ce9
gflags 설치도 완료되었습니다. 다음으로 lmdb를 설치합니다.
[root@sys-90994 build]# cd /home/imsi
[root@sys-90994 imsi]# git clone https://github.com/LMDB/lmdb
Cloning into 'lmdb'...
remote: Counting objects: 7846, done.
remote: Total 7846 (delta 0), reused 0 (delta 0), pack-reused 7845
Receiving objects: 100% (7846/7846), 1.76 MiB | 0 bytes/s, done.
Resolving deltas: 100% (3372/3372), done.
[root@sys-90994 imsi]# cd lmdb/libraries/liblmdb
[root@sys-90994 liblmdb]# make && make install
(생략)
for f in mdb_stat mdb_copy mdb_dump mdb_load; do cp $f /usr/local/bin; done
for f in liblmdb.a liblmdb.so; do cp $f /usr/local/lib; done
for f in lmdb.h; do cp $f /usr/local/include; done
for f in mdb_stat.1 mdb_copy.1 mdb_dump.1 mdb_load.1; do cp $f /usr/local/share/man/man1; done
이번엔 hdf5를 설치합니다.
[root@sys-90994 imsi]# wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.10.1.tar.gz
[root@sys-90994 imsi]# tar xvfz hdf5-1.10.1.tar.gz
[root@sys-90994 imsi]# cd hdf5-1.10.1
아래 --prefix는 설치 경로를 지정해줍니다.
[root@sys-90994 hdf5-1.10.1]# ./configure --prefix=/usr/local/hdf5 --enable-fortran --enable-cxx --build=powerpc64le-linux-gnu
수행이 잘 완료되면 make 명령으로 소스를 빌드하는데, 시간이 좀 걸리며 일부 warning이 발생하지만 에러 없이 수행됩니다.
[root@sys-90994 hdf5-1.10.1]# make
[root@sys-90994 hdf5-1.10.1]# make install
필요한 경우, leveldb도 설치해줍니다. leveldb는 rpm 패키지를 다운로드 받아서 설치할 수 있습니다.
[root@sys-90994 imsi]# wget https://rpmfind.net/linux/epel/7/ppc64le/Packages/l/leveldb-1.12.0-11.el7.ppc64le.rpm
[root@sys-90994 imsi]# rpm -Uvh leveldb-1.12.0-11.el7.ppc64le.rpm
[root@sys-90994 imsi]# wget https://www.rpmfind.net/linux/epel/7/ppc64le/Packages/l/leveldb-devel-1.12.0-11.el7.ppc64le.rpm
[root@sys-90994 imsi]# rpm -Uvh leveldb-devel-1.12.0-11.el7.ppc64le.rpm
완료되었습니다. /usr/local/include, /usr/local/lib 경로를 확인해보면, 위에서 설치한 boost, glog, gflags, lmdb, hdf5 관련 헤더와 라이브러리가 생성된 것이 보입니다.
[root@sys-90994 liblmdb]# ls /usr/local/include
boost gflags glog lmdb.h
[root@sys-90994 lib]# ls /usr/local/lib | grep boost | wc -l
107
[root@sys-90994 lib]# ls /usr/local/lib | grep glog | wc -l
5
[root@sys-90994 lib]# ls /usr/local/lib | grep gflags | wc -l
2
[root@sys-90994 lib]# ls /usr/local/lib | grep lmdb | wc -l
2
또다른 prerequisites인 OpenCV >= 2.4 가 설치되어 있는지 확인합니다.
[root@sys-90994 ~]# yum list opencv
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Installed Packages
opencv.ppc64le 2.4.5-3.el7 @optional
필요한 prerequisite 은 어느정도 준비가 되었습니다. 위에서 언급된 것 외에, 기본적으로 GPU 를 사용하기 위해 필요한 CUDA, cuDNN 등은 미리 설치가 되어 있어야 합니다.
bvlc-caffe를 Github 에서 복제합니다. Caffe build는 make 혹은 cmake 명령을 이용하여 가능합니다. 아래 예제에서는 make를 사용했습니다.
[root@sys-90994 imsi]# git clone https://github.com/BVLC/caffe.git
[root@sys-90994 imsi]# cd caffe
[root@sys-90994 caffe]# cp Makefile.config.example Makefile.config
Makefile.config 를 원하는 구성대로 변경합니다. 예를 들어 Build 하려는 Caffe에서 CPU Only로만 동작할 것인지, 특정 DB는 사용하지 않도록 할것인지, Anaconda를 사용할것인지, Python path는 어디서 사용할 것인지 등을 정할 수 있습니다. 현재 환경에서는 Makefile.config에 명시된 numpy 경로가 달라서, 이 부분만 수정하였습니다.
[root@sys-90994 caffe]# vi Makefile.config
# We need to be able to find Python.h and numpy/arrayobject.h.
#PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/lib/python2.7/dist-packages/numpy/core/include
PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/lib64/python2.7/site-packages/numpy/core/include/numpy
# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
BLAS_LIB := /usr/lib64/atlas
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/local/hdf5/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/local/hdf5/lib
[root@sys-90994 caffe]# make all
PROTOC src/caffe/proto/caffe.proto
CXX .build_release/src/caffe/proto/caffe.pb.cc
CXX src/caffe/blob.cpp
In file included from ./include/caffe/common.hpp:19:0,
from ./include/caffe/blob.hpp:8,
from src/caffe/blob.cpp:4:
./include/caffe/util/device_alternate.hpp:34:23: fatal error: cublas_v2.h: No such file or directory
#include <cublas_v2.h>
^
compilation terminated.
make: *** [.build_release/src/caffe/blob.o] Error 1
에러 발생 ==> 문제는 Makefile.config 기본 설정에서는 빌드할때 CPU,GPU 모두 사용하는 caffe로 빌드 시도했으나, 현재 PDP 에는 GPU가 없기 때문입니다. 이 에러는 GPU 있는 시스템에서 빌드해보면 정상적으로 빌드가 완료될 것입니다.
PDP VM에는 GPU가 없으므로, 이번에는 CPU 전용 caffe로 빌드해봅니다.
[root@sys-90994 caffe]# vi Makefile.config
# CPU-only switch (uncomment to build without GPU support).
CPU_ONLY := 1
# CUDA directory contains bin/ and lib/ directories that we need.
#CUDA_DIR := /usr/local/cuda
[root@sys-90994 caffe]# make all
LD -o .build_release/lib/libcaffe.so.1.0.0
/bin/ld: cannot find -lcblas
/bin/ld: cannot find -latlas
collect2: error: ld returned 1 exit status
make: *** [.build_release/lib/libcaffe.so.1.0.0] Error 1
만약 위와 같은 에러가 발생하는 경우, atlas를 설치하지 않았거나, 혹은 Makefile.config 에서 BLAS_LIB, BLAS_INCLUDE 경로를 제대로 지정해주지 않아서 발생합니다. 이를 반영했는데도 동일한 에러가 발생하는 경우는, 새로운 atlas의 라이브러리가 -lcblas, -latlas가 아니라 -lsatlas, -ltatlas 로 제공되어 기존 라이브러리 이름으로는 사용이 불가능할 수도 있습니다. 보통 atlas는 /usr/lib64/atlas 경로에 설치되므로, 해당 경로를 확인하면 어떤 라이브러리 이름으로 명기되어 있는지 보입니다.
[root@sys-90994 caffe]# ls /usr/lib64/atlas
libsatlas.so libsatlas.so.3 libsatlas.so.3.10 libtatlas.so libtatlas.so.3 libtatlas.so.3.10
제 경우에는 라이브러리 이름이 -lsatlas, -ltatlas 로 되어 있습니다. 따라서 Makefile 에서 LIBRARY 명을 정해주는 코드를 변경합니다.
[root@sys-90994 caffe]# vi Makefile
# BLAS configuration (default = ATLAS)
BLAS ?= atlas
ifeq ($(BLAS), mkl)
# MKL
LIBRARIES += mkl_rt
COMMON_FLAGS += -DUSE_MKL
MKLROOT ?= /opt/intel/mkl
BLAS_INCLUDE ?= $(MKLROOT)/include
BLAS_LIB ?= $(MKLROOT)/lib $(MKLROOT)/lib/intel64
else ifeq ($(BLAS), open)
# OpenBLAS
LIBRARIES += openblas
else
# ATLAS
ifeq ($(LINUX), 1)
ifeq ($(BLAS), atlas)
# Linux simply has cblas and atlas
# LIBRARIES += cblas atlas => 이 부분을 uncomment 처리 or 바꿔주면 됩니다.
LIBRARIES += satlas tatlas
[root@sys-90994 caffe]# make all
(생략)
CXX tools/upgrade_solver_proto_text.cpp
CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin
CXX examples/cifar10/convert_cifar_data.cpp
CXX/LD -o .build_release/examples/cifar10/convert_cifar_data.bin
CXX examples/cpp_classification/classification.cpp
CXX/LD -o .build_release/examples/cpp_classification/classification.bin
CXX examples/mnist/convert_mnist_data.cpp
CXX/LD -o .build_release/examples/mnist/convert_mnist_data.bin
CXX examples/siamese/convert_mnist_siamese_data.cpp
CXX/LD -o .build_release/examples/siamese/convert_mnist_siamese_data.bin
완료되었으면, 제대로 빌드 되었는지 테스트를 해봅니다.
[root@sys-90994 caffe]# make test
(생략)
LD .build_release/src/caffe/test/test_threshold_layer.o
LD .build_release/src/caffe/test/test_tile_layer.o
LD .build_release/src/caffe/test/test_upgrade_proto.o
LD .build_release/src/caffe/test/test_util_blas.o
runtest를 수행합니다. 수행에 앞서, LD_LIBRARY_PATH를 잡아줍니다. 위에서 prerequisite으로 설치했던 패키지들의 lib 경로가 모두 포함되어야 합니다. 그렇지 않으면, make runtest를 수행할때 특정 라이브러리를 찾을 수 없으므로, ".build_release/tools/caffe: error while loading shared libraries: libglog.so.0: cannot open shared object file: No such file or directory" 와 같은 에러가 발생합니다. (위 에러는 glog 라이브러리 에러)
[root@sys-90994 caffe]# export LD_LIBRARY_PATH=/usr/local/hdf5/lib:/usr/local/lib:/usr/lib:/usr/lib64
[root@sys-90994 caffe]# make runtest
(생략)
[----------] Global test environment tear-down
[==========] 1110 tests from 152 test cases ran. (85350 ms total)
[ PASSED ] 1110 tests.
위와 같이 나오면 설치가 모두 완료된 것입니다.
이제 mnist 와 같은 간단한 툴로 caffe가 제대로 수행되는지 확인해봅니다.
[root@sys-90994 caffe]# cd data/mnist
[root@sys-90994 mnist]# ./get_mnist.sh
[root@sys-90994 mnist]# cd ../../
create_mnist.sh 을 먼저 수정해줍니다. convert_mnist_data.bin 를 수행해야 하므로, 해당 파일이 있는 경로로 BUILD를 지정해줍니다. (#find . -name convert_mnist_data.bin)
[root@sys-90994 caffe]# vi examples/mnist/create_mnist.sh
#BUILD=build/examples/mnist
BUILD=.build_release/examples/mnist
[root@sys-90994 caffe]# ./examples/mnist/create_mnist.sh
Creating lmdb...
I0117 09:58:32.196631 12668 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_train_lmdb
I0117 09:58:32.197115 12668 convert_mnist_data.cpp:88] A total of 60000 items.
I0117 09:58:32.197132 12668 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I0117 09:58:40.927907 12668 convert_mnist_data.cpp:108] Processed 60000 files.
I0117 09:58:40.954234 12676 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_test_lmdb
I0117 09:58:40.954594 12676 convert_mnist_data.cpp:88] A total of 10000 items.
I0117 09:58:40.954603 12676 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I0117 09:58:45.823518 12676 convert_mnist_data.cpp:108] Processed 10000 files.
Done.
lenet을 수행하기 위한 solver.prototxt 를 수정합니다. 현재는 CPU 전용 caffe로 빌드되어있으므로,
[root@sys-90994 caffe]# vi ./examples/mnist/lenet_solver.prototxt
#solver mode: CPU or GPU
solver_mode: CPU
[root@sys-90994 caffe]# time ./examples/mnist/train_lenet.sh
---
I0117 10:05:26.000897 12939 caffe.cpp:259] Optimization Done.
수행이 정상적으로 종료되는지 확인합니다.
