使用极链/AutoDL云服务器复盘caffe安装

继上一次倒腾caffe安装以后，因为博士毕业等原因，旧的服务器已经不能再使用，最近因论文等原因，不得不继续来安装一下我的caffe。这次运气比较好，经历了一晚上和一早上的痛苦之后，最终安装成功了，而且由于使用极链AI云服务器或者AutoDL来安装的，所以应该这次的可复现性更强，下面总结一些必要的地方，以帮助大家复现安装我的caffe.

首先需要选择极链的云服务器。

- nvidia cuda和gpu的选择
这里我们使用的是极链-华北一区-GeForce RTX 2080 的gpu，cuda 10.1, python=2.7 ，禁用cudnn.

然后开始安装后面的包。

如果是使用AutoDL的云服务器，则可以选择：

环境可以选择Miniconda conda3、Python 3.8 、 Cuda 10.1，禁用cudnn.

1. 首先仍然需要参考《深度学习：21天实战caffe》的第5天那一章来进行安装，最后的安装列表为：

这里可以不管caffe和opencv，只要确保其他的文件如图所示即可。

所有必备的caffe软件【更新版】我们已经上传到百度云盘，请大家需要的可以下载：
链接：https://pan.baidu.com/s/13P9PiTTnjOD6ZiiS4-dCBg
提取码：zlut

2. 在安装各个包的过程中，一些包的安装会有些困难，列举如下：

- Boost （这里一定要强调把filesystem也要加上！）

- glog

初始时刻，我用glog-0.3.3.tar来安装的，总是出错，后来换成了glog-0.3.5.tar安装，就对了。

- opencv
这里我们采用了opencv-3.4.14。可以参考《复盘caffe安装》来安装opencv。

安装过程中，可能会卡在IPPICV: Download: ippicv_2020_lnx_intel64_20191018_general.tgz，这时果断停止编译。从刚才的百度云盘中下载ippicv_2020_lnx_intel64_20191018_general.tgz，然后上传到某一个位置，比如./opencv-3.4.14/3rdparty/ippicv/,

接着修改配置文件./opencv-3.4.14/3rdparty/ippicv/ippicv.cmake,

找到"https://raw.githubusercontent.com/opencv/opencv_3rdparty/${IPPICV_COMMIT}/ippicv/"

将其改为你保存ippicv_2020_lnx_intel64_20191018_general.tgz文件的路径，

我的是./opencv-3.4.14/3rdparty/ippicv/,如下：

最后再来编译。

unzip opencv-2.4.9.zip
cd opencv-2.4.9/
mkdir build;
cd build/
cmake ..  ## 初始时尝试这个命令，如果遇到ippicv的问题，则离线下载并修改路径后，执行下一条命令
cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=YOUR_INSTALL_PATH ..
make && make install

注意这里可以不用ccmake .. 也可以。

-------------------------------------------------------------------------------------------
opencv安装需要花费较多时间。安装完成以后，为了能后面使用opencv.pc，需要在root或者home下面的bashrc中增加如下命令，然后使用 source ~/.bashrc更新。

export PATH="/user-data/caffe_local_install/bin/:$PATH"
export PKG_CONFIG_PATH=/user-data/caffe_local_install/lib/pkgconfig/:$PKG_CONFIG_PATH
export LD_LIBRARY_PATH=/user-data/caffe_local_install/lib:$LD_LIBRARY_PATH 
##(静态库搜索路径) 程序编译期间查找动态链接库时指定查找共享库的路径
export LIBRARY_PATH=/user-data/caffe_local_install/lib:$LIBRARY_PATH ##c程序头文件搜索路径
export C_INCLUDE_PATH=/user-data/caffe_local_install/include:$C_INCLUDE_PATH 
##c++程序头文件搜索路径
export CPLUS_INCLUDE_PATH=/user-data/caffe_local_install/include:$CPLUS_INCLUDE_PATH

这样以后，就可以查看是否安装成功

pkg-config --modversion opencv

显示opencv版本。

- python依赖包的安装

我们需要提前创造一个python2.7的虚拟环境caffe.

由于极链已经预装了anaconda，所以可以直接创造虚拟环境，否则应该先按照anaconda, 再创建虚拟环境。可参考：安装依赖库常用命令 - 极链AI云支持中心

conda create -n your_env_name python=X.X（2.7、3.6等)我的为：
conda create -n caffe python=2.7

接着开始按照如下的python包

pip install opencv-python==4.2.0.32
pip install opencv-python==4.1.1.26  #二选一
pip install protobuf==2.6.1 
pip install scikit-image==0.14.2

3. 配置Makefile.config文件，

cp Makefile.config.example Makefile.config

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!# cuDNN acceleration switch (uncomment to build with cuDNN).USE_CUDNN := 0# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#	You should not set this flag if you will be reading LMDBs with any
#	possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1# Uncomment if you're using OpenCV 3OPENCV_VERSION := 3# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
CUDA_ARCH :=-gencode arch=compute_30,code=sm_30 \-gencode arch=compute_35,code=sm_35 \-gencode arch=compute_50,code=sm_50 \-gencode arch=compute_52,code=sm_52 \-gencode arch=compute_60,code=sm_60 \-gencode arch=compute_61,code=sm_61 \-gencode arch=compute_61,code=compute_61# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := open
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
BLAS_INCLUDE := /user-data/caffe_local_install/include
BLAS_LIB := /user-data/caffe_local_install/lib# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR :=  /data1/caiyong.wang/bin/matlab/ 
# MATLAB_DIR := /Applications/MATLAB_R2012b.app# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
#		/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.ANACONDA_HOME := /opt/conda/envs/caffePYTHON_INCLUDE := $(ANACONDA_HOME)/include \$(ANACONDA_HOME)/include/python2.7 \$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include# We need to be able to find libpythonX.X.so or .dylib.
# PYTHON_LIB := /usr/libPYTHON_LIB := $(ANACONDA_HOME)/lib# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib# Uncomment to support layers written in Python (will link against Python libs)WITH_PYTHON_LAYER := 1# Whatever else you find you need goes here.
INCLUDE_DIRS := /user-data/caffe_local_install/include  $(PYTHON_INCLUDE) /usr/local/include /usr/include
LIBRARY_DIRS := /user-data/caffe_local_install/lib  $(PYTHON_LIB) /usr/local/lib /usr/lib  /usr/lib64/ # If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
USE_PKG_CONFIG := 1# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0# enable pretty build (comment to see full commands)
Q ?= @

编译caffe

# Adjust Makefile.config (for example, if using Anaconda Python, or if cuDNN is desired)
make all -j8
make pycaffe

编译到最后的结果：

4. 编辑root或者home下面的bashrc,

export PYTHONPATH=/user-data/software/caffe_tifs/python/:$PYTHONPATH

确保编译成功的caffe被添加到python的路径中，最终新增的命令汇总为：

export PATH="/user-data/caffe_local_install/bin/:$PATH"
export PKG_CONFIG_PATH=/user-data/caffe_local_install/lib/pkgconfig/:$PKG_CONFIG_PATH
export PYTHONPATH=/user-data/software/caffe_tifs/python/:$PYTHONPATH
export LD_LIBRARY_PATH=/user-data/caffe_local_install/lib:$LD_LIBRARY_PATH 
##(静态库搜索路径) 程序编译期间查找动态链接库时指定查找共享库的路径
export LIBRARY_PATH=/user-data/caffe_local_install/lib:$LIBRARY_PATH ##c程序头文件搜索路径
export C_INCLUDE_PATH=/user-data/caffe_local_install/include:$C_INCLUDE_PATH 
##c++程序头文件搜索路径
export CPLUS_INCLUDE_PATH=/user-data/caffe_local_install/include:$CPLUS_INCLUDE_PATH

使用 source ~/.bashrc更新。

5. 验证caffe是否安装成功