ubuntu18.04运行DynaSLAM，实例分割+多视图几何+背景修复

博主电脑上之前就安装了ORB-SLAM2所需的环境，所以baseline为ORB-SLAM2的算法几乎都是直接./build.sh即可。
需要我们安装的内容就是python的环境，Pytorch或者TensorFlow，以及下载预训练模型和数据集。

一、安装ORB-SLAM2所需的库

C++11 or C++0x Compiler, Pangolin, OpenCV and Eigen3
Install boost libraries with the command sudo apt-get install libboost-all-dev.

DynaSLAM supports now both OpenCV 2.X and OpenCV 3.X.

二、安装Anaconda，创建python2.7的环境

Install python 2.7, keras2.0.9 and tensorflow1.14.0, and download the mask_rcnn_coco.h5 model from this GitHub repository: https://github.com/matterport/Mask_RCNN/releases.

在这里插入图片描述

三、克隆源代码

Clone this repo:

git clone https://github.com/BertaBescos/DynaSLAM.git
cd DynaSLAM

Place the mask_rcnn_coco.h5 model in the folder DynaSLAM/src/python/.
activate your python environment and execute this script python src/python/Check.py to evalutate its integrity.
sudo pip install scikit-image安装scikit-image
git clone https://github.com/waleedka/coco，cd coco/PythonAPI，sudo make install安装pycocotools

在这里插入图片描述

运行完指令之后就把pycocotools文件夹整个复制到src/python/下。输入python src/python/Check.py验证MaskRCNN是否能正常工作。

在这里插入图片描述

总之，在编译mask-rcnn所需的库文件这里出现了很多报错，锻炼分析问题-解决问题的能力吧。

在这里插入图片描述

四、编译源代码

cd DynaSLAM
chmod +x build.sh
./build.sh

在这里插入图片描述

五、成功运行，完结散花

修改TUM3.yaml文件，ORB特征点数量为3000，否则初始化会失败。

./Examples/RGB-D/rgbd_tum ./Vocabulary/ORBvoc.txt ./Examples/RGB-D/TUM3.yaml ../TUM_DataSet/freiburg3_walking_xyz/ ../TUM_DataSet/freiburg3_walking_xyz/associations.txt ./mask ./output

运行的速度很慢，因为是在CPU上对每帧先进行实例分割再提取特征点和匹配，并且还会根据前后的关键帧修复背景。

在这里插入图片描述

六、过程记录

(tf114py27) dzh@dzh-Lenovo-Legion-Y7000:~/slambook/DynaSLAM/coco$ cd PythonAPI/
(tf114py27) dzh@dzh-Lenovo-Legion-Y7000:~/slambook/DynaSLAM/coco/PythonAPI$ sudo make install
[sudo] dzh 的密码： 
# install pycocotools to the Python site-packages
python setup.py build_ext install
running build_ext
running install
running build
running build_py
running install_lib
creating /usr/local/lib/python2.7/dist-packages/pycocotools
copying build/lib.linux-x86_64-2.7/pycocotools/_mask.so -> /usr/local/lib/python2.7/dist-packages/pycocotools
copying build/lib.linux-x86_64-2.7/pycocotools/cocoeval.py -> /usr/local/lib/python2.7/dist-packages/pycocotools
copying build/lib.linux-x86_64-2.7/pycocotools/coco.py -> /usr/local/lib/python2.7/dist-packages/pycocotools
copying build/lib.linux-x86_64-2.7/pycocotools/__init__.py -> /usr/local/lib/python2.7/dist-packages/pycocotools
copying build/lib.linux-x86_64-2.7/pycocotools/mask.py -> /usr/local/lib/python2.7/dist-packages/pycocotools
byte-compiling /usr/local/lib/python2.7/dist-packages/pycocotools/cocoeval.py to cocoeval.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/pycocotools/coco.py to coco.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/pycocotools/__init__.py to __init__.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/pycocotools/mask.py to mask.pyc
running install_egg_info
Writing /usr/local/lib/python2.7/dist-packages/pycocotools-2.0.egg-info

(tf114py27) dzh@dzh-Lenovo-Legion-Y7000:~/slambook/DynaSLAM$ ./Examples/RGB-D/rgbd_tum ./Vocabulary/ORBvoc.txt ../TUM_DataSet/TUM3.yaml ../TUM_DataSet/freiburg3_walking_xyz/ ../TUM_DataSet/freiburg3_walking_xyz/associations.txt ./mask ./output
nImages: 633
Loading Mask R-CNN. This could take a while...
Importing Mask R-CNN Settings...
Using TensorFlow backend.
2023-05-14 11:22:28.304109: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2023-05-14 11:22:28.333932: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2023-05-14 11:22:28.334409: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a03614dc60 executing computations on platform Host. Devices:
2023-05-14 11:22:28.334430: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
Initializing Mask RCNN network...
./src/python
WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:488: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:63: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:3626: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:3454: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1818: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1204: calling reduce_max_v1 (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1238: calling reduce_sum_v1 (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /home/dzh/slambook/DynaSLAM/src/python/model.py:315: The name tf.log is deprecated. Please use tf.math.log instead.WARNING:tensorflow:From /home/dzh/slambook/DynaSLAM/src/python/model.py:366: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/dzh/slambook/DynaSLAM/src/python/model.py:390: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:2990: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/dzh/slambook/DynaSLAM/src/python/model.py:746: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are twooptions available in V2.- tf.py_function takes a python function which manipulates tf eagertensors instead of numpy arrays. It's easy to convert a tf eager tensor toan ndarray (just call tensor.numpy()) but having access to eager tensorsmeans `tf.py_function`s can use accelerators such as GPUs as well asbeing differentiable using a gradient tape.- tf.numpy_function maintains the semantics of the deprecated tf.py_func(it is not differentiable, and manipulates numpy arrays). It drops thestateful argument making all functions stateful.WARNING:tensorflow:From /home/dzh/anaconda3/envs/tf114py27/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:158: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.2023-05-14 11:22:32.947268: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
Initialated Mask RCNN network...
Creating net instance...
Loading net parameters...
Mask R-CNN loaded!DynaSLAM Copyright (C) 2018 Berta Bescos, University of Zaragoza.Input sensor was set to: RGB-DLoading ORB Vocabulary. This could take a while...
Vocabulary loaded!Camera Parameters: 
- fx: 0
- fy: 0
- cx: 0
- cy: 0
- k1: 0
- k2: 0
- p1: 0
- p2: 0
- fps: 30
- color order: RGB (ignored if grayscale)ORB Extractor Parameters: 
- Number of Features: 1000
- Scale Levels: 8
- Scale Factor: 1.2
- Initial Fast Threshold: 20
- Minimum Fast Threshold: 7Depth Threshold (Close/Far Points): -nan-------
Start processing sequence ...
Images in the sequence: 633Light Tracking not working because Tracking is not initialized...
New map created with 334 points
Light Tracking not working...
virtual int g2o::SparseOptimizer::optimize(int, bool): 0 vertices to optimize, maybe forgot to call initializeOptimization()
Light Tracking not working...
Track lost soon after initialisation, reseting...
System Reseting
Reseting Local Mapper... done
Reseting Loop Closing... done
Reseting Database... done
Light Tracking not working because Tracking is not initialized...
New map created with 340 points
段错误 (核心已转储)

 #warning "Using deprecated NumPy API, disable it by " \^~~~~~~
In file included from /usr/include/python2.7/numpy/ndarraytypes.h:1809:0,from /usr/include/python2.7/numpy/ndarrayobject.h:18,from /home/dzh/slambook/DynaSLAM/include/Conversion.h:17,from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26,from /home/dzh/slambook/DynaSLAM/Examples/Stereo/stereo_kitti.cc:19:
/usr/include/python2.7/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]#warning "Using deprecated NumPy API, disable it by " \^~~~~~~
In file included from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26:0,from /home/dzh/slambook/DynaSLAM/Examples/Monocular/mono_kitti.cc:20:
/home/dzh/slambook/DynaSLAM/include/Conversion.h:27:12: warning: ‘int DynaSLAM::failmsg(const char*, ...)’ declared ‘static’ but never defined [-Wunused-function]static int failmsg(const char *fmt, ...);^~~~~~~
/home/dzh/slambook/DynaSLAM/include/Conversion.h:45:18: warning: ‘PyObject* DynaSLAM::failmsgp(const char*, ...)’ declared ‘static’ but never defined [-Wunused-function]static PyObject* failmsgp(const char *fmt, ...);^~~~~~~~
In file included from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26:0,from /home/dzh/slambook/DynaSLAM/Examples/Monocular/mono_tum.cc:19:
/home/dzh/slambook/DynaSLAM/include/Conversion.h:27:12: warning: ‘int DynaSLAM::failmsg(const char*, ...)’ declared ‘static’ but never defined [-Wunused-function]static int failmsg(const char *fmt, ...);^~~~~~~
/home/dzh/slambook/DynaSLAM/include/Conversion.h:45:18: warning: ‘PyObject* DynaSLAM::failmsgp(const char*, ...)’ declared ‘static’ but never defined [-Wunused-function]static PyObject* failmsgp(const char *fmt, ...);^~~~~~~~
In file included from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26:0,from /home/dzh/slambook/DynaSLAM/Examples/RGB-D/rgbd_tum.cc:19:
/home/dzh/slambook/DynaSLAM/include/Conversion.h:27:12: warning: ‘int DynaSLAM::failmsg(const char*, ...)’ declared ‘static’ but never defined [-Wunused-function]static int failmsg(const char *fmt, ...);^~~~~~~
/home/dzh/slambook/DynaSLAM/include/Conversion.h:45:18: warning: ‘PyObject* DynaSLAM::failmsgp(const char*, ...)’ declared ‘static’ but never defined [-Wunused-function]static PyObject* failmsgp(const char *fmt, ...);^~~~~~~~
In file included from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26:0,from /home/dzh/slambook/DynaSLAM/Examples/Stereo/stereo_kitti.cc:19:
/home/dzh/slambook/DynaSLAM/include/Conversion.h:27:12: warning: ‘int DynaSLAM::failmsg(const char*, ...)’ declared ‘static’ but never defined [-Wunused-function]static int failmsg(const char *fmt, ...);^~~~~~~
/home/dzh/slambook/DynaSLAM/include/Conversion.h:45:18: warning: ‘PyObject* DynaSLAM::failmsgp(const char*, ...)’ declared ‘static’ but never defined [-Wunused-function]static PyObject* failmsgp(const char *fmt, ...);^~~~~~~~
/home/dzh/slambook/DynaSLAM/include/Conversion.h:25:18: warning: ‘DynaSLAM::opencv_error’ defined but not used [-Wunused-variable]static PyObject* opencv_error = 0;^~~~~~~~~~~~
In file included from /usr/include/python2.7/numpy/ndarrayobject.h:27:0,from /home/dzh/slambook/DynaSLAM/include/Conversion.h:17,from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26,from /home/dzh/slambook/DynaSLAM/Examples/Monocular/mono_kitti.cc:20:
/usr/include/python2.7/numpy/__multiarray_api.h:1453:1: warning: ‘int _import_array()’ defined but not used [-Wunused-function]_import_array(void)^~~~~~~~~~~~~
/home/dzh/slambook/DynaSLAM/include/Conversion.h:25:18: warning: ‘DynaSLAM::opencv_error’ defined but not used [-Wunused-variable]static PyObject* opencv_error = 0;^~~~~~~~~~~~
In file included from /usr/include/python2.7/numpy/ndarrayobject.h:27:0,from /home/dzh/slambook/DynaSLAM/include/Conversion.h:17,from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26,from /home/dzh/slambook/DynaSLAM/Examples/Monocular/mono_tum.cc:19:
/usr/include/python2.7/numpy/__multiarray_api.h:1453:1: warning: ‘int _import_array()’ defined but not used [-Wunused-function]_import_array(void)^~~~~~~~~~~~~
/home/dzh/slambook/DynaSLAM/include/Conversion.h:25:18: warning: ‘DynaSLAM::opencv_error’ defined but not used [-Wunused-variable]static PyObject* opencv_error = 0;^~~~~~~~~~~~
In file included from /usr/include/python2.7/numpy/ndarrayobject.h:27:0,from /home/dzh/slambook/DynaSLAM/include/Conversion.h:17,from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26,from /home/dzh/slambook/DynaSLAM/Examples/RGB-D/rgbd_tum.cc:19:
/usr/include/python2.7/numpy/__multiarray_api.h:1453:1: warning: ‘int _import_array()’ defined but not used [-Wunused-function]_import_array(void)^~~~~~~~~~~~~
/home/dzh/slambook/DynaSLAM/include/Conversion.h:25:18: warning: ‘DynaSLAM::opencv_error’ defined but not used [-Wunused-variable]static PyObject* opencv_error = 0;^~~~~~~~~~~~
In file included from /usr/include/python2.7/numpy/ndarrayobject.h:27:0,from /home/dzh/slambook/DynaSLAM/include/Conversion.h:17,from /home/dzh/slambook/DynaSLAM/include/MaskNet.h:26,from /home/dzh/slambook/DynaSLAM/Examples/Stereo/stereo_kitti.cc:19:
/usr/include/python2.7/numpy/__multiarray_api.h:1453:1: warning: ‘int _import_array()’ defined but not used [-Wunused-function]_import_array(void)^~~~~~~~~~~~~
/home/dzh/slambook/DynaSLAM/Examples/Monocular/mono_tum.cc: In function ‘int main(int, char**)’:
/home/dzh/slambook/DynaSLAM/Examples/Monocular/mono_tum.cc:96:48: warning: ‘MaskNet’ may be used uninitialized in this function [-Wmaybe-uninitialized]maskRCNN = MaskNet->GetSegmentation(im,string(argv[4]),vstrImageFilenames[ni].replace(0,4,"")); //0 background y 1 foreground~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dzh/slambook/DynaSLAM/Examples/Monocular/mono_kitti.cc: In function ‘int main(int, char**)’:
/home/dzh/slambook/DynaSLAM/Examples/Monocular/mono_kitti.cc:97:48: warning: ‘MaskNet’ may be used uninitialized in this function [-Wmaybe-uninitialized]maskRCNN = MaskNet->GetSegmentation(im,string(argv[4]),vstrImageFilenames[ni].replace(~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~vstrImageFilenames[ni].begin(), vstrImageFilenames[ni].end()-10,""));~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/dzh/slambook/DynaSLAM/Examples/RGB-D/rgbd_tum.cc: In function ‘int main(int, char**)’:
/home/dzh/slambook/DynaSLAM/Examples/RGB-D/rgbd_tum.cc:125:48: warning: ‘MaskNet’ may be used uninitialized in this function [-Wmaybe-uninitialized]maskRCNN = MaskNet->GetSegmentation(imRGB,string(argv[5]),vstrImageFilenamesRGB[ni].replace(0,4,""));~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 90%] Linking CXX executable ../Examples/Monocular/mono_kitti
[ 93%] Linking CXX executable ../Examples/Monocular/mono_tum
[ 96%] Linking CXX executable ../Examples/RGB-D/rgbd_tum
/home/dzh/slambook/DynaSLAM/Examples/Stereo/stereo_kitti.cc: In function ‘int main(int, char**)’:
/home/dzh/slambook/DynaSLAM/Examples/Stereo/stereo_kitti.cc:101:53: warning: ‘MaskNet’ may be used uninitialized in this function [-Wmaybe-uninitialized]maskRightRCNN = MaskNet->GetSegmentation(imRight, string(argv[4])+"/imRight",~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~vstrImageRight[ni].replace(vstrImageRight[ni].begin(), vstrImageRight[ni].end() - 10,""));~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[100%] Linking CXX executable ../Examples/Stereo/stereo_kitti
[100%] Built target mono_kitti
[100%] Built target mono_tum
[100%] Built target rgbd_tum
[100%] Built target stereo_kitti

Initialated Mask RCNN network...
Creating net instance...
Loading net parameters...
Mask R-CNN loaded!DynaSLAM Copyright (C) 2018 Berta Bescos, University of Zaragoza.Input sensor was set to: RGB-DLoading ORB Vocabulary. This could take a while...
Vocabulary loaded!Camera Parameters: 
- fx: 535.4
- fy: 539.2
- cx: 320.1
- cy: 247.6
- k1: 0
- k2: 0
- p1: 0
- p2: 0
- fps: 30
- color order: RGB (ignored if grayscale)ORB Extractor Parameters: 
- Number of Features: 3000
- Scale Levels: 8
- Scale Factor: 1.2
- Initial Fast Threshold: 20
- Minimum Fast Threshold: 7Depth Threshold (Close/Far Points): 2.98842-------
Start processing sequence ...
Images in the sequence: 633Light Tracking not working because Tracking is not initialized...
New map created with 842 points
MakeCurrent: Not available with non-pangolin window.
-------median tracking time: 2.69929
mean tracking time: 2.92223Saving camera trajectory to CameraTrajectory.txt ...trajectory saved!Saving keyframe trajectory to KeyFrameTrajectory.txt ...trajectory saved!