图像拼接——基于homography的特征匹配算法

1. 任务要求

输入：同一个场景的两张待拼接图像（有部分场景重合）。
任务：从输入的两张图像中提取特征点和描述子，可以使用现有的图像处理库来执行此任务。自己实现特征匹配算法，将来自两张图像的特征点进行匹配。最后根据匹配的特征点估计单应性变换，从而通过映射拼接成一张全景图。
输出：
- 显示两张图像上提取的特征点的位置；
- 显示特征点匹配对应的结果；
- 显示经过几何变换后的图像叠加的结果；
- 显示最终拼接的结果。

2. 数据集

其中两组图像“cat”和“bridge”拍摄于杭州。
其他图像分别来自测试数据链接和其他来源。

3. 基于homography的特征匹配算法

基于homography的特征匹配算法在图像拼接中起着关键作用，它能够定位和匹配两张待拼接图像中的特征点，从而实现图像的对齐和融合。该算法主要包括以下实现步骤：

特征点提取和描述：使用ORB和SIFT等特征检测器对待拼接图像进行特征点提取。这些特征点具有在不同尺度和旋转下的不变性。对每个特征点计算其对应的特征描述子，用于后续的特征匹配。
特征匹配：对两幅待拼接图像中的特征点进行匹配。我们使用基于最近邻的匹配，其中对于每个特征点，找到其在另一幅图像中的最佳匹配点。通过计算特征描述子之间的距离或相似度，确定最佳匹配点。
计算homography矩阵：使用筛选后的特征点匹配对应的坐标，计算homography矩阵。homography矩阵可以将一个图像上的点映射到另一个图像上，从而实现图像的对齐。
图像校准和拼接：使用计算得到的homography矩阵对多张图像进行透视变换，使其对齐。将校准后的图像进行融合，生成拼接结果图像。

import mathimport cv2 as cv
import numpy as npclass FeatureMatcher:def __init__(self, matcher_type="homography", range_width=-1, **kwargs):if matcher_type == "homography":if range_width == -1:self.matcher = cv.detail_BestOf2NearestMatcher(**kwargs)else:self.matcher = cv.detail_BestOf2NearestRangeMatcher(range_width, **kwargs)else:raise ValueError("Unknown matcher type")def match_features(self, features, *args, **kwargs):pairwise_matches = self.matcher.apply2(features, *args, **kwargs)self.matcher.collectGarbage()return pairwise_matches@staticmethoddef draw_matches_matrix(imgs, features, matches, conf_thresh=1, inliers=False, **kwargs):matches_matrix = FeatureMatcher.get_matches_matrix(matches)for idx1, idx2 in FeatureMatcher.get_all_img_combinations(len(imgs)):match = matches_matrix[idx1, idx2]if match.confidence < conf_thresh:continueif inliers:kwargs["matchesMask"] = match.getInliers()yield idx1, idx2, FeatureMatcher.draw_matches(imgs[idx1], features[idx1], imgs[idx2], features[idx2], match, **kwargs)@staticmethoddef get_confidence_matrix(pairwise_matches):matches_matrix = FeatureMatcher.get_matches_matrix(pairwise_matches)match_confs = [[m.confidence for m in row] for row in matches_matrix]match_conf_matrix = np.array(match_confs)return match_conf_matrix

4. 拼接流程展示

4.1 图片实例

为了演示图像拼接的整体实现流程，这里我们选择一组我本人拍摄的玉泉校内的两只猫和周边环境图——“cat”，两幅有重叠画面的原图如下图所示。其中下面那只白猫几乎是静止不动的，上面的带橘色斑点的白猫在两幅图中的位置有相对移动。

from stitching.images import Images# 1. load
images, low_imgs, medium_imgs, final_imgs = load_images(img_path)
images_to_match = medium_imgs# 2. plot original images
plot_images(images_to_match, (20, 20), save=f'{save_path}/1-original.png')# 3. print image size
print(f'Original image size: {images_to_match[0].shape}')################ Load images ####################
def load_images(img_path):images = Images.of(img_path)medium_imgs = list(images.resize(Images.Resolution.MEDIUM))low_imgs = list(images.resize(Images.Resolution.LOW))final_imgs = list(images.resize(Images.Resolution.FINAL))return images, low_imgs, medium_imgs, final_imgs################ Plot function####################
def plot_image(img, figsize_in_inches=(10, 10), save=None):
"""N_image = 1"""def plot_images(imgs, figsize_in_inches=(10, 10), save=None):
"""N_images > 1"""

在这里插入图片描述

4.2 特征点位图

根据特征检测器提取的特征点，生成特征点位置图。这里我们以ORB特征检测器为例，下图中的绿色小圈展示了待拼接图像中检测到的特征点的分布情况。

from stitching.feature_detector import FeatureDetector# 4. Feature detection: ORB, SIFT
finder = FeatureDetector(detector=detector)
features = [finder.detect_features(img) for img in images_to_match]key_points_img = []
for i in range(len(images_to_match)):key_points_img.append(finder.draw_keypoints(images_to_match[i], features[i]))plot_images(key_points_img, (20, 20), save=f'{save_path}/2-key_points.png')

在这里插入图片描述

4.3 特征点匹配结果

通过homography特征匹配算法（具体代码见第3节），将两张待拼接图像中匹配的特征点进行连接，生成特征点匹配结果图。下图中的绿色线段展示了特征点之间的对应关系。

from Feature_matcher import *# 5. Feature matching: homography
matcher = FeatureMatcher()
matches = matcher.match_features(features)print(matcher.get_confidence_matrix(matches))# 6. plot matching
all_relevant_matches = matcher.draw_matches_matrix(images_to_match, features, matches, conf_thresh=1,inliers=True, matchColor=(0, 255, 0))for idx1, idx2, img in all_relevant_matches:print(f"Matches Image {idx1 + 1} to Image {idx2 + 1}")plot_image(img, (20, 10), save=f'{save_path}/3-matching.png')

4.4 相机校准结果

根据homography矩阵，对两张图像进行透视变换，使其对齐，生成校准结果图。下图的子图a为校准过程得到的mask图，子图b展示了经过校准后的待拼接图像，最终拼接图的大小与待拼接图像的大小一致。

from stitching.camera_estimator import CameraEstimator
from stitching.camera_adjuster import CameraAdjuster
from stitching.camera_wave_corrector import WaveCorrector
from stitching.warper import Warper
from stitching.timelapser import Timelapser# 7. Camera Estimation, Adjustion and Correction
cameras = camera_correction(features, matches)# 8. Warp images
(warped_low_imgs, warped_low_masks, low_corners, low_sizes,warped_final_imgs, warped_final_masks, final_corners, final_sizes, frame) \= warp_image(images, cameras, low_imgs, final_imgs)plot_images(warped_low_imgs, (10, 10), save=f'{save_path}/4-warped_low_imgs.png')
plot_images(warped_low_masks, (10, 10), save=f'{save_path}/4-warped_low_masks.png')
plot_images(frame, (20, 10), save=f'{save_path}/4-warped_final_imgs.png')################ Camera Estimation ##################
def camera_correction(features, matches):camera_estimator = CameraEstimator()camera_adjuster = CameraAdjuster()wave_corrector = WaveCorrector()cameras = camera_estimator.estimate(features, matches)cameras = camera_adjuster.adjust(features, matches, cameras)cameras = wave_corrector.correct(cameras)return cameras
################ Warp images ####################
def warp_image(images, cameras, low_imgs, final_imgs):warper = Warper()warper.set_scale(cameras)low_sizes = images.get_scaled_img_sizes(Images.Resolution.LOW)camera_aspect = images.get_ratio(Images.Resolution.MEDIUM,Images.Resolution.LOW)  # since cameras were obtained on medium imgswarped_low_imgs = list(warper.warp_images(low_imgs, cameras, camera_aspect))warped_low_masks = list(warper.create_and_warp_masks(low_sizes, cameras, camera_aspect))low_corners, low_sizes = warper.warp_rois(low_sizes, cameras, camera_aspect)final_sizes = images.get_scaled_img_sizes(Images.Resolution.FINAL)camera_aspect = images.get_ratio(Images.Resolution.MEDIUM, Images.Resolution.FINAL)warped_final_imgs = list(warper.warp_images(final_imgs, cameras, camera_aspect))warped_final_masks = list(warper.create_and_warp_masks(final_sizes, cameras, camera_aspect))final_corners, final_sizes = warper.warp_rois(final_sizes, cameras, camera_aspect)# Timelapsertimelapser = Timelapser('as_is')timelapser.initialize(final_corners, final_sizes)frame = []for img, corner in zip(warped_final_imgs, final_corners):timelapser.process_frame(img, corner)frame.append(timelapser.get_frame())return (warped_low_imgs, warped_low_masks, low_corners, low_sizes,warped_final_imgs, warped_final_masks, final_corners, final_sizes, frame)

在这里插入图片描述

4.5 拼接结果

将经过校准的两张图像进行融合，生成拼接结果图。根据用户的选择，可以提供剪裁相机校准结果的选项（stitching(crop = True)，默认为False）。图1分别展示了未剪裁和剪裁后的校准图（5a&c）和拼接图时的接缝（5b&d）。最后拼接图结果见图2，上面三幅图不包括剪裁步骤，下面三幅存在剪裁步骤。可以看到，在拼接之前剪裁至规则的四边形对拼接时的seam line的选取有较大的影响，有一定概率导致最终的拼接图像不符合预期。

from stitching.cropper import Cropper
from stitching.seam_finder import SeamFinder# 9. Crop images
if crop:(cropped_low_imgs, cropped_low_masks, cropped_final_imgs,cropped_final_masks, final_corners, final_sizes, frame) = (crop_image(images, warped_low_imgs, warped_low_masks, low_corners, low_sizes,warped_final_imgs, warped_final_masks, final_corners, final_sizes))plot_images(frame, (20, 10), save=f'{save_path}/5-cropped_final_imgs.png')
else:cropped_low_imgs = warped_low_imgscropped_low_masks = warped_low_maskscropped_final_imgs = warped_final_imgscropped_final_masks = warped_final_masks# 10. Seam Masks
seam_finder, seam_masks_plots, compensated_imgs, seam_masks = (seam(cropped_low_imgs, low_corners, cropped_low_masks,cropped_final_masks, cropped_final_imgs, final_corners))
plot_images(seam_masks_plots, (15, 10), save=f'{save_path}/6-seam_masks.png')# 11. Matching result
blender = Blender()
blender.prepare(final_corners, final_sizes)
for img, mask, corner in zip(compensated_imgs, seam_masks, final_corners):blender.feed(img, mask, corner)
panorama, _ = blender.blend()
blended_seam_masks = seam_finder.blend_seam_masks(seam_masks, final_corners, final_sizes)plot_image(panorama, (20, 20), save=f'{save_path}/7-matched_result.png')
plot_image(seam_finder.draw_seam_lines(panorama, blended_seam_masks, linesize=3), (15, 10),save=f'{save_path}/8-seam_lines.png')
plot_image(seam_finder.draw_seam_polygons(panorama, blended_seam_masks), (15, 10),save=f'{save_path}/9-seam_polygons.png')# 12. Done
print('Done!')################ Crop images ####################
def crop_image(images, warped_low_imgs, warped_low_masks, low_corners, low_sizes,warped_final_imgs, warped_final_masks, final_corners, final_sizes):cropper = Cropper()mask = cropper.estimate_panorama_mask(warped_low_imgs, warped_low_masks, low_corners, low_sizes)lir = cropper.estimate_largest_interior_rectangle(mask)low_corners = cropper.get_zero_center_corners(low_corners)rectangles = cropper.get_rectangles(low_corners, low_sizes)overlap = cropper.get_overlap(rectangles[1], lir)intersection = cropper.get_intersection(rectangles[1], overlap)cropper.prepare(warped_low_imgs, warped_low_masks, low_corners, low_sizes)cropped_low_masks = list(cropper.crop_images(warped_low_masks))cropped_low_imgs = list(cropper.crop_images(warped_low_imgs))low_corners, low_sizes = cropper.crop_rois(low_corners, low_sizes)lir_aspect = images.get_ratio(Images.Resolution.LOW, Images.Resolution.FINAL)  # since lir was obtained on low imgscropped_final_masks = list(cropper.crop_images(warped_final_masks, lir_aspect))cropped_final_imgs = list(cropper.crop_images(warped_final_imgs, lir_aspect))final_corners, final_sizes = cropper.crop_rois(final_corners, final_sizes, lir_aspect)# Redo the timelapse with cropped Images:timelapser = Timelapser('as_is')timelapser.initialize(final_corners, final_sizes)frame = []for img, corner in zip(cropped_final_imgs, final_corners):timelapser.process_frame(img, corner)frame.append(timelapser.get_frame())return (cropped_low_imgs, cropped_low_masks, cropped_final_imgs,cropped_final_masks, final_corners, final_sizes, frame)

在这里插入图片描述

图1. Crop images. a, Warpped. b, Wrapped to seam. c, Warp and crop. b, Cropped to seam.

在这里插入图片描述

图2. Stitching results. a, Original result. b, Result with seam line. c, Result with seam ploygons. d-f, Cropped results.

5. 部分图像拼接结果展示

在这里插入图片描述

Fig. S1. Examples using ORB detector and without crop&mask. a, bridge. b, building. c, sportfield. d, door. e, barcode. f, exposure_error. Left, Original. Middle, Stitching results. Right, Result with seam ploygons.

创作不易，麻烦点点赞和关注咯！