Sparse Reconstruction
You now have all the pieces you need to generate a full 3D reconstruction.
Write a test script python/test temple coords.py that does the following:
import numpy as np
from numpy.lib.arraysetops import in1d
import helper as hlp
import skimage.io as io
import submission as sub
import matplotlib.pyplot as plt
import cv2 as cv
import numpy.linalg as la
import skimage.color as col
from PIL import Image
import os
path = os.path.abspath(__file__)
dir_path = os.path.dirname(path)
os.chdir(dir_path)
1. Load the two images and the point correspondences from data/some corresp.npz
파일 로딩하기
image1 = Image.open('../data/im1.png')
image2 = Image.open('../data/im2.png')
M1 = image1.size
M2 = image2.size
i1 = cv.imread('../data/im1.png')
i2 = cv.imread('../data/im2.png')
# 1. Load the two temple images and the points from data/some_corresp.npz
I1 = cv.cvtColor(i1, cv.COLOR_BGR2GRAY).astype(np.float32)
I2 = cv.cvtColor(i2, cv.COLOR_BGR2GRAY).astype(np.float32)
plt.imshow(I1, cmap='gray')
plt.imshow(I2, cmap='gray')
# plt.show()
2. Run eight point to compute the fundamental matrix F
8 포인트 알고리즘을 이용하여 Fundamental Matrix F를 계산하자. 포스팅 참조
corresp = np.load('../data/some_corresp.npz')
pts1, pts2 = corresp['pts1'], corresp['pts2']
F_true, mask = cv.findFundamentalMat(pts1,pts2,cv.FM_8POINT) #ref
F_true2, mask = cv.findFundamentalMat(pts1,pts2,cv.FM_LMEDS) #ref
F = sub.eight_point(pts1, pts2, M1)
# We select only inlier points
pts1 = pts1[mask.ravel()==1]
pts2 = pts2[mask.ravel()==1]
n = np.size(pts1,0)
hlp.displayEpipolarF(i1, i2, F)
3. Load the points in image 1 contained in data/temple coords.npz and run your epipolar correspondences on them to get the corresponding points in image 2
Pass
coords = np.load('../data/temple_coords.npz')
4. Load data/intrinsics.npz and compute the essential matrix E.
F와 E의 관계를 이용하여 E를 계산해보자. (값이 다르게 나오는듯?)
pts2 = sub.epipolar_correspondences(I1, I2, F_true, pts1)
hlp.epipolarMatchGUI(i1, i2, E_true[0])
5. Compute the first camera projection matrix P1 and use camera2 to compute the four candidates for P2
K1, R1, t1을 이용하여 카메라 행렬 P1를 만들고, intrinsic 파라메터를 이용하여 카메라 행렬 P2를 위한 4개의 후보군을 생성하자.
# stereo image에서 image1을 기준으로 하므로 extrinsic parameter는 회전/이동이 없다고 가정한다.
I33 = np.eye(3,3)
Z31 = np.zeros([3,1])
# P = K[R|t]
R1 = I33
t1 = Z31
P1 = sub.makeCameraMat(R1,t1,K1)
6. Run your triangulate function using the four sets of camera matrix candidates, the points from data/temple coords.npz and their computed correspondences.
4개의 카메라 행렬 P2위 후보군을 Triangulation 하여 각 연관된 3차원 점을 찾아보자.
for i in range(n):
for j in range(4):
P2_candidate = K2 @ M2s[:,:,j]
kp1 = np.array(pts1[i,:].T).reshape(1,2)
kp2 = np.array(pts2[i,:].T).reshape(1,2)
kp1_3D = np.ones((3, kp1.shape[0]))
kp2_3D = np.ones((3, kp2.shape[0]))
kp1_3D[0], kp1_3D[1] = kp1[:, 0].copy(), kp1[:, 1].copy()
kp2_3D[0], kp2_3D[1] = kp2[:, 0].copy(), kp2[:, 1].copy()
X = cv.triangulatePoints(P1[:3], P2_candidate[:3], kp1_3D[:2], kp2_3D[:2])
X /= X[3] # 3차원의 크기를 1로 설정
X1 = P1[:3] @ X
X2 = P2_candidate[:3] @ X
#pts3Ds_true[i,3,j] = X
pts3Ds_true = X
pts3Ds = sub.triangulate(P1, pts1[i,:], P2_candidate, pts2[i,:])
7. Figure out the correct P2 and the corresponding 3D points. Hint: You’ll get 4 projection matrix candidates for camera2 from the essential matrix. The correct configuration is the one for which most of the 3D points are in front of both cameras (positive depth).
4개의 P2 후보군에서 가장 맞는 3차원 점이 무엇인지 찾아보자.
8. Use matplotlib’s scatter function to plot these point correspondences on screen. 9
그림을 그려보자.
9. Save your computed extrinsic parameters (R1,R2,t1,t2) to data/extrinsics.npz. These extrinsic parameters will be used in the next section.
파라메터를 저장한다.
