Estimate Homography to do 2D to 3D integration (33): Results of accessing EPnP algorithm in 4 different scenarios

I. The results show the successful projections from "3D points from Kinect camera (in World-Coordinate-System)" to "2D positions on Webcam-Image-Plane":

    - Rotation case
    - Zoom-in and Zoom-out case
    - Tilt case
    - Viewpoint Change case

II. Study how to do "Inverse transformation" comparing to the above results

    - Key words: Ray Tracing, 3D Coordinate from 2D calibrated camera


III. Future work:

     - Restart my previous work in Feature Detection, which would be the input to EPnP algorithm. Then, this work will become more like an automatic system

Estimate Homography to do 2D to 3D integration (32): Collecting Testing videos from Webcam (2D_Info)

I. Draft and modify my GUI for manually choosing markers but automatically record their position in 2D:

II. Collect 4 videos from Webcam:

    - These videos include Rotation, Zoom-in-&-Zoom-out, ViewpointChange, Tilt
    - Total are 703 frames
    - Frame size is 640x480

III.Using the tool showing in "I." for manually choosing markers but automatically record their position in 2D in each of these 703 frames:

    - 703 frames are belongs to the sum of the videos in "II."
    - Choose 8 selected points: 4 points to be the corresponding position of 3D selected ones (in green rectangle), and the rest of 4 points to be the testing data (in red circles).
    - The order of selection in each set of 4 points: 4->1->3->2

Summary of Weekly Meeting - (15)

I. To validation:
    - Take videos from Webcam
    - Use only few points: 4 is decided
    - Do "3D-World-position -> 2D-webcam-Image-position of points" with output parameters of EPnP including Rp, Tp, and Intrinsic
    - Read references and document related to "Ray tracing" & "3D coordinate from 2D calibrated camera (...interesting approach)"

George's hint.

II. Preliminary testing results for Kin2Webcam with EPnP:

Case 1: Chessboard lies on the table.

Case 2: Chessboard lies on the table.

Case 3: Chessboard lies above the table.

Case 4: Chessboard lies above the table.

III. Write a report to list and summarize my steps for how to using EPnP


IV. Write codes to do "3D-World-position -> 2D-webcam-Image-position of points", a pre-testing result like following:

     - In this case, the error is 3.419 by using EPnP+Gaussian

Left: Kinect frame (3D Info); Right: Webcam Img (2D Info)

Top: Projection result of 4 selected points from 3D-World-Coord -> 2D-Webcam-Img-Coord; Bottom: "+GT" is in Kinect-World-Coord and "+EGN/+E" is in Webcam-Img-Coord. GT means ground truth 3D position, +EGN/+EN means values computing by EPnP.

Estimate Homography to do 2D to 3D integration (31): EPnP results by using "Kin2Kin case"

Intrinsic parameter of Kinect camera. 

I. Estimation mapping position of 50 points in the camera reference:

EGNs are the position after considering translation (Tp_G), Es are not.

Rotation matrix generating from EPnP.

Translation matrix generating from EPnP.

Ground truth position. 

Estimated position before considering translation.

Estimated position after considering translation.

II. Estimation mapping position of 4 points in the camera reference:

EGNs are the position after considering translation (Tp_G), Es are not.

Rotation matrix generating from EPnP.

Translation matrix generating from EPnP.

Ground truth position. 

Estimated position before considering translation.

Estimated position after considering translation.

III. Errors when comparing the position of 50-point-case / 4-point-case to the Ground Truth:
       
       - The statistics shows a correlated results that "Giving more points up to 50" is able to provide more stable and precise estimation in "computing the 3D corresponding position to 2D".

Ground truth.

Estimated position of EGN-50 after considering translation.

Estimated position of EGN-4 after considering translation.

Differences of  "EGN-50 vs GT"  and  "EGN-4 vs GT

IV. Do the same experiment on the Webcam:

       - Check the calibration result is reasonable

       - Generating testing images

       - Locating 50 points the mapped 2D points in images to the 3D ones in Kinect frame (...using the Kinect frame showing above)

"Lie on plane" case 1 with marks.

"Lie on plane" case 2 with marks.

"Above the plane" case 1 with marks.

"Above the plane" case 2 with marks.

Estimate Homography to do 2D to 3D integration (30): Manually get the coordinates of 50 interesting points on moving camera

I. Case 1: Object lies on the plane:

frameColor2 (with marks)_lieOnPlane1.jpg.

frameColor2 (with marks)_lieOnPlane3.jpg.

frameColor2 (with marks)_lieOnPlane4.jpg.

II. Case 2: Object lies above the plane:

frameColor2 (with marks)_abovePlane2.jpg

frameColor2 (with marks)_abovePlane3.jpg

frameColor2 (with marks)_abovePlane5.jpg