Work of Camera Calibration (3) : Related Resources

I. Useful Toolbox in MATLAB:

。Computer Vision System Toolbox: http://www.mathworks.co.uk/products/computer-vision/

。Machine Vision Toolbox:
      - http://www.petercorke.com/Machine_Vision_Toolbox.html
      - Manual: http://www.petercorke.com/MVTB/vision.pdf
      - Taught Website: http://www.petercorke.com/RVC/
      - http://www.zhizhihu.com/html/y2010/1550.html (in Chinese)

      - EPnP (just same as the one downloaded via cvlab directly):
        http://stackoverflow.com/questions/20180162/camera-pose-estimation-using-qr-codes-in-matlab

。Camera Calibration Toolbox: http://www.vision.caltech.edu/bouguetj/calib_doc/

EPnP: Summary of the paper - (I)

I. ... complete this page during weekend, to copy my note from notebook to here. (The note is a
   summary of the EPnP paper proposed in IJCV2008)

II. Other progress: 
     
     。Reviewing the ppt files used in MRes course, which provided by Dr. Su-Lin et al. in the
         'Image guided Intervention'. These document give some idea about geometry.

     。Spending a bit of time to survey prospective work using SIFT in matching among/between 2D and 3D
         space.

Summary of Weekly Meeting - (4)

I. Treat the EPnP method as a black box:

   。Knowing how to use it to accomplish the work of '2D to 3D mapping' is the main part of this stage.

II. Literature review & reading materials searching based on following keywords:

   。3D geometry

   。2D to 3D geometry

   。3D to 2D mapping

   。proposed PnP method (especially the ones cited in the 'EPnP paper')

III. Trying an alternative method for 2D to 3D mapping

     。Using SIFT method to do image matching between head-mounted frame and
         Kinect frame-of-reference (without depth, it's 2D), and then introduce depth information (generated
         from Kinect camera) to show the viewpoint of head-mounted in the 3D Kinect frame-of-reference.

     。Still trying to apply EPnP method in our project, doing this job in parallel

Work of Camera Calibration (2) : Experimental Setup

I would like to use 2 sets of testing datasets for a preliminary examination of the EPnP source code in MATLAB. This examination is to try to understand the way how EPnP method works. This process is also the prospective solution in integrating 2D position of a head-mounted camera into a 3D Kinect frame-of-reference.

In terms of the idea described in the proposed papers will be summarized in other notes within this blog. Here are the 2 datasets:

I. Testing dataset 1:

  。Kinect frame-of-reference:
      Each frame size is 640x480. The first image is a color image, which is generated from Kinect video
      sensor (in RGB format). The second image is a depth image, which range of a pixel value is from 0 to 
      4000.

。head-mounted camera frame:
    (use a digital camera taking photos as a simulation of generating vieww from a head-mounted camera)
    I take photos from 4 directions in a clockwise way, from the right to the left. No specific angles are
    considering in this moment, this is only for preliminary testing. Each frame size is 4608x3456.

II. Testing dataset 2:

。Kinect frame-of-reference:

。head-mounted camera frame:
    (use a digital camera taking photos as a simulation of generating vieww from a head-mounted camera)

Kinect with Matlab - (2)

I follow 2 tutorials from the following 2 websites to generate 'Color' & 'Depth' information produced by a Kinect camera and then use an example of  'Skeleton Tracking' as a testing.

In terms of showing both of the information in Color and Depth, it is playing in real-time across a sequential time-domain. This could be useful in displaying any results later (showing a dot of head-mounted camera in Kinect frames continuously).

I. Generate the Color and Depth:
   。References:
        - http://www.mathworks.co.uk/videos/using-kinect-for-windows-with-matlab-77799.html
        - http://www.mathworks.co.uk/help/imaq/examples/using-the-kinect-r-for-windows-r-from-image-acquisition-toolbox-tm.html

   。Results:

II. An Example: Skeleton Tracking:

   。Reference:
       - http://www.mathworks.co.uk/help/imaq/examples/using-the-kinect-r-for-windows-r-from-image-acquisition-toolbox-tm.html
       - http://www.mathworks.co.uk/help/imaq/acquiring-image-and-skeletal-data-using-the-kinect.html

   。Results:

Work of Camera Calibration (1) : Source Code of EPnP-based

I. The reference  is from: 
   。http://cvlab.epfl.ch/software/EPnP/
   
II. About could not extract the source code of file in C++:
   。Have inquired the author, not yet to receive a response
   。Yet... I find the solution to solve the issue. The extracted file is in '.tar', 
        which need to use at least '7-Zip' to unpack it. Using WinRaR is not appropriate. 

        Here is some reference of this matter: 
        http://www.haskell.org/haskellwiki/How_to_unpack_a_tar_file_in_Windows

III. Today's process: 
     。Tracing the source code in Matlab (not finish, but is already to be performing 
         and have results which those authors wanted to demonstrate)

Solution to back-and-forth between OpenNI v1.5 and OpenNI v2.2

Here is the website in Chinese which describes the whole processes in detail: http://blog.csdn.net/chenyusiyuan/article/details/8773007, I make a summary about this.

Just for a note, all libraries are installed already, including OpenNI v1.5.4, Sensor v5.1, OpenNI v2.2, Microsoft Kinect SDK v1.8.

I. From PrimeSense to Microsoft Kinect SDK

   1. Choose 'Update the driver':

    2. 'Automatically searching for appropriate driver'

    3. Done! Then Kinect camera is recognized as 'Kinect for Windows Camera'

 
    4. We can see the it on this list.

    5. Do not need to re-start the system

II. From Microsoft Kinect SDK to PrimeSense:

    1. Choose 'Update the driver':

    2. Choose the either of the way, the important thing is that the path of '*.inf' should be covered under the
        path (this path is listed in step 3)

 
    3. Choose the *.inf file via this path 'C:\Program Files (x86)\OpenNI\Driver' in x64-bit platform (attention
        to the path, which is not the one listed in the reference website in the beginning of this article)
                                
    4. Done! Then Kinect camera is recognized as 'Kinect4Windows Camera'

    5. We can see the it on this list. (this is a different name from the in 'I.')

    6. Need to re-start the system, or any examples written by OpenNI v1.x could not find
        PrimeSense driver, then some error or problem is going to show up.

III. The importance of using OpenNI v2.x 

1. Starting from OpenNI v2.x, the driver will be Microsoft Kinect SDK, and this is for C++
    platform. At the same time, in Matlab, after v2013a, the is a Kinect for Microsoft library can be
    add-on. 

    After trying for several times, a programmar can do programming in either C++ or Matlab
    without considering changing the Kinect driver. The driver both languages use is Microsoft
    Kinect SDK. 

    *
    But if we want to go back to OpenNI v1.5 in C++, then the driver should be changed to the
    PrimeSense one. The way to go-back between the 2 different drivers are already described on
    this page.  

2. OpenNI v2.x is the latest one, maybe since the end of 2012 or the early 2013

3. OpenNI v2.x simplifies the functions and variables, and combines them into a minimal number

Sample of Kinect Calibration in C++ - (2)

I. The hardware issue for setting up multiple Kinect cameras on same computer:

。Even it looks like having more than 1 USB ports on a computer on the outside, in the side of
    computer, all of the ports are controlled by the same USB controller. This will be a reason why
    we are not able to control multiple Kinect cameras at the same time: http://0rz.tw/qqGSX

II. The modifying version of source code compatible for OpenNI 1.5.2 version:

。Libraries: OpenNI 1.5.2 (x86), SenseKinect 5.1.0 (x86), Qt 5.1.1 (for VS2010 32-bit with OpenGL),
    Qt5 Visual studio Add-in, VS2010 professional
。Tutorial: http://kinectcar.ronsper.com/docs/openni/index.html
。Code: http://0rz.tw/igyvG

III. The modifying version of source code compatible for OpenNI 2.1 version:

。Libraries: OpenNI 2.1  (x86), Microsoft Kinect SDK 1.8, NITE 2.0.0  (x86),
    Microsoft Kinect Develop Toolkit 1.8

。How to Install OpenNI 2 + Nite 2 + Kinect SDK 1.6 + windows 7 32/64 bit: 
       - http://www.youtube.com/watch?v=m5uTH3S9P9g (in English)
      - http://blog.csdn.net/zhuxiaoyang2000/article/details/9087571 (in Chinese)
      - Some results for making sure that I have successfully configured the environment out: (The first 2
         pictures related to hand_tracking, the last 2 pictures related to human_tracking with
        skeleton calibration)

。The way to substitute variables in OpenNI 1.5 into OpenNI 2.x:
    http://www.openni.org/openni-migration-guide/
    http://www.openni.org/reference-guide/?t=namespaceopenni.html

。The essential idea of improving configuration in OpenNI 2.x:

  。 Here is the reference of new classes:

 

。Code: still working on!

Kinect with Matlab - (1)

I. Demo:

http://www.mathworks.co.uk/videos/using-kinect-for-windows-with-matlab-77799.html

II. Introduction:

http://www.mathworks.co.uk/hardware-support/kinect-windows.html 

http://www.timzaman.com/?p=504

http://labrigger.com/blog/2013/06/07/matlab-2013a-natively-supports-kinect/

III. Tutorial of setup:

http://www.mathworks.co.uk/help/imaq/installing-the-kinect-for-windows-runtime.html

IV. Tutorial of using Matlab to communicate with Kinect:

http://www.mathworks.co.uk/help/imaq/examples/using-the-kinect-r-for-windows-r-from-image-acquisition-toolbox-tm.html

I have installed Matlab 2013b, and installed the 'Kinect SDK for Matlab' via Adds-on. I follow the instruction within the upper link, which gives a simple process to interact with Kinect in a basic way.  Here is the preliminary results of testing:

This picture shows that Matlab is able to recognize Kinect.

The last 2 pictures show that we are setting to activate the color camera & depth camera though manual way instead of letting it being triggered immediately. The programme will pre-record about 100 frames in order to compensate the latency of detection.

It is just a testing. I found some information that Microsoft Kinect SDK can support the latest version of Matlab as an additional library. The 'PnP 2D-3D mapping algorithm' provided both codes in Matlab and C++. Just wanna try whether it is possible to build up our project fully based on Matlab platform or not. So far, I can generate a color image from Kinect, but not sure about the depth and color information are also recorded.

-------

Memo:

1. Old version of Kinect example:
     。http://www.javaforge.com/wiki/102821
    。Trying this code first, this is a code without integrate Kinect application & functions into Matlab
         library. Let's see how is the speed and result. 
    。After this, I will uninstall my current Matlab version (2012a) and re-install in 2013b

2. Computer Vision toolbox for Matlab:
    。http://www.mathworks.co.uk/products/computer-vision/
    。http://www.mathworks.co.uk/products/computer-vision/description4.html

Sample of Kinect Calibration in C++ - (1)

I. The tutorial of learning OpenNI:

http://www.openni.org/openni-migration-guide/

II. Debugging the sample code of Kinect Camera calibration: 

Here is a simple example about how to use 2 Kinect cameras to cover the 3D environment from different viewpoints: 
。http://viml.nchc.org.tw/blog/paper_info.php?CLASS_ID=1&SUB_ID=1&PAPER_ID=216 
。the other explanation webpage:
    http://kheresy.wordpress.com/2011/01/20/read_kinect_depth_data_via_openni/

In the beginning, this the example is not able to be executed correctly, because of mis-linking to Qt library. After searching solution from forum, I should modified the existing setup in the code to my current path of library of Qt. 
。C++/general:
     $(QTDIR)\include, 
     $(QTDIR)\include\QtWidgets
     $(QTDIR)\include\QtCore
     $(QTDIR)\include\QtOpenGL
     $(QTDIR)\include\QtGui
。Linker/general:
     $(QTDIR)\lib
。Linker/input:
     $(QTDIR)\lib\Qt5OpenGLd.lib
     $(QTDIR)\lib\Qt5Widgetsd.lib
     $(QTDIR)\lib\Qt5Guid.lib
     $(QTDIR)\lib\Qt5Cored.lib

III. Problems after solving linking Qt libraries: 

However, currently this code is for OpenNI 1.5 version, not for OpenNI 2.x version. There is kind of conflict issue when OpenNI 1.5 + SensorKinect 5.1.6.6 and OpenNI 2.x + Microsoft Kinect SDK are installed together on the same Windows platform. The problem I found are:
。Problem 1: XnCppWrapper.h can not be include in VS2010: if we use OpenNI 2.x version.
    There is no this h-file in the latest version of OpenNI.
。Problem 2: OpenNI.dll missing: once we installed OpenNI 1.x version together with version 2.x 

Speaking of problem 1, XnCooWrapper.h is the header file in OpenNI 1.5; we need to change it into OpenNI.h which is the header file in OpenNI 2.x.

To solve problem 2, I decide to re-install the libraries related Kinect. The following page shows the libraries for OpenNI 1.5.4 version: 
。http://pointclouds.org/downloads/windows.html (Here provides the accessible libraries in OpenNI 1.5 version)
。The steps for installation of Kinect libraries
    (http://kheresy.wordpress.com/2012/03/15/2012_openni_env_kinect_setup/)
    (http://studentguru.gr/b/vangos/archive/2011/01/20/how-to-successfully-install-kinect-windows-openni-nite)

       1. OpenNI 1.5.4 - WinX32:
           This is the based platform for Kinect. 

       2. SenseKinect 5.1.6.6- WinX32:
           The driver of Kinect for supporting OpenNI (OpenNI 2.2 version could just install Kinect SDK,
           the official driver released by Microsoft) which is release from PrimeSense.  

       3. NiTE 1.5.2 - WinX32:
           Mainly this one is for extracting information for skeleton and gesture etc.; because we just 
           need RGB & Depth information, this one is just installed for preparation for possible future 
           usage. 
       
           (*Furthermore, due to the value of resolution of Kinect camera is 640x480, we need to modify 
              the corresponding value in the document of NiTE from 340x240 to 640x480 instead. The 
              simple way is use all XML files within 'Sensor: /SampleXMLs/NITE/Data/', and substitute 
              the 3 in 'NITE: /Data/')

       4. Plugging in the Kinect to computer:
           The yellow signal light should be sparkling now.

       5. Testing the examples 'NiViewer.exe' & 'NiSimpleViewer.exe':
            Both these 2 applications are located in 'OpenNI: Samples/Bin/Release/'. 
            Here is the figure of executing results.

       6. Testing the examples 'NiUserTracker.exe':
            This programme shows different depth information of objects and mark objects by using
            different colors in gray-level on the depth map, and using colors in RGB space on the real 
            scene. Here is the results:

IV. Here is a way to make OpenNI 1.5 + SensorKinect 5.1.6.6 and OpenNI 2.x + Microsoft 

      KinectSDK exist simultaneously: (http://blog.csdn.net/chenyusiyuan/article/details/8773007)

      1. Open the 'Device Manager'

     2. Currently, in 'PrimeSense' item, this means Kinect is driving by SenseKinect driver, not                  Microsoft Kinect SDK. We could change the driver of 'Kinect4Windows Camera', from 

         SenseKinect (PrimeSense) into Mocrosoft Kinect SDK. Yet, we have to uninstall this 

         'Kinect4Windows Camera' first by right click the mouse button as the pointer is on this item.

         To point out, while we install the Microsoft SDK, we could just choose the option 'Search              automatically for updated software':

         

         and then choose the 2nd one 'No, let me choose the operation to execute':

     3. If we would like to back to PrimeSense one again, just repeat the same process.

         To point out, while we install the PrimeSense driver, we should choose the opposite option:

      

     4. This is the way to make both of this 2 system configuration be existed at the same time!

Calibration of 2 Kinect Camera

[References]

1. http://www.mobile01.com/topicdetail.php?f=282&t=1894309

2. http://idav.ucdavis.edu/~okreylos/ResDev/Kinect/index.html

3. http://viml.nchc.org.tw/blog/paper_info.php?CLASS_ID=1&SUB_ID=1&PAPER_ID=274 (VIP)

----

[Examples]

1. http://www.dotblogs.com.tw/ouch1978/archive/2011/08/07/kinectsdk-videoframe.aspx?pg=1

2. http://viml.nchc.org.tw/blog/paper_info.php?CLASS_ID=1&SUB_ID=1&PAPER_ID=274 (VIP)

----

[Video]

----

[News]

1. http://mag.udn.com/mag/digital/storypage.jsp?f_MAIN_ID=319&f_SUB_ID=2940&f_ART_ID=286202

2. http://social.msdn.microsoft.com/Forums/zh-CN/dfccab8f-0d87-4e20-9fb9-3b484c6c3f5b/kinect?forum=k4wdevelopment

PCL Installation

1.
    http://www.cnblogs.com/Akunwjy/p/PCL.html

2. Variables Setup:
    http://www.cnblogs.com/uestc-wxp/p/3569400.html

OpenNI範例

http://blog.csdn.net/chenxin_130/article/details/8580706

http://blog.csdn.net/chenxin_130/article/details/8619909

http://blog.csdn.net/chenxin_130/article/details/6696187

OpenCV: Using Kinect and other OpenNI compatible depth sensors

References in English:

1. OpenCV, HighGUI: http://docs.opencv.org/doc/user_guide/ug_highgui.html

2. GETTING STREAMS FROM KINECT USING OPENNI2:
    http://tnq177.wordpress.com/2013/02/06/getting-streams-from-kinect-using-openni2/

----

References in Chinese:

1. opencv2.3读取kinect深度信息和彩色图像:
    http://blog.csdn.net/moc062066/article/details/6949910

2. OpenCV中Kinect的使用（1）: 
   http://www.cnblogs.com/steven-blog/p/3389904.html

3. 用 OpenCV 畫出 OpenNI 2 的深度、彩色影像:
    http://kheresy.wordpress.com/2013/01/09/%E7%94%A8-opencv-%E7%95%AB%E5%87%BA-openni-2-%E7%9A%84%E6%B7%B1%E5%BA%A6%E3%80%81%E5%BD%A9%E8%89%B2%E5%BD%B1%E5%83%8F/

4. Kinect+OpenNI学习笔记之4(OpenNI获取的图像结合OpenCV显示):
    http://www.cnblogs.com/tornadomeet/archive/2012/10/01/2709851.html

5. Kinect SDK(1)：读取彩色、深度、骨骼信息并用OpenCV显示:
    http://blog.csdn.net/yangtrees/article/details/16106271

*. Just for remark on Google Search:
    https://www.google.co.uk/search?q=opencv+%E4%BD%BF%E7%94%A8kinect&oq=%E4%BD%BF%E7%94%A8kinect+&aqs=chrome.1.69i57j0.11520j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8

----

Installation of OpenCV:

1. http://wiki.opencv.org.cn/index.php/VC_2010%E4%B8%8B%E5%AE%89%E8%A3%85OpenCV2.4.4

2. Library set on local:
    http://tim12332013.blogspot.co.uk/2013/08/opencv-visual-studio-2012-opencv-246.html

3. Library not set on local:
    http://www.dotblogs.com.tw/v6610688/archive/2013/10/25/image_process_intro_opencv.aspx

4. https://www.librehat.com/visual-studio-2012-with-opencv-2-4-7-installation/




----

Demo:

1. Extending Depth Range (using Kinect SDK/OpenCV): 
    http://www.youtube.com/watch?v=38tJ5KBZ69s

2. PCL+OpenNI2 +OpenCV 2.4.6 + PrimeSense:
    http://www.youtube.com/watch?v=Ghm_64C3hPU

3. 如何使用 Kinect 辨識判斷手勢動作(For Windows SDK V1)
   http://www.youtube.com/watch?v=vWnfjSD7GgU

Library for Kinect

1. Openni: http://www.openni.org/openni-sdk/

2. PCL: http://pointclouds.org/about/

3. Official Website for Kinect: http://www.microsoft.com/en-us/kinectforwindows/

------

4. Open Kinect: http://openkinect.org/wiki/Gallery

5. OpenCV: http://docs.opencv.org/doc/user_guide/ug_highgui.html


------

<Installation Sequence>

1. OpenNI 2.x + Kinect:
    http://blog.csdn.net/zhuxiaoyang2000/article/details/9087571

2. Kinect开发教程六：OpenNI2简介、安装与VS开发环境配置:
    http://blog.csdn.net/chenxin_130/article/details/8580636
   
    Kinect开发教程七：OpenNI2自带范例程序的编译执行:
    http://blog.csdn.net/chenxin_130/article/details/8580706
   

    Kinect学习笔记(1)-Kinect在PCL下的安装和配置:
    http://blog.sina.com.cn/s/blog_6599e0770101cow3.html

3. http://blog.csdn.net/zhuxiaoyang2000/article/details/9087571

4. For OpenNI 1.5.x:
    http://www.cnblogs.com/tornadomeet/archive/2012/09/26/2704046.html


<How to Make OpenNI communicate with Kinect SDK>

1. 讓 OpenNI 可以透過微軟 Kinect SDK 讀取 Kinect 的資料:
     http://kheresy.wordpress.com/2011/07/01/using_kinect_via_microsoft_sdk/

2. 強制更換 Kinect 的驅動程式（for OpenNI）:
     http://kheresy.wordpress.com/2012/07/24/modify-diver-of-kinect-for-openni/


<The Difference between OpenNI & Kinect SDK>

1.http://blog.csdn.net/dingluseu/article/details/11629245