Advanced Computer Vision

The goal of the lab is to implement a 3D reconstruction from multi-views of human pose on a calibrated system.

For each view, the deepnet OpenPose detects the position of the joints together with the reliability score, there are 25 joints in total as we can see in the image below :

In this lab, using the matching joints of multi-views, a 3D skeleton of the person is expected to be reconstructed by using structure from motion, camera projection and optimization knowledge.

By reading the 2DTXT file we can draw the keypoints on the video and we get the following results :

The 2DTXT file contains the coordinates output of 25 joints on each sequence of each subject in which each line is a frame that look like this :

xJ1 yJ1 rJ1 xJ2 yJ2 rJ2 xJ3 yJ3 rJ3....... xJ25 yJ25 rJ25

where x and y are horizontal and vertical pixels on image, r is the reliability score of that joint according to the deepnet (0-1)

We notice that the reconstruction is quite good and represents well the person in the video.

But we can also read the 3DTXT file to do the same thing and demonstrate how origin shifting from one camera to another affects the results

The 3DTXT file contains the coordinates output of the first 15 joints after triangulation from multiviews of 2D joints in which each line of coordinates represents a frame.

In this case, the reconstruction is also good no matter the camera's position