Your mission in this project is to construct an indoor 3D scanner using a robot arm (link). You will experiment with two camera systems: a calibrated stereo pair and a structured lighting Zivid (link).
Note that coming into the project, you will be provided with all the software you need to control the arm and the cameras. Concerning the stereo camera system, you will be given a means to calibrate the cameras and generate 2.5D images of the scanned object. The robot arm software you are provided with should also give you accurate (sub-millimetre) pose estimates, allowing you to determine the relative positions of each 2.5D image captured to assemble a full 3D reconstruction.
Your primary focus will be on combining individual frames into a cohesive 3D model, which can be tricky. Typically, a significant overlap exists between frames, which leads to considerable data redundancy. Measurement noise (along with minor changes in lighting and shadows) is another issue that can produce noticeable data inconsistencies. Your challenge in this project is to find ways to combine individual point clouds into a single cohesive model with an appropriately sized data footprint, appropriate noise filtering, and sufficient interpolation between frames to minimize "seam" artefacts as optimally as possible.
Concerning theory, older techniques such as marching cubes, Poisson surface reconstruction and ball pivoting lack robustness and require high-quality raw point clouds. Volumetric approaches are promising, and it is worth looking at Kinfu libraries in OpenCV's "contrib" modules - although this approach is limited in resolution. Modern machine learning-based approaches are currently leading the way [1,2].
Note that you will have access to a 3D printer that might be a helpful tool for assessing models.
Your research will be done in the context of the CARES (link) robotics faculty centre, and your work could contribute to ground-truthing robotic vision systems. Note that in this context, there will likely be a focus on 3D reconstruction for agricultural applications.
[1] Wang, Fangjinhua, et al. "Patchmatchnet: Learned multi-view patchmatch stereo." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
[2] Azinović, Dejan, et al. "Neural rgb-d surface reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
Undergraduate
The outcomes of this project should be as follows:
Students should be competent Python programmers, as well as have a workable understanding of linear algebra.
Robotics (405.652, Lab)