Pose estimation and point cloud perception

Algorithm development(2022)

Pose estimation and point cloud perception

Pose estimation

My undergraduate thesis focused on the estimation of object poses. I input RGB images and the 3D models of the detected objects, mapping the 2D image pixels to the 3D point cloud of the model. Based on this, I utilized PnP (Perspective-n-Point) and RANSAC algorithms to regress the pose and achieve object pose recognition. Additionally, this approach incorporated a deep learning-based refinement algorithm to further improve the pose accuracy based on the initial pose estimation from PnP. Experimental results demonstrated that, compared to other related works, establishing the mapping relationship between the 2D plane and 3D space enabled higher accuracy in pose estimation.

Overall algorithm framework design
Overall algorithm framework design

Creating a virtual dataset using randomly posed 3D models and COCO as the background
Creating a virtual dataset using randomly posed 3D models and COCO as the background

Dataset generation results
Dataset generation results

The principle of UV mapping
The principle of UV mapping

Mapping relationship between UV mapping and surface points (point cloud) of an object
Mapping relationship between UV mapping and surface points (point cloud) of an object

Design of UV map generation network
Design of UV map generation network

Result of UV map generation
Result of UV map generation

Comparison between UV map generation results and calibration images
Comparison between UV map generation results and calibration images

Comparison between generated point cloud results and calibrated point cloud
Comparison between generated point cloud results and calibrated point cloud

The overall approach of using RANSAC+PnP for initial pose regression of objects
The overall approach of using RANSAC+PnP for initial pose regression of objects

Design of a deep learning-based pose regression network
Design of a deep learning-based pose regression network

Pose recognition results
Pose recognition results

Point cloud perception

During my senior year, I primarily focused on point cloud perception in the laboratory at Jilin University. I accomplished the data format conversion from the Livox dataset to the KITTI dataset.

Data format conversion
Data format conversion

Additionally, I successfully trained and implemented the PointPillars model using the Livox dataset for forward inference.

Forward inference and recognition Result1
Forward inference and recognition Result1
Forward inference and recognition Result2
Forward inference and recognition Result2

Through this experience, I gained knowledge of various methods for point cloud processing and became aware of their limitations. It also sparked my interest in exploring the fusion of point cloud and visual perception.

There is significant room for improvement in using purely point cloud-based methods for model recognition.
There is significant room for improvement in using purely point cloud-based methods for model recognition.

pose_estimation