Deep Photometric Stereo

Deep learning approaches have shown their potential in solving the highly nonlinear problem for photometric stereo, but the main challenge preventing their practical application in process metrology lies in the difficulties in the generation of a comprehensive dataset for training the deep learning model.

This project proposes a new Deep-learning based Point-light Photometric Stereo method, DPPS, which utilizes a multi-channel deep convolutional neural network (CNN) to achieve end-to-end prediction for both the surface normal and height maps in a semi-calibrated fashion (Yang et al., 2023). The key contribution is a new dataset generation method combining both physics-based and data-driven approaches, which minimizes the training cost and enables DPPS to handle reflective metal surfaces with unknown surface roughness.

We propsoed a new coarse-to-fine network to improve the performance by exploiting the relationship between initial normal and depth predictions. Furthermore, the pixel-wise confidence associated with predictions is also estimated without requiring the ground truth, making a contribution to enhancing both performance and practicality. The experimental results on our synthetic dataset and real samples demonstrate the effectiveness of the proposed method on both normal/depth and confidence estimation (Zhang et al., 2024)

Codes are released on GitHub.

2024

Confidence-Aware Photometric Stereo Networks Enabling End-to-End Normal and Depth Estimation for Smart Metrology

Yahui Zhang , Ru Yang , and Ping Guo

IEEE/ASME Transactions on Mechatronics, 2024

Abstract PDF Video Code

The acquisition of geometric 3-D information is crucial for ensuring quality standards and monitoring procedures in various manufacturing applications. Photometric stereo is an established technique in computer vision to recover 3-D surfaces of objects. However, existing photometric stereo methods mainly focus on normal estimation of objects, without considering the depth estimation. On the other hand, current methods tend to prioritize accuracy while overlooking the confidence of predictions, which holds valuable information within the industry. In this article, we propose a deep learning-based photometric stereo system, consisting of hardware implementation, dataset generation, and algorithm design, to reconstruct 3-D information of physical objects, represented by normal and depth maps. In terms of the proposed algorithm, a coarse-to-fine network is introduced to improve the performance by exploiting the relationship between initial normal and depth predictions. Furthermore, the pixel-wise confidence associated with predictions is also estimated without requiring the ground truth, making a contribution to enhancing both performance and practicality. The experimental results on our synthetic dataset and real samples demonstrate the effectiveness of the proposed method on both normal/depth and confidence estimation.

2023

DPPS: A deep-learning based point-light photometric stereo method for 3D reconstruction of metallic surfaces

Ru Yang , Yaoke Wang , Shuheng Liao , and Ping Guo

Measurement, 2023

Abstract PDF Video Code

Three-dimensional (3D) measurement provides essential geometric information for quality control and process monitoring in many manufacturing applications. Photometric stereo is one of the potential solutions for in-process metrology and active geometry compensation, which takes multiple images of an object under different illuminations as inputs and recovers its surface normal map based on a reflectance model. Deep learning approaches have shown their potential in solving the highly nonlinear problem for photometric stereo, but the main challenge preventing their practical application in process metrology lies in the difficulties in the generation of a comprehensive dataset for training the deep learning model. This paper presents a new Deep-learning based Point-light Photometric Stereo method, DPPS, which utilizes a multi-channel deep convolutional neural network (CNN) to achieve end-to-end prediction for both the surface normal and height maps in a semi-calibrated fashion. The key contribution is a new dataset generation method combining both physics-based and data-driven approaches, which minimizes the training cost and enables DPPS to handle reflective metal surfaces with unknown surface roughness. Even trained only with fully synthetic and high-fidelity dataset, our DPPS surpasses the state-of-the-art with an accuracy better than 0.15 cm over a 10 cm × 10 cm area and its real-life experimental results are on par with commercial 3D scanners. The demonstrated results provide guidance on improving the generalizability and robustness of deep-learning based computer vision metrology with minimized training cost as well as show the potential for in-process 3D metrology in advanced manufacturing processes.

References

2024

2023