View the interpolation network to reproduce the appearance of the material

With the current spread of Internet shopping, buying products online has become possible and common. However, in many cases, because users are only presented with a few photos of the product and cannot actually hold the product, they may not be able to understand the product material.

In this study, we propose looking at interpolation networks to reproduce the appearance of the material. We implemented the U-Net, the existing image transformation network, and the network that uses additional information in the middle layer of the U-Net. The network is trained to produce images from intermediate perspectives from four cameras placed at the corners of a square on a sphere. As a first step to reproducing the appearance of complex materials, we used relatively simple objects with metallic reflections to verify the effectiveness of the method.

By comparing the methods and formats of the training data, we determine that single image output is better for display interpolation than image array output. We also determined that using random viewpoint images is better than using fixed viewpoint images as the training data format for the network. Future research should address three main issues.

First, experiments must be carried out using objects with more complex materials. Only objects with relatively simple materials were used in our experiments. Further experiments using objects with complex optical properties such as transmission, refraction, and internal scattering, are needed to reproduce more common appearances of materials.

Second, the degrees of freedom of position and viewing angle orientation must be expanded. In this study, the camera is placed on a sphere, always facing the center of the sphere, which has two degrees of freedom. If the camera is panned back and forth or oriented in any other direction, interpolation of views with higher degrees of freedom is possible.

Finally, future research should focus on improving accuracy. The images produced in the experiment looked almost natural; however, the output image failed to reproduce some minor details. This study uses the simple mean square error as a loss function. The accuracy can be improved by using a loss function which is more suitable for the purpose of reproducing the appearance of a material. In addition, blurring can be reduced by using a GAN-based network.