Generative models can produce realistic images from random vectors by training with an adversarial loss. A recent paper published on arXiv.org shows that, by imposing known physical facts about images, the distribution of images produced by a StyleGAN can be significantly enriched.
Researchers show how to generate images that are unlike those in the training set but still realistic. They rely on the fact that the same scene can be lit in different ways to produce images with strong lighting effects not included in the training set. Similarly, it is possible to produce images where surface colors have changed, but layout and lighting have not.
The evaluation confirms that the proposed method can generate all images that a vanilla StyleGAN can. At the same time, a vanilla StyleGAN cannot generate many of the images the proposed method can.
StyleGAN generates novel images of a scene from latent codes which are impressively disentangled. But StyleGAN generates images that are “like” its training set. This paper shows how to use simple physical properties of images to enrich StyleGAN’s generation capacity. We use an intrinsic image method to decompose an image, then search the latent space of a pretrained StyleGAN to find novel directions that fix one component (say, albedo) and vary another (say, shading). Therefore, we can change the lighting of a complex scene without changing the scene layout, object colors, and shapes. Or we can change the colors of objects without changing shading intensity or their scene layout. Our experiments suggest the proposed method, StyLitGAN, can add and remove luminaires in the scene and generate images with realistic lighting effects — cast shadows, soft shadows, inter-reflections, glossy effects — requiring no labeled paired relighting data or any other geometric supervision. Qualitative evaluation confirms that our generated images are realistic and that we can change or fix components at will. Quantitative evaluation shows that pre-trained StyleGAN could not produce the images StyLitGAN produces; we can automatically generate realistic out-of-distribution images, and so can significantly enrich the range of images StyleGAN can produce.
Research article: Bhattad, A. and Forsyth, D. A., “Enriching StyleGAN with Illumination Physics”, 2022. Link: https://arxiv.org/abs/2205.10351