Texture Representation via Analysis and Synthesis with Generative Adversarial Networks

Lin, Jue

doi:https://doi.org/10.21985/n2-atf9-mx42

Work

Texture Representation via Analysis and Synthesis with Generative Adversarial Networks

Public

Download PDF

Download All Files (.zip)

We consider data-driven approaches for universal texture modeling via generative adversarial networks and inversion methods. We investigate the properties of the learned representation spaces and demonstrate that a strong link between texture analysis and synthesis is the key to successful texture modeling. First, we visit the problem of texture synthesis in the context of StyleGAN-2 networks. We present a universal texture synthesis approach that incorporates a novel multiscale texton broadcasting module in the StyleGAN-2 framework. The texton broadcasting module introduces an inductive bias, enabling generation of a broader range of textures, from those with regular structures to completely stochastic ones. For training and evaluation of the proposed approach, we constructed a comprehensive high-resolution dataset, NUUR-Texture500, that captures the diversity of natural textures as well as stochastic variations within each perceptually uniform texture. Experimental results demonstrate that the proposed approach yields significantly better quality textures than the original \mbox{StyleGAN-2}. Second, we adopt StyleGAN-3 for synthesis and demonstrate that it produces diverse textures beyond those represented in the training data. For texture analysis, we propose GAN inversion using a novel latent domain reconstruction consistency criterion for synthesized textures, and iterative refinement with Gramian loss for real textures. For training and testing of the proposed approaches, we compiled a larger and more diverse set of spatially homogeneous textures, ranging from stochastic to regular. We propose perceptual procedures for evaluating network capabilities, exploring the global and local behavior of latent space trajectories, and comparing with existing texture analysis-synthesis techniques. Our results indicate that the proposed framework provides good generalization ability at the cost of some sacrifice in fidelity of texture analysis/synthesis.

Creator