Data-driven approaches for texture and motion

Access to vast amounts of visual information and the increase in computing power are facilitating the emergence of a new class of simple data-driven approaches for image analysis and synthesis. Instead of building an explicit parametric model of a phenomenon, the data-driven techniques rely on the underlying data to serve as its own representation. This dissertation presents work in two different domains, visual texture and human motion, where data-driven approaches have been particularly successful. Part I describes two algorithms for texture synthesis, the problem of synthesizing visual texture (e.g. grass, bricks, pebbles) from an example image. The goal is to produce novel samples of a given texture, which, while not identical, will be perceived by humans as the same texture. The proposed texture synthesis process grows a new image outward from an initial seed, one pixel/patch at a time. A Markov random field model is assumed, and the conditional distribution of a pixel/patch given all its neighbors synthesized so far is estimated by querying the sample image and finding all similar neighborhoods. The degree of randomness is controlled by a single perceptually intuitive parameter. The method aims at preserving as much local structure as possible and produces good results for a wide variety of synthetic and real-world textures. One discussed application of the method is texture transfer, a novel technique that allows texture from one object to be “painted” onto a different object. Part II presents an algorithm for the analysis and synthesis of human motion from video. Instead of reconstructing a 3D model of the human figure (which is hard), the idea is to “explain” a novel motion sequence with bits and pieces from a large collection of stored video data. This simple method can be used for both action recognition as well as motion transfer—synthesizing a novel person imitating the actions of another person (“Do as I Do” synthesis) or performing actions according to the specified action labels (“Do as I Say” synthesis).