This is the accepted version of the journal article: de Souza, César Roberto; Gaidon, Adrien; Cabon, Yohann; [et al.]. «Generating human action videos by coupling 3D game engines and probabilistic graphical models». International Journal of Computer Vision, Vol. 128, issue 5 (May 2020), p. 1505-1536. 32 pàg. DOI 10.1007/s11263-019-01222-z This version is available at https://ddd.uab.cat/record/274822 under the terms of the license Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical Models César Roberto de Souza · Adrien Gaidon · Yohann Cabon · Naila Murray · Antonio Manuel López Abstract Deep video action recognition mod- cedural Human Action Videos”. PHAV con- els have been highly successful in recent years tains a total of 39, 982 videos, with more than but require large quantities of manually an- 1, 000 examples for each of 35 action cate- notated data, which are expensive and labo- gories. Our video generation approach is not rious to obtain. In this work, we investigate limited to existing motion capture sequences: the generation of synthetic training data for 14 of these 35 categories are procedurally de- video action recognition, as synthetic data have fined synthetic actions. In addition, each video been successfully used to supervise models for is represented with 6 different data modalities, a variety of other computer vision tasks. We including RGB, optical flow and pixel-level se- propose an interpretable parametric genera- mantic labels. These modalities are generated tive model of human action videos that re- almost simultaneously using the Multiple Ren- lies on procedural generation, physics models der Targets feature of modern GPUs. In or- and other components of modern game en- der to leverage PHAV, we introduce a deep gines. With this model we generate a diverse, multi-task (i.e. that considers action classes realistic, and physically plausible dataset of from multiple datasets) representation learn- human action videos, called PHAV for “Pro- ing architecture that is able to simultaneously learn from synthetic and real video datasets, even when their action categories differ. Our César De Souza, Naila Murray, Yohann Cabon NAVER LABS Europe experiments on the UCF-101 and HMDB-51 6 chemin de Maupertuis, 38240 Meylan, France benchmarks suggest that combining our large Tel.: +33 4 76 61 50 50 E-mail: {firstname.lastname}@naverlabs.com set of synthetic videos with small real-world Adrien Gaidon datasets can boost recognition performance. Toyota Research Institute Our approach also significantly outperforms 4440 El Camino Real, Los Altos, CA 94022, USA Tel.: +1 650-673-2365 video representations produced by fine-tuning E-mail: adrien.gaidon@tri.global state-of-the-art unsupervised generative mod- Antonio Manuel López els of videos. Centre de Visió per Computador Universitat Autònoma de Barcelona Edifici O, Cerdanyola del Vallès, Barcelona, Spain Tel.: +34 935 81 18 28 E-mail: antonio@cvc.uab.es Pre-print of the article accepted for publication in the Spe- Keywords procedural generation · human cial Issue on Generating Realistic Visual Data of Human action recognition · synthetic data · physics Behavior of the International Journal of Computer Vision. 2 César Roberto de Souza et al. Fig. 1: Procedurally generated human action videos. Depicted actions include: car hit, walking, kick ball, walking hug. Some are based on variations of existent MoCap sequences for these actions, whereas others have been programatically defined, with the final movement sequences being created on-the-fly through ragdoll physics and simulating the effect of physical interactions. For more example frames and an explanation of the legend icons seen here, cf. Appendix A. 1 Introduction error-prone, raises privacy concerns, and re- quires massive human intervention for every Successful models of human behavior in videos new task. This is often impractical, especially incorporate accurate representations of ap- for videos, or even unfeasible for pixel-wise pearance and motion. These representations ground truth modalities like optical flow or involve either carefully handcrafting features depth. using prior knowledge, e.g., the dense trajec- tories of Wang et al. (2013), or training high- Using synthetic data generated from virtual capacity deep networks with a large amount worlds alleviates these issues. Thanks to mod- of labeled data, e.g., the two-stream network ern modeling, rendering, and simulation soft- of Simonyan and Zisserman (2014). These two ware, virtual worlds allow for the efficient gen- complementary families of approaches have of- eration of vast amounts of controlled and algo- ten been combined to achieve state-of-the-art rithmically labeled data, including for modali- action recognition performance (Wang et al., ties that cannot be labeled by a human. This 2015; De Souza et al., 2016). However, in approach has recently shown great promise for this work we adopt the second family of ap- deep learning across a breadth of computer vi- proaches, which has recently proven highly suc- sion problems, including optical flow (Mayer cessful for action recognition (Wang et al., et al., 2016), depth estimation (Lin et al., 2016b; Carreira and Zisserman, 2017). This 2014), object detection (Vazquez et al., 2014; success is due in no small part to large labeled Peng et al., 2015), pose and viewpoint estima- training sets with crowd-sourced manual anno- tion (Shotton et al., 2011; Su et al., 2015a), tations, e.g., Kinetics (Carreira and Zisserman, tracking (Gaidon et al., 2016), and semantic 2017) and AVA (Gu et al., 2018). However, segmentation (Ros et al., 2016; Richter et al., manual labeling is costly, time-consuming, 2016). Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 3 In this work, we investigate procedural gen- level data were generated efficiently using Mul- eration of synthetic human action videos from tiple Render Targets (MRT). Our dataset, virtual worlds in order to generate training called PHAV for “Procedural Human Action data for human behavior modeling. In partic- Videos” (cf. Figure 1 for example frames), is ular, we focus on action recognition models. publicly available for download1 . Our proce- Procedural generation of such data is an open dural generative model took approximately 2 problem with formidable technical challenges, months of 2 engineers to be programmed and as it requires a full generative model of videos our PHAV dataset 3 days to be generated using with realistic appearance and motion statis- 4 gaming GPUs. tics conditioned on specific action categories. We investigate the use of this data in con- Our experiments suggest that our procedurally junction with the standard UCF-101 (Soomro generated action videos can complement scarce et al., 2012) and HMDB-51 (Kuehne et al., real-world data. 2011) action recognition benchmarks. To al- low for generic use, and as predefined proce- Our first contribution is a parametric gen- dural action categories may differ from un- erative model of human action videos rely- known a priori real-world target ones, we pro- ing on physics, scene composition rules, and pose a multi-task (i.e. that considers action procedural animation techniques like “ragdoll classes from multiple datasets) learning archi- physics” that provide a much stronger prior tecture based on the Temporal Segment Net- than just considering videos as tensors or se- work (TSN) of Wang et al. (2016b). We call quences of frames. We show how to procedu- our model Cool-TSN (cf. Figure 17) in ref- rally generate physically plausible variations of erence to the “cool world” of Vázquez et al. different types of action categories obtained by (2011), as we mix both synthetic and real MoCap datasets, animation blending, physics- samples at the mini-batch level during train- based navigation, or entirely from scratch us- ing. Our experiments show that the genera- ing programmatically defined behaviors. We tion of our synthetic human action videos can use naturalistic actor-centric randomized cam- significantly improve action recognition accu- era paths to film the generated actions with racy, especially with small real-world training care for physical interactions of the camera. sets, in spite of differences in appearance, mo- Furthermore, our manually designed genera- tion, and action categories. Moreover, we out- tive model has interpretable parameters that perform other state-of-the-art generative video allow to either randomly sample or precisely models (Vondrick et al., 2016) when combined control discrete and continuous scene (weather, with the same number of real-world training lighting, environment, time of day, etc), ac- examples. tor, and action variations to generate large This paper extends (De Souza et al., 2017) amounts of diverse, physically plausible, and in two main ways. First, we significantly ex- realistic human action videos. pand our discussion of the generative model we Our second contribution is a quantitative use to control our virtual world and the gen- experimental validation using a modern and eration of synthetic human action videos. Sec- accessible game engine (Unity®Pro) to syn- ond, we describe our use of MRT for generating thesize a dataset of 39, 982 videos, correspond- multiple ground-truths efficiently, rather than ing to more than 1, 000 examples for each simply rendering RGB frames. In addition, we of 35 action categories: 21 grounded in Mo- describe in detail the additional modalities we Cap data, and 14 entirely synthetic ones de- generate, with special attention to semantic fined procedurally. In addition to action la- segmentation and optical flow. bels, this dataset contains pixel-level and per- frame ground-truth modalities, including opti- 1 Dataset and tools are available for download cal flow and semantic segmentation. All pixel- in http://adas.cvc.uab.es/phav/ 4 César Roberto de Souza et al. The rest of the paper is organized as fol- Rather than relying solely on reality-based lows. Section 2 presents a brief review of re- data, synthetic data has been used to train lated work. In Section 3, we present our con- visual models for object detection and recog- trollable virtual world and relevant procedural nition, pose estimation, indoor scene under- generation techniques we use within it. In Sec- standing, and autonomous driving (Marı́n tion 4 we present our probabilistic generative et al., 2010; Vazquez et al., 2014; Xu et al., model used to control our virtual world. In Sec- 2014; Shotton et al., 2011; Papon and Schoeler, tion 5 we show how we use our model to instan- 2015; Peng et al., 2015; Handa et al., 2015; tiate PHAV. In Section 6 we present our Cool- Hattori et al., 2015; Massa et al., 2016; Su TSN deep learning algorithm for action recog- et al., 2015b,a; Handa et al., 2016; Dosovitskiy nition, reporting our quantitative experiments et al., 2017). Haltakov et al. (2013) used a vir- in Section 7. We then discuss possible impli- tual racing circuit to generate different types cations of this research and offer prospects for of pixel-wise ground truth (depth, optical flow future work in Section 8, before finally drawing and class labels). Ros et al. (2016) and Richter our conclusions in Section 9. et al. (2016) relied on game technology to train deep semantic segmentation networks, while Gaidon et al. (2016) used it for multi-object 2 Related work tracking, Shafaei et al. (2016) for depth esti- mation from RGB, and Sizikova1 et al. (2016) Most works on action recognition rely ex- for place recognition. clusively on reality-based datasets. In this work, we compare to UCF-101 and HMDB-51, Several works use synthetic scenarios to two standard action recognition benchmarks evaluate the performance of different feature that are widely used in the literature. These descriptors (Kaneva et al., 2011; Aubry and datasets differ not only in the number of ac- Russell, 2015; Veeravasarapu et al., 2015, 2016) tion categories and videos they contain (cf. Ta- and to train and test optical and/or scene ble 1), but also in the average length of their flow estimation methods (Meister and Konder- clips and their resolution (cf. Table 2), and in mann, 2011; Butler et al., 2012; Onkarappa the different data modalities and ground-truth and Sappa, 2015; Mayer et al., 2016), stereo annotations they provide. Their main charac- algorithms (Haeusler and Kondermann, 2013), teristics are listed below: or trackers (Taylor et al., 2007; Gaidon et al., 2016). They have also been used for learn- – UCF-101 (Soomro et al., 2012) contains ing artificial behaviors such as playing Atari 13,320 video clips distributed over 101 dis- games (Mnih et al., 2013), imitating players in tinct classes. This is the dataset used in the shooter games (Asensio et al., 2014), end-to- THUMOS’13 challenge (Jiang et al., 2013). end driving/navigating (Chen et al., 2015; Zhu – HMDB-51 (Kuehne et al., 2011) contains et al., 2017; Dosovitskiy et al., 2017), learning 6,766 videos distributed over 51 distinct ac- common sense (Vedantam et al., 2015; Zitnick tion categories. Each class in this dataset et al., 2016) or physical intuitions (Lerer et al., contains at least 100 videos, with high 2016). intra-class variability. Finally, virtual worlds have also been While these works have been quite success- explored from an animator’s perspective. ful, they suffer from a number of limitations, Works in computer graphics have investigated such as: the reliance on human-made and error- producing animations from sketches (Guay prone annotations, intensive and often not well et al., 2015b), using physical-based mod- remunerated human labor, and the absence of els to add motion to sketch-based anima- pixel-level ground truth annotations that are tions (Guay et al., 2015a), and creating con- required for pixel-level tasks. strained camera-paths (Galvane et al., 2015). Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 5 Table 1: Statistics for action recognition datasets according to their organization. Number of videos (with aggregate statistics for a single split) Training set Validation set Dataset Classes Total Total Per class (s.d.) Range Total Per class (s.d.) Range UCF-101 101 13,320 9,537 94.42 (13.38) 72-121 3,783 37.45 (5.71) 28-49 HMDB-51 51 6,766 3,570 70.00 (0.00) 70-70 1,530 30.00 (0.00) 30-30 This work 35 39,982 39,982 1142.34 (31.61) 1059-1204 - Averages are per class considering only the first split of each dataset. Table 2: Statistics for action recognition datasets according to their contents. Width Height Frames per second Number of frames Dataset Mean (s.d.) Range Mean (s.d.) Range Mean (s.d.) Range Total Mean (s.d.) Range UCF-101 240.99 (0.24) 320-400 320.02 (1.38) 226-240 25.90 (1.94) 25.00-29.97 2,484,199 186.50 (97.76) 29-1,776 HMDB-51 366.81 (77.61) 176-592 240.00 (0.00) 240-240 30.00 (0.00) 30.00-30.00 639,307 94.488 (68.10) 19-1,063 This work 340.00 (0.00) 340-340 256.00 (0.00) 256-256 30.00 (0.00) 30.00-30.00 5,996,286 149.97 (66.40) 25-291 Averages are among all videos in the dataset (and not per-class as in Table 1). However, due to the formidable complexity tion thanks to our realistic synthetic data, and of realistic animation, video generation, and our multi-task learning formulation to combine scene understanding, these approaches focus real and synthetic data. on basic controlled game environments, mo- An alternative to our procedural generative tions, and action spaces. model that also does not require manual video To the best of our knowledge, ours is the labeling is the unsupervised Video Genera- first work to investigate virtual worlds and tive Adversarial Network (VGAN) of Vondrick game engines to generate synthetic training et al. (2016) and its recent variations (Saito videos for action recognition. Although some of et al., 2017; Tulyakov et al., 2018). Instead the aforementioned related works rely on vir- of leveraging prior structural knowledge about tual characters, their actions are not the focus, physics and human actions, Vondrick et al. not procedurally generated, and often reduced (2016) view videos as tensors of pixel values to just walking. and learn a two-stream GAN on 5, 000 hours The related work of Matikainen et al. of unlabeled Flickr videos. This method fo- (2011) uses MoCap data to induce realistic cuses on tiny videos and capturing scene mo- motion in an “abstract armature” placed in tion assuming a stationary camera. This ar- an empty synthetic environment, generating chitecture can be used for action recognition 2, 000 short 3-second clips at 320 × 240 and in videos when complemented with prediction 30FPS. From these non-photo-realistic clips, layers fine-tuned on labeled videos. Compared handcrafted motion features are selected as rel- to this approach, our proposal allows to work evant and later used to learn action recognition with any state-of-the-art discriminative archi- models for 11 actions in real-world videos. In tecture, as video generation and action recog- contrast, our approach does not just replay Mo- nition are decoupled steps. We can, therefore, Cap, but procedurally generates new action cat- benefit from a strong ImageNet initialization egories – including interactions between per- for both appearance and motion streams as sons, objects and the environment – as well as in (Wang et al., 2016b) and network inflation random physically plausible variations. More- as in (Carreira and Zisserman, 2017). over, we jointly generate and learn deep repre- Moreover, in contrast to (Vondrick et al., sentations of both action appearance and mo- 2016), we can decide what specific actions, sce- 6 César Roberto de Souza et al. Fig. 2: Orthographic view of different world regions during day and night. Time of the day affects lighting and shadows of the world, with urban lights activating at dusk and deactivating at dawn. narios, and camera-motions to generate, en- forcing diversity thanks to our interpretable parametrization. While more recent works such as the Conditional Temporal GAN of Saito et al. (2017) enable certain control over which action class should be generated, they do not offer precise control over every single parame- ter of a scene, and neither are guaranteed to Fig. 3: World location shared between generate the chosen action in case these mod- PHAV and Virtual KITTI (Gaidon et al., els did not receive sufficient training (obtain- 2016), as seen from within the Unity® editor. ing controllable models for video generation has been an area of active research, e.g., Hao et al. (2018); Li et al. (2018); Marwah et al. multiple (and possibly synthetic) sources to (2017)). For these reasons, we show in Section 7 include even more diversity in our generated that, given the same amount of labeled videos, videos. Moreover, unlike approaches based on our model achieves nearly two times the per- GANs, our approach has the unique advantage formance of the unsupervised features shown of being able to generate pixel-perfect ground- in (Vondrick et al., 2016). truth for multiple tasks besides image classifi- cation, as we show in Section 5.1. In general, GANs have found multiple ap- plications for video, including face reenact- ing (Wu et al., 2018), generating time-lapse videos (Xiong et al., 2018), generating artic- ulated motions (Yan et al., 2017), and human motion generation (Yang et al., 2018). From 3 Controllable virtual world those, the works of Yan et al. (2017) and Yang et al. (2018) are able to generate articulated In this section we describe the procedural gen- motions which could be readily integrated into eration techniques we leverage to randomly works based on 3D game engines such as ours. sample diverse yet physically plausible appear- Those works are therefore complimentary to ance and motion variations, both for MoCap- ours, and we show in Section 3.3 how our sys- grounded actions and programmatically de- tem can leverage animation sequences from fined categories. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 7 3.1 Action scene composition In order to generate a human action video, we place a protagonist performing an action in an environment, under particular weather condi- tions at a specific period of the day. There can be one or more background actors in the scene, as well as one or more supporting characters. We film the virtual scene using a parametric camera behavior. The protagonist is the main human model performing the action. For actions involving two or more people, one is chosen to be the protagonist. Background actors can freely walk in the current virtual environment, while sup- porting characters are actors with a secondary Fig. 4: Representation of our Kite camera. role whose performance is necessary in order to complete an action (e.g., hold hands). The action is a human motion belonging to e.g., selected scenes from the Virtual KITTI a predefined semantic category originated from dataset of Gaidon et al. (2016), cf. Figure 3. one or more motion data sources (described in Section 3.3), including predetermined motions from a MoCap dataset, or programmatic ac- 3.2 Camera tions defined using procedural animation tech- niques (Egges et al., 2008; van Welbergen et al., We use a physics-based camera which we call 2009), in particular ragdoll physics. In addi- the Kite camera (cf. Figure 4) to track the pro- tion, we use these techniques to sample physi- tagonist in a scene. This physics-aware cam- cally plausible motion variations (described in era is governed by a rigid body attached by Section 3.4) to increase diversity. a spring to a target position that is, in turn, The environment refers to a region in the attached to the protagonist by another spring. virtual world (cf. Figure 2), which consists By randomly sampling different parameters for of large urban areas, natural environments the drag and weight of the rigid bodies, as (e.g., forests, lakes, and parks), indoor scenes, well as elasticity and length of the springs, we and sports grounds (e.g., a stadium). Each can achieve cameras with a wide range of shot of these environments may contain moving types, 3D transformations, and tracking be- or static background pedestrians or objects – haviors, such as following the actor, following e.g., cars, chairs – with which humans can the actor with a delay, or stationary. physically interact, voluntarily or not. The out- Another parameter controls the direction door weather in the virtual world can be rainy, and strength of an initial impulse that starts overcast, clear, or foggy. The period of the day moving the camera in a random direction. can be dawn, day, dusk, or night. With different rigid body parameters, this im- Similar to Gaidon et al. (2016) and Ros pulse can cause our camera to simulate a hand- et al. (2016), we use a library of pre-made held camera, move in a circular trajectory, or 3D models obtained from the Unity Asset freely bounce around in the scene while filming Store, which includes artist-designed human, the attached protagonist. A representation of object, and texture models, as well as semi- the camera attachment in the virtual world is automatically created realistic environments shown in Figure 5. 8 César Roberto de Souza et al. Fig. 5: In-editor representation of the Kite camera. The camera is a physical object capable of interacting with other objects in the world, which avoids trespassing walls or filming from unfeasible locations. The camera focuses on a point (contact point between orange and blue cords) which is simultaneously attached to the protagonist and to the camera. 3.3 Actions pose to use these sources to form a library of atomic motions to procedurally generate re- Our approach relies on two main existing data alistic action categories. We consider atomic sources for basic human animations. First, we motions as individual movements of a limb use the CMU MoCap database (Carnegie Mel- in a larger animation sequence. For example, lon Graphics Lab, 2016), which contains 2605 atomic motions in a “walk” animation include sequences of 144 subjects divided in 6 broad movements such as rising a left leg, rising a categories, 23 subcategories and further de- right leg, and pendular arm movements. Cre- scribed with a short text. We leverage relevant ating a library of atomic motions enables us to motions from this dataset to be used as a mo- later recombine those atomic actions into new tion source for our procedural generation based higher-level animation sequences, e.g., “hop” on a simple filtering of their textual motion de- or “stagger”. scriptions. Second, we use a large amount of Our PHAV dataset contains 35 different ac- hand-designed realistic motions made by ani- tion classes (cf. Table 3), including 21 sim- mation artists and available on the Unity Asset ple categories present in HMDB-51 and com- Store. posed directly of some of the aforementioned The key insight of our approach is that atomic motions. In addition to these actions, these sources need not necessarily contain mo- we programmatically define 10 action classes tions from predetermined action categories of involving a single actor and 4 action classes interest, neither synthetic nor target real-world involving two person interactions. We create actions (unknown a priori). Instead, we pro- these new synthetic actions by taking atomic Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 9 Base motion Decompose Synthesize Fig. 6: We decompose existing action sequences (left) into atomic motions (middle) and then re- combine them into new animation sequences using procedural animation techniques, like blending and ragdoll physics. This technique can be used to both generate new motion variations for an existing action category, and to synthesize new motion sequences for entirely synthetic categories which do not exist in the data source using simple programmable rules e.g., by tying the ragdoll hands together (right). The physics engine enforces that the performed ragdoll manipulations result in physically plausible animations. Table 3: Action categories included in PHAV. Type Count Actions brush hair, catch, clap, climb stairs, golf, 21 jump, kick ball, push, pick, pour, pull up, sub-HMDB run, shoot ball, shoot bow, shoot gun, sit, stand, swing baseball, throw, walk, wave car hit, crawl, dive floor, flee, hop, leg split, One-person synthetic 10 limp, moonwalk, stagger, surrender walking hug, walk holding hands, walk the Two-people synthetic 4 line, bump into each other motions as a base and using procedural an- 3.4 Physically plausible motion variations imation techniques like blending and ragdoll physics (cf. Section 3.4) to compose them in a physically plausible manner according to We now describe procedural animation tech- simple rules defining each action, such as ty- niques (Egges et al., 2008; van Welbergen et al., ing hands together (e.g., “walk hold hands”, 2009) to randomly generate large amounts of cf. Figure 6), disabling one or more muscles physically plausible and diverse human action (e.g., “crawl”, “limp”), or colliding the protag- videos, far beyond what can be achieved by onist against obstacles (e.g., “car hit”, “bump simply replaying atomic motions from a static into each other”). animation source. 10 César Roberto de Souza et al. Ragdoll physics. A key component of our work Head is the use of ragdoll physics. Ragdoll physics are limited real-time physical simulations that can be used to animate a model (e.g., a human Right upper model) while respecting basic physics prop- arm Left upper erties such as connected joint limits, angular Right lower arm limits, weight and strength. We consider rag- arm dolls with 15 movable body parts (referenced Left lower Chest arm herein as muscles), as illustrated in Figure 7. For each action, we separate those 15 muscles into two disjoint groups: those that are strictly Hips Right necessary for performing the action, and those hand that are complementary (altering their move- ment should not interfere with the semantics Right Left Left of the currently considered action). The rag- upper upper hand doll allows us to introduce variations of dif- leg leg ferent nature in the generated samples. The other modes of variability generation described Right Left in this section will assume that the physical lower lower plausibility of the models is being kept by the leg leg use of ragdoll physics. We use RootMotion’s PuppetMaster2 for implementing and control- Right foot Left foot ling human ragdolls in Unity® Pro. Random perturbations. Inspired by Perlin Fig. 7: Ragdoll configuration with 15 muscles. (1995), we create variations of a given mo- tion by adding random perturbations to mus- cles that should not alter the semantic cate- Action blending. Similarly to modern video gory of the action being performed. Those per- games, we use a blended ragdoll technique to turbations are implemented by adding a rigid constrain the output of a pre-made anima- body to a random subset of the complementary tion to physically plausible motions. In action muscles. Those bodies are set to orbit around blending, we randomly sample a different mo- the muscle’s position in the original animation tion sequence (coming either from the same or skeleton, drifting the movement of the puppet’s from a different action class, which we refer to muscle to its own position in a periodic oscil- as the base motion) and replace the movements lating movement. More detailed references on of current complementary muscles with those how to implement variations of this type can be from this new sequence. We limit the number found in (Perlin, 1995; Egges et al., 2008; Per- of blended sequences in PHAV to be at most lin and Seidman, 2008; van Welbergen et al., two. 2009) and references therein. Muscle weakening. We vary the strength of the Objects. The last physics-based source of vari- avatar performing the action. By reducing its ation is the use of objects. First, we manu- strength, the actor performs an action with ally annotated a subset of the MoCap actions seemingly more difficulty. marking the instants in time where the actor 2 RootMotion’s PuppetMaster is an advanced started or ended the manipulation of an object. active ragdoll physics asset for Unity®. For more Second, we use inverse kinematics to generate details, please see http://root-motion.com plausible programmatic interactions. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 11 Table 4: Overview of key random variables of our generative model of human action videos . Parameter Variable Count Possible values Human Model H 20 models designed by artists Environment E 7 simple, urban, green, middle, lake, stadium, house interior Weather W 4 clear, overcast, rain, fog Period of day D 4 night, dawn, day, dusk Variation V 5 none, muscle perturbation, muscle weakening, action blending, objects where H is a human model, A an action cate- gory, L a video length, B a set of basic mo- tions (from MoCap, manual design, or pro- grammed), V a set of motion variations, C a camera, E an environment, D a period of the day, W a weather condition, and possible val- ues for those parameters are shown in Table 4. Given this definition, a simplified version for Fig. 8: A simplified view of the graphical model our generative model (cf. Figure 8) for an ac- for our generator (cf. Section 4.2 for the mean- tion video X can then be given by: ing of each variable). A complete and more de- P (X) =P (H) P (A) P (L | B) P (B | A) tailed version is shown in Figure 9. P (Θv | V ) P (V | A) P (Θe | E) P (E | A) 4 Generative models for world control (2) P (Θc | C) P (C | A, E) In this section we introduce our interpretable P (Θd | D) P (D) parametric generative model of videos depict- P (Θw | W ) P (W ) ing particular human actions, and show how we where Θw is a random variable on weather- use it to generate our PHAV dataset. We start specific parameters (e.g., intensity of rain, by providing a simplified version of our model clouds, fog), Θc is a random variable on (cf. Figure 8), listing the main variables in our camera-specific parameters (e.g., weights and approach, and giving an overview of how our stiffness for Kite camera springs), Θe is a ran- model is organized. After this brief overview, dom variable on environment-specific param- we show our complete model (cf. Figure 9) eters (e.g., current waypoint, waypoint loca- and describe its multiple components in detail. tions, background pedestrian starting points and destinations), Θd is a random variable on period-specific parameters (e.g., amount of 4.1 Overview sunlight, sun orientation), and Θv is a ran- dom variable on variation-specific parameters We define a human action video as a random (e.g., strength of each muscle, strength of per- variable: turbations, blending muscles). The probabil- ity functions associated with categorical vari- X = hH, A, L, B, V, C, E, D, W i (1) ables (e.g., A) can be either uniform, or config- 12 César Roberto de Souza et al. ured manually to use pre-determined weights. adjusted in order to drive the sample genera- Similarly, probability distributions associated tion. These parameters include the θA parame- with continuous values (e.g., Θc ) are either set ters of a categorical distribution on action cat- using a uniform distribution with finite sup- egories A, the θW for weather conditions W , port, or using triangular distributions with pre- θD for day phases D, θH for model models H, determined support and most likely value. θV for variation types V , and θC for camera behaviors C. Additional parameters include the condi- 4.2 Variables tional probability tables of the dependent vari- ables: a matrix of parameters θAE where each We now proceed to define the complete version row contains the parameters for categorical dis- of our generative model. We start by giving tributions on environments E for each action a more precise definition for its main random category A, the matrix of parameters θAC on variables. Here we focus only on critical vari- camera behaviors C for each action A, the ma- ables that are fundamental in understanding trix of parameters θEC on camera behaviors the orchestration of the different parts of our C for each environment E, and the matrix of generation, whereas all part-specific variables parameters θAB on motions B for each action are shown in Section 4.3. The categorical vari- A. ables that drive most of the procedural gener- Finally, other relevant parameters include ation are: Tmin , Tmax , and Tmod , the minimum, maxi- H : h ∈ {model1 , model2 , . . . , model20 } mum and most likely durations for the gener- A : a ∈ {clap, . . . , bump into each other} ated video. We denote the set of all parameters in our model by θ. B : b ∈ {motion1 , motion2 , . . . , motion862 } V : v ∈ {none, random perturbation, weakening, objects, blend} 4.3 Model C : c ∈ {kite, indoors, closeup, static} The complete interpretable parametric prob- E : e ∈ {urban, stadium, middle, abilistic model used by our generation pro- green, house, lake} cess, given our generation parameters θ, can D : d ∈ {dawn, day, dusk, night} be written as: W : w ∈ {clear, overcast, rain, f og} P (H,A, L, B, V, C, E, D, W | θ) = (3) P1 (D, W | θ) P2 (H | θ) (4) where H is the human model to be used by P3 (A, L, B, V, C, E, W | θ) the protagonist, A is the action category for which the video should be generated, B is the where P1 , P2 and P3 are defined by the proba- motion sequence (e.g., from MoCap, created bilistic graphical models represented on Figure by artists, or programmed) to be used as a base 9a, 9b and 9c, respectively. We use extended upon which motion variations can be applied plate notation (Bishop, 2006) to indicate re- (e.g., blending it with secondary motions), V peating variables, marking parameters (non- is the motion variation to be applied to the variables) using filled rectangles. base motion, C is the camera behavior, E is the environment of the virtual world where the action will take place, D is the day phase, and 4.4 Distributions W is the weather condition. These categorical variables are in turn con- The generation process makes use of four main trolled by a group of parameters that can be families of distributions: categorical, uniform, Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 13 (a) Probabilistic graphical model for P1 (D, W | θ), the (b) Probabilistic graphical model for P2 (H | θ), the sec- first part of our parametric generator (world time and ond part of our parametric generator (human models). weather). (c) Probabilistic graphical model for P3 (A, L, B, V, C, E, W | θ), the third part of our parametric generator (scene and action preparation). Fig. 9: Our complete probabilistic graphical model, divided in three parts. 14 César Roberto de Souza et al. Bernoulli and triangular. We adopt the follow- P (W = clear | θW ) = 1/4 ing three-parameter formulation for the trian- P (W = overcast | θW ) = 1/4 gular distribution: (7) P (W = rain | θW ) = 1/4  P (W = f og | θW ) = 1/4,  0 for x < a, 2(x−a)  for a ≤ x < c,      (b−a)(c−a) ensuring all weather conditions are present.  2 T r(x | a, b, c) = b−a for x = c,   2(b−x) Camera. In addition to the Kite camera, we   (b−a)(b−c) for c < x ≤ b, also included specialized cameras that can be     0 for b < x. enabled only for certain environments (In-  (5) doors), and certain actions (Close-Up). We fixed the parameter θC such that: All distributions are implemented using the open-source Accord.NET Framework3 (De P (C = kite | θC ) = 1/3 Souza, 2014). While we have used mostly uni- P (C = closeup | θC ) = 1/3 (8) form distributions to create the dataset used in P (C = indoors | θC ) = 1/3. our experiments, we have the possibility to bias the generation towards values that are closer to However, we have also fixed θCE and θAC such real-world dataset statistics. that the Indoors camera is only available for the house environment, and that the Close-Up Day phase. As real-world action recognition camera can also be used for the BrushHair ac- datasets are more likely to contain video tion in addition to Kite. recordings captured during daylight, we fixed the parameter θD such that: Environment, human model and variations. We fixed the parameters θE , θH , and θV us- P (D = dawn | θD ) = 1/3 ing equal weights, such that the variables E, P (D = day | θD ) = 1/3 H, and V can have uniform distributions. (6) P (D = dusk | θD ) = 1/3 Base motions. We select a main motion se- P (D = night | θD ) = 0. quence which will be used as a base upon which a variation V is applied (cf. Section 4.2). Base motions are weighted according to the mini- We note that although our system can also gen- mum video length parameter Tmin , where mo- erate night samples, we do not include them tions whose duration is less than Tmin are as- in PHAV at this moment to reflect better the signed weight zero, and others are set to uni- contents of real world datasets. form, such that: Weather. In order to support a wide range of ( 1 if length(b) ≥ Tmin applications of our dataset, we fixed the pa- P (B = b|Tmin ) ∝ . rameter θW such that: 0 otherwise (9) 3 The Accord.NET Framework is a framework This weighting is used to ensure that the mo- for image processing, computer vision, machine tion that will be used as a base is long enough learning, statistics, and general scientific comput- ing in .NET. It is available for most .NET plat- to fill the minimum desired duration for a forms, including Unity®. For more details, see video. We then perform the selection of a mo- http://accord-framework.net tion B given a category A by introducing a Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 15 list of regular expressions associated with each a large number of samples in the borders be- of the action categories. We then compute tween two periods of the day, where the distinc- matches between the textual description of the tion between both phases is blurry, we use dif- motion in its source, e.g., short text descrip- ferent triangular distributions associated with tions by Carnegie Mellon Graphics Lab (2016), each phase, giving a larger probability to hours and these expressions, such that: of interest (sunset, dawn, noon) and smaller ( probabilities to hours at the transitions. We 1 if match(regexa , descb ) therefore define the distribution of the world (θAB )ab = 0 otherwise (10) clock times P (T ) as: ∀a ∈ A, ∀b ∈ B. X P (T = t | D) ∝ P (T = t | D = d) (12) d∈D We then define θAB such that4 : where: P (B = b | A = a, θAB ) ∝ (θAB )a,b . (11) P (T = t | D =dawn) = T r(t |7h, 10h, 9h) In this work, we use 859 motions from MoCap P (T = t | D =day) = T r(t |10h, 16h, 13h) and 3 designed by animation artists. These P (T = t | D =dusk) = T r(t |17h, 20h, 18h) 862 motions then serve as a base upon which the procedurally defined (i.e. composed mo- P (T = t | D =night ) = T r(t |20h, 7h, 0h). tions based on programmable rules, cf. Fig- (13) ure 6) and procedurally generated (i.e. mo- tions whose end result will be determined by Generated video duration. The selection of the the value of other random parameters and clip duration L given the selected motion b is their effects and interactions during the run- performed considering the motion length Lb , time, cf. Section 3.4) are created. In order to the maximum video length Tmin and the de- make the professionally designed motions also sired mode Tmod : searchable by Eq.(10), we also annotate them P (L = l | B = b) = T r(a = Tmin , with small textual descriptions. b = min(Lb , Tmax ), Weather elements. The selected weather W af- c = min(Tmod , Lb )). fects world parameters such as the sun bright- (14) ness, ambient luminosity, and multiple boolean variables that control different aspects of the Actors placement and environment. Each en- world (cf. Figure 9a). The activation of one of vironment E has at most two associated way- these boolean variables (e.g., fog visibility) can point graphs. One graph refers to possible po- influence the activation of others (e.g., clouds) sitions for the protagonist, while an additional according to Bernoulli distributions (p = 0.5). second graph gives possible positions BW G for spawning background actors. Indoor scenes World clock time. The world time is controlled (cf. Figure 10) do not include background ac- depending on D. In order to avoid generating tor graphs. After an environment has been se- lected, a waypoint PW is randomly selected from the graph using a uniform distribution. 4 Please note that a base motion can be as- The protagonist position Pxyz is then set ac- signed to more than one category, and therefore cording to the position of PW . The Sxyz po- columns of this matrix do not necessarily sum up sition of each supporting character, if any, is to one. An example is “car hit”, which could use motions that may belong to almost any other cat- set depending on Pxyz . The position and des- egory (e.g., “run”, “walk”, “clap”) as long as the tinations for the background actors are set de- character gets hit by a car during its execution. pending on BW G . 16 César Roberto de Souza et al. Fig. 10: Examples of indoor (top) and outdoor (bottom) locations in PHAV. Fig. 11: Example generation failure cases. First row: too strong perturbations (small model, brushing hair looks like dancing). Second row: limitation in the physics engine together with ragdoll system and MoCap action can lead to physics violations (passing through a wall). Third row: problems in the automatic configuration of the ragdoll model can result in overconstrained joints and unintended variations. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 17 Camera placement and parameters. After a erated. While this could be improved, our ex- camera has been selected, its position Cxyz periments in Section 7 show that the accuracy and the position Txyz of the target are set de- of neural network models do increase when pending on the position Pxyz of the protag- trained with this data. We also compare our onist. The camera parameters are randomly results to an earlier version of this dataset with sampled using uniform distributions on sensi- an increased level of noise and show it has lit- ble ranges according to the observed behav- tle to no effect in terms of final accuracy in ior in Unity®. The most relevant secondary real-world datasets. variables for the camera are shown in Fig- ure 9c. They include Unity-specific parameters for the camera-target (CTs , CTm ) and target- protagonist springs (T Ps , CTm ) that can be 5 Generating a synthetic action dataset used to control their strength and a minimum distance tolerance zone in which the spring has We validate our approach for synthetic video no effect (remains at rest). In our generator, generation by generating a new dataset for ac- the minimum distance is set to either 0, 1 or tion recognition, such that the data from this 2 meters with uniform probabilities. This set- dataset could be used to complement the train- ting is responsible for a “delay” effect that al- ing set of existing target real-world datasets lows the protagonist to not be always in the in order to obtain action classification mod- center of camera focus (and thus avoiding cre- els which perform better in their respective ating such bias in the data). real-world tasks. In this section we give details about how we have used the aforedescribed Action variations. After a variation mode has model to generate our PHAV dataset. been selected, the generator needs to select a subset of the ragdoll muscles (cf. Figure 7) to In order to create PHAV, we generate be perturbed (random perturbations) or to be videos with lengths between 1 and 10 seconds, replaced with movement from a different mo- at 30 FPS, and resolution of 340 × 256 pixels, tion (action blending). These muscles are se- as this is the same resolution expected by re- lected using a uniform distribution on muscles cent action recognition models such as (Wang that have been marked as non-critical depend- et al., 2016b). We use anti-aliasing, motion ing on the previously selected action category blur, and standard photo-realistic cinematic ef- A. When using weakening, a subset of muscles fects (cf. Figure 12). We have generated 55 will be chosen to be weakened with varying pa- hours of videos, with approximately 6M frames rameters independent of the action category. and at least 1, 000 videos per action category. When using objects, the choice of objects to Our parametric model can generate fully- be used and how they have to be used is also annotated action videos (including depth, dependent on the action category. flow, semantic segmentation, and human pose ground-truths) at 3.6 FPS using one consumer- Failure cases. Although our approach uses grade gaming GPU (NVIDIA GTX 1070). In physics-based procedural animation tech- contrast, the average annotation time for data- niques, unsupervised generation of large annotation methods such as (Richter et al., amounts of random variations with a focus on 2016; Cordts et al., 2016; Brostow et al., 2009) diversity inevitably causes edge cases where are significantly below 0.5 FPS. While those physical models fail. This results in glitches works deal with semantic segmentation (where reminiscent of typical video game bugs (cf. Fig- the cost of annotation is higher than for action ure 11). Using a random 1% sample of our classification), we can generate all modalities dataset, we manually estimated that this cor- for roughly the same cost as RGB using Mul- responds to less than 10% of the videos gen- tiple Render Targets (MRT). 18 César Roberto de Souza et al. Fig. 12: Comparison between raw (left) vs. post-processed (right) RGB frames. Multiple Render Targets. This technique al- to be duplicated for each different modal- lows for a more efficient use of the GPU by ity. This increase in the number of shaders grouping together multiple draw calls of an ob- also increases the chances of designer- and ject into a single call. The standard approach programmer-error when replacing shaders of to generate multiple image modalities for the every object in a scene with shaders that sup- same object is to perform multiple rendering port different ground-truths. passes over the same object with variations of their original shaders that output the data modalities we are interested in (e.g., the se- On the other hand, besides being more effi- mantic segmentation ground-truth for an ob- cient, the use of MRT allows us to concentrate ject would be obtained by replacing the stan- the generation of multiple outputs at the defi- dard texture shader used by each object in the nition of a single shader, removing the hurdle scene with a shader that can output a constant of having to switch shaders during both design- color without any light reflection effects). and run-time. In order to use this technique, we However, replacing shaders for every modify Unity®’s original shader definitions. ground-truth is also an error prone pro- For every shader, we alter the fragment shader cess. Certain objects with complex geometry at their final rendering pass to generate, along- (e.g., tree leaves) require special complex ver- side RGB, all the extra visual modalities we tex and geometry shaders which would need mention next. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 19 5.1 Data modalities Our generator outputs multiple data modali- ties for a single video, which we include in our public release of PHAV (cf. Figure 13). Those data modalities are rendered roughly at the same time using MRT, resulting in a super- linear speedup as the number of simultaneous output data modalities grows. The modalities in our public release include: Rendered RGB Frames. These are the RGB frames that constitute the action video. They are rendered at 340 × 256 resolution and 30 FPS such that they can be directly fed to two- stream style networks. Those frames have been post-processed with 2x Supersampling Anti- Aliasing (SSAA) (Molnar, 1991; Carter, 1997), motion blur (Steiner, 2011), bloom (Steiner, 2011), ambient occlusion (Ritschel et al., 2009; Miller, 1994; Langer and Bülthoff, 2000), screen space reflection (Sousa et al., 2011), color grading (Selan, 2012), and vi- gnette (Zheng et al., 2009). Semantic Segmentation. These are the per- pixel semantic segmentation ground-truths containing the object class label annotations for every pixel in the RGB frame. They are en- coded as sequences of 24-bpp PNG files with the same resolution as the RGB frames. We provide 63 pixel classes (cf. Table 10 in Ap- pendix A), which include the same 14 classes used in Virtual KITTI (Gaidon et al., 2016), classes specific for indoor scenarios, classes for dynamic objects used in every action, and 27 classes depicting body joints and limbs (cf. Fig- Fig. 13: Example frames and data modali- ure 14). ties for a synthetic action (car hit, left) and MoCap-based action (sit, right). From top Instance Segmentation. These are the per- to bottom: Rendered RGB Frames, Semantic pixel instance segmentation ground-truths Segmentation, Instance Segmentation, Depth containing the person identifier encoded as dif- Map, Horizontal Optical Flow, and Vertical ferent colors in a sequence of frames. They Optical Flow. Depth image brightness has been are encoded in exactly the same way as the adjusted in this figure to ensure visibility on semantic segmentation ground-truth explained paper. above. 20 César Roberto de Souza et al. Depth Map. These are depth map ground- truths for each frame. They are represented as a sequence of 16-bit grayscale PNG images with a fixed far plane of 655.35 meters. This en- coding ensures that a pixel intensity of 1 can correspond to a 1cm distance from the camera plane. Optical Flow. These are the ground-truth (for- ward) optical flow fields computed from the current frame to the next frame. We provide separate sequences of frames for the horizon- tal and vertical directions of optical flow rep- resented as sequences of 16-bpp JPEG images with the same resolution as the RGB frames. We provide the forward version of the optical flow field in order to ensure that models based on the Two-Stream Networks of Simonyan and Zisserman (2014) could be readily applicable to our dataset, since this is the optical flow format they have been trained with (forward TV-`1 ). However, this poses a challenge from the gen- eration perspective. In order to generate frame t one must know frame t + 1 ahead of time. In order to achieve this, we store every transfor- mation matrix from all objects in the virtual scene from frame t, and then change all vertex and geometry shaders of all shaders to return both the previous and current positions. Raw RGB Frames. These are the raw RGB frames before any of the post-processing effects mentioned above are applied. This modality is mostly included for completeness, and has not been used in experiments shown in this work. Pose, location and additional information. Al- Fig. 14: Semantic segmentation ground-truth though not an image modality, our genera- for human bodies in PHAV. In order to tor also produces extended metadata for every make our approach scalable, body segments frame. This metadata includes camera param- are determined automatically for every model eters, 3D and 2D bounding boxes, joint loca- through a series of line and distance tests with tions in screen coordinates (pose), and mus- models in a standardized key position. The cle information (including muscular strength, spatial resolution of the segments are deter- body limits and other physical-based annota- mined by the resolution of their meshes. tions) for every person in a frame. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 21 Procedural Video Parameters. We also include Table 5: Statistics of PHAV. the internal state of our generator and virtual world at the beginning of the data generation Statistic Value process of each video. This data can be seen as Total dataset clips 39,982 large, sparse vectors that determine the con- Total dataset frames 5,996,286 tent of a procedurally generated video. These Total dataset duration 2d07h31m vectors contain the values of all possible pa- Average video duration 4.99s rameters in our video generation model, in- Average number of frames 149.97 cluding detailed information about roughly ev- Frames per second 30 ery rigid body, human characters, the world, Video width 340 and otherwise every controllable variable in Video height 256 our virtual scene, including the random seed Average clips per category 1,142.3 which will then influence how those values will Image modalities (streams) 6 evolve during the video execution. As such, these vectors include variables that are dis- crete (e.g., visibility of the clouds), continuous focused on generating videos that would be (e.g., x-axis position of the protagonist), piece- more similar to those in the target datasets. wise continuous (e.g., time of the day), and an- gular (e.g., rotation of the Earth). These vec- 6 Cool Temporal Segment Networks tors can therefore be seen as procedural recipes for each of our generated videos. We propose to demonstrate the usefulness of our PHAV dataset via deep multi-task repre- sentation learning. Our main goal is to learn an 5.2 Statistics end-to-end action recognition model for real- world target categories by combining a few ex- In this section we show and discuss some key amples of labeled real-world videos with a large statistics for the dataset we generate, PHAV. number of procedurally generated videos for A summary of those statistics can be seen in different surrogate categories. Our hypothesis Table 5. Compared to UCF-101 and HMDB- is that, although the synthetic examples dif- 51 (cf. Tables 1 and 2), we provide at least one fer in statistics and tasks, their realism, quan- order of magnitude more videos per categories tity, and diversity can act as a strong prior and than these datasets, supplying about 3× more regularizer against overfitting, towards data- RGB frames in total. Considering that we pro- efficient representation learning that can op- vide 6 different visual data modalities, our re- erate with few manually labeled real videos. lease contains a total of 36K images ready to Figure 17 depicts our learning algorithm in- be used for a variety of tasks. spired by Simonyan and Zisserman (2014), but adapted for the Temporal Segment Networks A detailed view of the number of videos (TSN) of Wang et al. (2016b) with the “cool generated for each action class is presented in worlds” of Vázquez et al. (2011), i.e. mixing Figure 16. As can be seen, the number is higher real and virtual data during training. than 1,000 samples for all categories. We also show the number of videos gen- erated by value of each main random genera- 6.1 Temporal Segment Networks tion variable in Figure 15, demonstrating these histograms reflect the probability values pre- The recent TSN architecture of Wang et al. sented in Section 4.4. We also note that, while (2016b) improves significantly on the original our parametric model is flexible enough to gen- two-stream architecture of Simonyan and Zis- erate a wide range of world variations, we have serman (2014). It processes both RGB frames 22 César Roberto de Souza et al. Fig. 15: Number of videos per parameter value Fig. 16: Plot of the number of videos generated for multiple variables defined in Section 4: for each category in PHAV (cf. Table 3). As weather, environment, phase of the day, mo- can be seen, the number is higher than 1,000 tion variation, and camera behavior. samples for all categories. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 23 Fig. 17: Our “Cool-TSN” deep multi-task learning architecture for action recognition in videos. and stacked optical flow frames using a deeper where F(Tk ; W ) is a function representing a Inception architecture (Szegedy et al., 2015) CNN architecture with weight parameters W with Batch Normalization (Ioffe and Szegedy, operating on short snippet Tk from video seg- 2015) and DropOut Srivastava et al. (2014). ment k. Although it still requires massive labeled train- ing sets, this architecture is more data effi- cient, and therefore more suitable for action 6.2 Multi-task learning in a Cool World recognition in videos. In particular, Wang et al. (2016b) shows that both the appearance and As illustrated in Figure 17, the main differ- motion streams of TSNs can benefit from a ences we introduce with our “Cool-TSN” ar- strong initialization on ImageNet, which is one chitecture are at both ends of the training pro- of the main factors responsible for the high cedure: (i) the mini-batch generation, and (ii) recognition accuracy of TSN. the multi-task prediction and loss layers. Another improvement of TSN is the ex- Cool mixed-source mini-batches. Inspired by plicit use of long-range temporal structure by Vázquez et al. (2011); Ros et al. (2016), jointly processing random short snippets from we build mini-batches containing a mix of a uniform temporal subdivision of a video. real-world videos and synthetic ones. Follow- TSN computes separate predictions for K dif- ing Wang et al. (2016b), we build minibatches ferent temporal segments of a video. These of 256 videos divided in blocks of 32 dispatched partial predictions are then condensed into a across 8 GPUs for efficient parallel training us- video-level decision using a segmental consen- ing MPI5 . Each 32 block contains 10 random sus function G. We use the same parameters synthetic videos and 22 real videos in all our as Wang et al. (2016b): a number of segments experiments, as we observed it roughly bal- K = 3, and the consensus function ances the contribution of the different losses during backpropagation. Note that although K we could use our generated ground truth flow 1 X G= F(Tk ; W ), (15) K 5 github.com/yjxiong/temporal-segment-networks k=1 24 César Roberto de Souza et al. for the PHAV samples in the motion stream, benchmarks. We quantify the impact of multi- we use the same fast optical flow estimation task representation learning with our procedu- algorithm as Wang et al. (2016b), i.e. TV- rally generated PHAV videos on real-world ac- `1 (Zach et al., 2007), for all samples in order to curacy, in particular in the small labeled data fairly estimate the usefulness of our generated regime. We also compare our method with the videos. state of the art on both fully supervised and unsupervised methods. Multi-task prediction and loss layers. Starting from the last feature layer of each stream, we create two separate computation paths, one for 7.1 Real world action recognition datasets target classes from the real-world dataset, and We consider the two most widely used real- another for surrogate categories from the vir- world public benchmarks for human ac- tual world. Each path consists of its own seg- tion recognition in videos. The HMDB-51 mental consensus, fully-connected prediction, (Kuehne et al., 2011) dataset contains 6,849 and softmax loss layers. As a result, we obtain fixed resolution videos clips divided between the following multi-task loss: 51 action categories. The evaluation metric X for this dataset is the average accuracy over L(y, G) = δ{y∈Cz } wz Lz (y, G) three data splits. The UCF-101 (Soomro z∈{real,virtual} et al., 2012; Jiang et al., 2013) dataset contains (16) 13,320 video clips divided among 101 action   classes. Like HMDB-51, its standard evalua- X X tion metric is the average mean accuracy over Lz (y, G) = − yi Gi − log exp Gj  three data splits. Similarly to UCF-101 and i∈Cz j∈Cz HMDB-51, we generate three random splits on (17) our PHAV dataset, with 80% for training and the rest for testing, and report average accu- where z indexes the source dataset (real or racy when evaluating on PHAV. Please refer virtual) of the video, wz is a loss weight (we to Tables 2 and 1 in Section 2 for more details use the relative proportion of z in the mini- about these datasets. batch), Cz denotes the set of action categories for dataset z, and δ{y∈Cz } is the indicator func- tion that returns one when label y belongs to 7.2 Temporal Segment Networks Cz and zero otherwise. We use standard SGD with backpropagation to minimize that objec- In our first experiments (cf. Table 6), we repro- tive, and as every mini-batch contains both real duce the performance of the original TSN in and virtual samples, every iteration is guaran- UCF-101 and HMDB-51 using the same learn- teed to update both shared feature layers and ing parameters as in Wang et al. (2016b). For separate prediction layers in a common descent simplicity, we use neither cross-modality pre- direction. We discuss the setting of the learn- training nor a third warped optical flow stream ing hyper-parameters (e.g., learning rate, iter- like Wang et al. (2016b), as their impact on ations) in the following experimental section. TSN is limited with respect to the substan- tial increase in training time and computa- tional complexity, degrading only by −1.9% on 7 Experiments HMDB-51, and −0.4% on UCF-101. We also estimate performance on In this section, we detail our action recognition PHAV separately, and fine-tune PHAV net- experiments on widely used real-world video works on target datasets. Training and testing Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 25 Table 6: Performance comparison for three target datasets. We show results for the original TSN, our reproduced results, and our two proposed methods for leveraging the extra training data from PHAV. Target Model Spatial (RGB) Temporal (Flow) Full (RGB+Flow) PHAV TSN 65.9 81.5 82.3 UCF-101 Wang et al. (2016b) 85.1 89.7 94.0 UCF-101 TSN 84.2 89.3 93.6 UCF-101 TSN-FT 86.1 89.7 94.1 UCF-101 Cool-TSN 86.3 89.9 94.2 HMDB-51 Wang et al. (2016b) 51.0 64.2 68.5 HMDB-51 TSN 50.4 61.2 66.6 HMDB-51 TSN-FT 51.0 63.0 68.9 HMDB-51 Cool-TSN 53.0 63.9 69.5 Average mean accuracy (mAcc) across all dataset splits. Wang et al. uses TSN with cross-modality training. on PHAV yields an average accuracy of 82.3%, claim that procedural generation of synthetic which is between that of HMDB-51 and human action videos can indeed act as a strong UCF-101. This sanity check confirms that, prior (TSN-FT) and regularizer (Cool-TSN) just like real-world videos, our synthetic videos when learning deep action recognition net- contain both appearance and motion patterns works. that can be captured by TSN to discriminate We further validate our hypothesis by in- between our different procedural categories. vestigating the impact of reducing the num- We use this network to perform fine-tuning ber of real world training videos (and itera- experiments (TSN-FT), using its weights as a tions), with or without the use of PHAV. Our starting point for training TSN on UCF101 results reported in Table 7 and Figure 18 con- and HMDB51 instead of initializing directly firms that reducing training data from the tar- from ImageNet as in (Wang et al., 2016b). We get dataset impacts more severely TSN than discuss learning parameters and results below. Cool-TSN. HMDB displays the largest gaps. We partially attribute this to the smaller size of HMDB and also because some categories of 7.3 Cool Temporal Segment Networks PHAV overlap with some categories of HMDB. Our results show that it is possible to replace In Table 6 we also report results of our Cool- half of HMDB with procedural videos and still TSN multi-task representation learning, (Sec- obtain comparable performance to using the tion 6.2) which additionally uses PHAV to full dataset (65.8 vs. 67.8). In a similar way, train UCF-101 and HMDB-51 models. We stop and although actions differ more, we show that training after 3, 000 iterations for RGB streams reducing UCF-101 to a quarter of its origi- and 20, 000 for flow streams, all other param- nal training set still yields a Cool-TSN model eters as in (Wang et al., 2016b). Our results that rivals competing methods (Wang et al., suggest that leveraging PHAV through either 2016c; Simonyan and Zisserman, 2014; Wang Cool-TSN or TSN-FT yields recognition im- et al., 2015). This shows that our procedural provements for all modalities in all datasets, generative model of videos can indeed be used with advantages in using Cool-TSN especially to augment different small real-world training for the smaller HMDB-51. This provides quan- sets and obtain better recognition accuracy at titative experimental evidence supporting our a lower cost in terms of manual labor. 26 César Roberto de Souza et al. Table 7: TSN and Cool-TSN with different fractions of real-world training data. Fraction of real UCF101 UCF101+PHAV HMDB51 HMDB51+PHAV -world samples (TSN) (Cool-TSN) (TSN) (Cool-TSN) 1% 25.9 27.7 8.1 12.7 5% 68.5 71.5 30.7 37.3 10% 80.9 84.4 44.2 49.7 25% 89.0 90.4 54.8 60.7 50% 92.5 92.7 62.9 65.8 100% 92.8 93.3 67.8 70.1 Mean Accuracy (mAcc) in split 1 of each respective real-world dataset. Table 8: Performance comparison considering ually labeled training data like the Sports-1M an increased number of failure cases (noise). dataset (Karpathy et al., 2014). More impor- tantly, PHAV does not require a specific model Target Noise Spatial Temporal Full to be leveraged and thus can be combined with UCF-101 20% 86.1 90.1 94.2 more recent models from the current and fu- UCF-101 10% 86.3 89.9 94.2 ture state of the art. Our approach also leads to HMDB-51 20% 52.4 64.1 69.5 better performance than the current best gen- HMDB-51 10% 53.0 63.9 69.5 erative video model VGAN (Vondrick et al., 2016) on UCF101, for the same amount of Average mean accuracy across all dataset splits. manually labeled target real-world videos. We note that while VGAN’s more general task is We also evaluate the impact of the fail- quite challenging and different from ours, Von- ure cases described in Section 4. Using an ear- drick et al. (2016) has also explored VGAN lier version of this dataset containing a sim- as a way to learn unsupervised representations ilar amount of videos but an increased level useful for action recognition, thus enabling our of procedural noise, we retrained our models comparison. and compare them in Table 8. Our results show that, even though this kind of noise can result in small performance variations in individual 8 Discussion streams, it has little effect when both streams are combined. Our approach combines standard techniques from computer graphics (notably procedu- ral generation) with deep learning for action 7.4 Comparison with the state of the art recognition. This opens interesting new per- spectives for video modeling and understand- In this section, we compare our model with the ing, including action recognition models that state of the art in action recognition (Table can leverage algorithmic ground truth genera- 9). We separate the current state of the art tion for optical flow, depth, semantic segmen- into works that use one or multiple sources of tation, or pose. In this section, we discuss some training data (such as by pre-training, multi- of these ideas, leaving them as indications for task learning or model transfer). We note that future work. all works that use multiple sources can poten- tially benefit from PHAV without any modi- Integration with GANs. Generative models fications. Our results indicate that our meth- like VGAN (Vondrick et al., 2016) can be com- ods are competitive with the state of the art, bined with our approach by being used for dy- including methods that use much more man- namic background generation, domain adapta- Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 27 Table 9: Comparison against the state of the art in action recognition. UCF-101 HMDB-51 Method %mAcc %mAcc iDT+FV Wang and Schmid (2013) 84.8 57.2 One source iDT+StackFV Peng et al. (2014) - 66.8 iDT+SFV+STP Wang et al. (2016a) 86.0 60.1 iDT+MIFS Lan et al. (2015) 89.1 65.1 VideoDarwin Fernando et al. (2015) - 63.7 2S-CNN Simonyan and Zisserman (2014) 88.0 59.4 TDD Wang et al. (2015) 90.3 63.2 TDD+iDT Wang et al. (2015) 91.5 65.9 C3D+iDT Tran et al. (2015) 90.4 - Multiple sources Actions∼Trans Wang et al. (2016c) 92.0 62.0 2S-Fusion Feichtenhofer et al. (2016) 93.5 69.2 Hybrid-iDT De Souza et al. (2016) 92.5 70.4 3-TSN Wang et al. (2016b) 94.0 68.5 9-TSN Wang et al. (2017) 94.9 - I3D Carreira and Zisserman (2017) 97.9 80.2 CMSN (C3D) Zolfaghari et al. (2017) 91.1 69.7 CMSN (TSN) Zolfaghari et al. (2017) 94.1 - RADCD Zhao et al. (2018) 95.9 - OFF Sun et al. (2018) 96.0 74.2 VGAN Vondrick et al. (2016) 52.1 - Cool-TSN This work 94.2 69.5 Average Mean Accuracy (mAcc) across all dataset splits. tion of synthetic data, or real-to-synthetic style ing and/or doing other activities) with those transfer, e.g., as Gatys et al. (2016). In ad- of a car (entering in a collision with the per- dition, since our parametric model is able to son), followed by the person falling in a physi- leverage MoCap sequences, this opens the pos- cally plausible fashion. However, while atomic sibility of seeding our approach with synthetic actions have been validated as an effective de- sources of motion sequences, e.g., from works composition for the recognition of potentially such as (Yan et al., 2018), while enforcing phys- complex actions (Gaidon et al., 2013), we have ical plausibility (thanks to our use of ragdoll not studied how this approach would scale with physics and a physics engine) and generating the complexity of the actions, notably due to pixel-perfect ground-truth for tasks such as se- the combinatorial nature of complex events. mantic segmentation, instance segmentation, depth estimation, and optical flow. Learning from a real world dataset. While we initialize most of our parameters using uni- Extension to complex activities. Using ragdoll form distributions, it is also possible to have physics and a large enough library of atomic them learned from real world datasets using actions, it is possible to create complex actions attribute predictors, e.g., (Nian et al., 2017) or by hierarchical composition. For instance, our by adapting (Abdulnabi et al., 2015) to video. “Car Hit” action is procedurally defined by We note that θD can be initialized by first composing atomic actions of a person (walk- training a classifier to distinguish between day 28 César Roberto de Souza et al. phases in video (or images) then applying it to all clips (or frames, followed by pooling) of UCF-101 in order to retrieve the histogram of day phases in this dataset. We could then use the relative frequency of this histogram to ini- tialize θD We note that this technique could be used to initialize directly θD , θW , and θC . It could also be used to initialize θE and θH if those variables are further decomposed into more readily interpretable characteristics that could be easily annotated by crowdsourcing, e.g., “presence of grass”, “presence of water”, “filmed indoors”. Then, it becomes possible to learn classifiers for these attributes and es- tablish a mapping between these and the dif- ferent environments and cameras we use. It should also be possible to go further and learn attribute predictors for our virtual world as well, and embed attributes for virtual and real worlds in the same embedding space in order to learn this mapping automatically. Including representative action classes. In our experiments, we have found that certain classes benefit more from the extra virtual data avail- able than others, e.g., “throw”. In the case of UCF-101, the top classes that improved the most were those that related the most with the virtual classes we included in our dataset, e.g., “fall floor” (“dive floor” in PHAV), “throw”, “jump”, “push”, and “shoot ball”. This indicates that one of the crucial factors in improving the performance of classification models for target real-world datasets is indeed to include synthetic data for action classes also present in such datasets. Furthermore, one could also perform a ceteris paribus analysis in order to determine the impact of other param- eters besides the action class (e.g., weather, presence of objects). 9 Conclusion In this work, we have introduced a genera- Fig. 18: TSN and Cool-TSN results for differ- tive model for videos combining probabilistic ent amounts of real-world training data, for graphical models and game engines, and have each separate stream, and for each dataset. used it to instantiate PHAV, a large synthetic Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 29 dataset for action recognition based on a pro- definition ground truth database. Pattern cedural generative model of videos. Although Recognition Letters 30(20) our model does not learn video representations Butler D, Wulff J, Stanley G, Black M (2012) like VGAN, it can generate many diverse train- A naturalistic open source movie for optical ing videos thanks to its grounding in strong flow evaluation. In: ECCV prior physical knowledge about scenes, objects, Carnegie Mellon Graphics Lab (2016) Carnegie lighting, motions, and humans. Mellon University motion capture database We provide quantitative evidence that our Carreira J, Zisserman A (2017) Quo vadis, ac- procedurally generated videos can be used as tion recognition? a new model and the Ki- a simple complement to small training sets netics dataset. In: CVPR of manually labeled real-world videos. Impor- Carter MP (1997) Computer Graphics: Princi- tantly, we show that we do not need to generate ples and practice, vol 22 training videos for particular target categories Chen C, Seff A, Kornhauser A, Xiao J (2015) fixed a priori. Instead, surrogate categories de- DeepDriving: Learning affordance for direct fined procedurally enable efficient multi-task perception in autonomous driving. In: ICCV representation learning for potentially unre- Chen LC, Papandreou G, Kokkinos I, Mur- lated target actions that have few real-world phy K, Yuille AL (2018) Deeplab: Semantic training examples. image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. T-PAMI 40(4) Acknowledgements Antonio M. López acknowl- edges the financial support by the Spanish Cordts M, Omran M, Ramos S, Rehfeld T, TIN2017-88709-R (MINECO/AEI/FEDER, UE). Enzweiler M, Benenson R, Franke U, Roth As CVC/UAB researcher, Antonio also acknowl- S, Schiele B (2016) The Cityscapes dataset edges the Generalitat de Catalunya CERCA Pro- for semantic urban scene understanding. In: gram and its ACCIO agency. Antonio M. López also acknowledges the financial support by ICREA un- CVPR der the ICREA Academia Program. De Souza CR (2014) The Accord.NET Frame- work, a framework for scientific comput- ing in .NET. Available in: http://accord- framework.net References De Souza CR, Gaidon A, Vig E, López AM (2016) Sympathy for the details: Dense tra- Abdulnabi AH, Wang G, Lu J, Jia K (2015) jectories and hybrid classification architec- Multi-task cnn model for attribute pre- tures for action recognition. In: ECCV diction. IEEE Transactions on Multimedia De Souza CR, Gaidon A, Cabon Y, López AM 17(11):1949–1959 (2017) Procedural generation of videos to Asensio JML, Peralta J, Arrabales R, Bedia train deep action recognition networks. In: MG, Cortez P, López A (2014) Artificial CVPR intelligence approaches for the generation Dosovitskiy A, Ros G, Codevilla F, Lopez A, and assessment of believable human-like be- Koltun V (2017) CARLA: An open urban haviour in virtual characters. Expert Sys- driving simulator. In: Proceedings of the 1st tems With Applications 41(16) Annual Conference on Robot Learning Aubry M, Russell B (2015) Understanding Egges A, Kamphuis A, Overmars M (eds) deep features with computer-generated im- (2008) Motion in Games, vol 5277 agery. In: ICCV Feichtenhofer C, Pinz A, Zisserman A (2016) Bishop CM (2006) Pattern Recognition and Convolutional two-stream network fusion for Machine Learning video action recognition. In: CVPR Brostow G, Fauqueur J, Cipolla R (2009) Semantic object classes in video: A high- 30 César Roberto de Souza et al. Fernando B, Gavves E, Oramas MJ, Ghodrati Hattori H, Boddeti VN, Kitani KM, Kanade A, Tuytelaars T (2015) Modeling video evo- T (2015) Learning scene-specific pedestrian lution for action recognition. In: CVPR detectors without real data. In: CVPR Gaidon A, Harchaoui Z, Schmid C (2013) Tem- Ioffe S, Szegedy C (2015) Batch normaliza- poral localization of actions with actoms. T- tion: Accelerating deep network training by PAMI 35(11) reducing internal covariate shift. In: ICML, Gaidon A, Wang Q, Cabon Y, Vig E (2016) vol 37 Virtual worlds as proxy for multi-object Jhuang H, Gall J, Zuffi S, Schmid C, Black MJ tracking analysis. In: CVPR (2013) Towards understanding action recog- Galvane Q, Christie M, Lino C, Ronfard R nition. In: ICCV (2015) Camera-on-rails: Automated compu- Jiang YG, Liu J, Roshan Zamir A, Laptev tation of constrained camera paths. In: SIG- I, Piccardi M, Shah M, Sukthankar R GRAPH (2013) THUMOS Challenge: Action recog- Gatys LA, Ecker AS, Bethge M (2016) Image nition with a large number of classes style transfer using convolutional neural net- Kaneva B, Torralba A, Freeman W (2011) works. In: CVPR Evaluation of image features using a pho- Gu C, Sun C, Ross D, Vondrick C, Pantofaru torealistic virtual world. In: ICCV C, Li Y, Vijayanarasimhan S, Toderici G, Karpathy A, Toderici G, Shetty S, Leung T, Ricco S, Sukthankar R, et al. (2018) Ava: A Sukthankar R, Fei-Fei L (2014) Large-scale video dataset of spatio-temporally localized video classification with convolutional neural atomic visual actions. In: CVPR networks. In: CVPR Guay M, Ronfard R, Gleicher M, Cani MP Kuehne H, Jhuang HH, Garrote-Contreras E, (2015a) Adding dynamics to sketch-based Poggio T, Serre T (2011) HMDB: a large character animations. Sketch-Based Inter- video database for human motion recogni- faces and Modeling tion. In: ICCV Guay M, Ronfard R, Gleicher M, Cani MP Lan Z, Lin M, Li X, Hauptmann AG, Raj B (2015b) Space-time sketching of character (2015) Beyond gaussian pyramid: Multi-skip animation. ACM Transactions on Graphics feature stacking for action recognition. In: 34(4) CVPR Haeusler R, Kondermann D (2013) Synthesiz- Langer MS, Bülthoff HH (2000) Depth discrim- ing real world stereo challenges. In: German ination from shading under diffuse lighting. Conference on Pattern Recognition Perception 29(6) Haltakov V, Unger C, Ilic S (2013) Framework Lerer A, Gross S, Fergus R (2016) Learning for generation of synthetic ground truth data physical intuition of block towers by exam- for driver assistance applications. In: Ger- ple. In: Proceedings of Machine Learning Re- man Conference on Pattern Recognition search, vol 48 Handa A, Patraucean V, Badrinarayanan V, Li Y, Min MR, Shen D, Carlson DE, Carin L Stent S, Cipolla R (2015) SynthCam3D: Se- (2018) Video generation from text. In: AAAI mantic understanding with synthetic indoor Lin TY, Maire M, Belongie S, Hays J, Perona scenes. CoRR arXiv:abs/1505.00171 P, Ramanan D, Dollár P, Zitnick C (2014) Handa A, Patraucean V, Badrinarayanan V, Microsoft COCO: Common objects in con- Stent S, Cipolla R (2016) Understanding real text. In: ECCV world indoor scenes with synthetic data. In: Marı́n J, Vázquez D, Gerónimo D, López AM CVPR (2010) Learning appearance in virtual sce- Hao Z, Huang X, Belongie S (2018) Control- narios for pedestrian detection. In: CVPR lable video generation with sparse trajecto- Marwah T, Mittal G, Balasubramanian VN ries. In: CVPR (2017) Attentive semantic video generation Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 31 using captions. In: ICCV Perlin K, Seidman G (2008) Autonomous dig- Massa F, Russell B, Aubry M (2016) Deep ex- ital actors. In: Motion in Games emplar 2D-3D detection by adapting from Richter S, Vineet V, Roth S, Vladlen K (2016) real to rendered views. In: CVPR Playing for data: Ground truth from com- Matikainen P, Sukthankar R, Hebert M (2011) puter games. In: ECCV Feature seeding for action recognition. In: Ritschel T, Grosch T, Seidel HP (2009) Ap- ICCV proximating dynamic global illumination in Mayer N, Ilg E, Hausser P, Fischer P, Cremers image space. Proceedings of the 2009 sympo- D, Dosovitskiy A, Brox T (2016) A large sium on Interactive 3D graphics and games dataset to train convolutional networks for - I3D ’09 disparity, optical flow, and scene flow esti- Ros G, Sellart L, Materzyska J, Vázquez D, mation. In: CVPR López A (2016) The SYNTHIA dataset: a Meister S, Kondermann D (2011) Real versus large collection of synthetic images for se- realistically rendered scenes for optical flow mantic segmentation of urban scenes. In: evaluation. In: CEMT CVPR Miller G (1994) Efficient algorithms for local Saito M, Matsumoto E, Saito S (2017) Tempo- and global accessibility shading. In: SIG- ral generative adversarial nets with singular GRAPH value clipping. In: ICCV Mnih V, Kavukcuoglu K, Silver D, Graves Selan J (2012) Cinematic color. In: SIG- A, Antonoglou I, Wierstra D, Riedmiller M GRAPH (2013) Playing Atari with deep reinforce- Shafaei A, Little J, Schmidt M (2016) Play and ment learning. In: NIPS Workshops learn: Using video games to train computer Molnar S (1991) Efficient supersampling an- vision models. In: BMVC tialiasing for high-performance architec- Shotton J, Fitzgibbon A, Cook M, Sharp T, tures. Tech. rep., North Carolina University Finocchio M, Moore R, Kipmanand A, Blake at Chapel Hill A (2011) Real-time human pose recogni- Nian F, Li T, Wang Y, Wu X, Ni B, Xu tion in parts from a single depth image. In: C (2017) Learning explicit video attributes CVPR from mid-level representation for video cap- Simonyan K, Zisserman A (2014) Two-stream tioning. Computer Vision and Image Under- convolutional networks for action recogni- standing 163:126–138 tion in videos. In: NIPS Onkarappa N, Sappa A (2015) Synthetic se- Sizikova1 E, Singh VK, Georgescu B, Hal- quences and ground-truth flow field gener- ber M, Ma K, Chen T (2016) Enhanc- ation for algorithm validation. Multimedia ing place recognition using joint intensity - Tools and Applications 74(9) depth analysis and synthetic data. In: ECCV Papon J, Schoeler M (2015) Semantic pose Workshops using deep networks trained on synthetic Soomro K, Zamir AR, Shah M (2012) UCF101: RGB-D. In: ICCV A dataset of 101 human actions classes from Peng X, Zou C, Qiao Y, Peng Q (2014) Action videos in the wild. CoRR arXiv:1212.0402 recognition with stacked fisher vectors. In: Sousa T, Kasyan N, Schulz N (2011) Secrets ECCV of cryengine 3 graphics technology. In: SIG- Peng X, Sun B, Ali K, Saenko K (2015) Learn- GRAPH ing deep object detectors from 3D models. Srivastava N, Hinton G, Krizhevsky A, In: ICCV Sutskever I, Salakhutdinov R (2014) Perlin K (1995) Real time responsive anima- Dropout: A simple way to prevent neu- tion with personality. IEEE Transactions on ral networks from overfitting. Journal on Visualization and Computer Graphics 1(1) Machine Learning Research 15 32 César Roberto de Souza et al. Steiner B (2011) Post processing effects. In- Vondrick C, Pirsiavash H, Torralba A (2016) stitute of Graphics and Algorithms, Vienna Generating videos with scene dynamics. In: University of Technology. Bachelour’s thesis NIPS Su H, Qi C, Yi Y, Guibas L (2015a) Render Wang H, Schmid C (2013) Action recognition for CNN: viewpoint estimation in images us- with improved trajectories. In: ICCV ing CNNs trained with rendered 3D model Wang H, Kläser A, Schmid C, Liu CL (2013) views. In: ICCV Dense trajectories and motion boundary de- Su H, Wang F, Yi Y, Guibas L (2015b) 3D- scriptors for action recognition. IJCV 103 assisted feature synthesis for novel views of Wang H, Oneata D, Verbeek J, Schmid C an object. In: ICCV (2016a) A robust and efficient video rep- Sun S, Kuang Z, Sheng L, Ouyang W, Zhang resentation for action recognition. IJCV W (2018) Optical flow guided feature: A fast 119(3) and robust motion representation for video Wang L, Qiao Y, Tang X (2015) Action action recognition. In: CVPR recognition with trajectory-pooled deep- Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, convolutional descriptors. In: CVPR Anguelov D, Erhan D, Vanhoucke V, Rabi- Wang L, Xiong Y, Wang Z, Qiao Y, Lin D, novich A (2015) Going deeper with convolu- Tang X, van Gool L (2016b) Temporal seg- tions. In: CVPR ment networks: Towards good practices for Taylor G, Chosak A, Brewer P (2007) OVVV: deep action recognition. In: ECCV Using virtual worlds to design and evaluate Wang L, Xiong Y, Wang Z, Qiao Y, Lin surveillance systems. In: CVPR D, Tang X, van Gool L (2017) Temporal Tran D, Bourdev L, Fergus R, Torresani L, segment networks for action recognition in Paluri M (2015) Learning spatiotemporal videos. CoRR arXiv:abs/1705.02953 features with 3D convolutional networks. In: Wang X, Farhadi A, Gupta A (2016c) Actions CVPR ∼ Transformations. In: CVPR Tulyakov S, Liu MY, Yang X, Kautz J (2018) van Welbergen H, van Basten BJH, Egges MoCoGAN: Decomposing motion and con- A, Ruttkay ZM, Overmars MH (2009) Real tent for video generation. In: CVPR time character animation: A trade-off be- Vázquez D, López A, Ponsa D, Marı́n J (2011) tween naturalness and control. In: Proceed- Cool world: domain adaptation of virtual ings of the Eurographics and real worlds for human detection using Wu W, Zhang Y, Li C, Qian C, Loy CC (2018) active learning. In: NIPS Workshops Reenactgan: Learning to reenact faces via Vazquez D, López AM, Marı́n J, Ponsa boundary transfer. In: ECCV D, Gerónimo D (2014) Virtual and real Xiong W, Luo W, Ma L, Liu W, Luo J (2018) world adaptation for pedestrian detection. Learning to generate time-lapse videos using T-PAMI 36(4) multi-stage dynamic generative adversarial Vedantam R, Lin X, Batra T, Zitnick C, Parikh networks. In: CVPR D (2015) Learning common sense through Xu J, Vázquez D, López A, Marı́n J, Ponsa visual abstraction. In: ICCV D (2014) Learning a part-based pedestrian Veeravasarapu V, Hota R, Rothkopf C, detector in a virtual world. T-ITS 15(5) Visvanathan R (2015) Simulations for Yan X, Rastogi A, Villegas R, Sunkavalli K, validation of vision systems. CoRR Shechtman E, Hadap S, Yumer E, Lee H arXiv:1512.01030 (2018) MT-VAE: learning motion transfor- Veeravasarapu V, Rothkopf C, Visvanathan mations to generate multimodal human dy- R (2016) Model-driven simulations for namics. In: ECCV, vol 11209 deep convolutional neural networks. CoRR Yan Y, Xu J, Ni B, Zhang W, Yang X (2017) arXiv:1605.09582 Skeleton-aided articulated motion genera- Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 33 tion. In: ACM-MM Yang C, Wang Z, Zhu X, Huang C, Shi J, Lin D (2018) Pose guided human video generation. In: ECCV, vol 11214 Zach C, Pock T, Bischof H (2007) A dual- ity based approach for realtime tv-l1 opti- cal flow. In: Proceedings of the 29th DAGM Conference on Pattern Recognition Zhao Y, Xiong Y, Lin D (2018) Recognize actions by disentangling components of dy- namics. In: CVPR Zheng Y, Lin S, Kambhamettu C, Yu J, Kang SB (2009) Single-image vignetting correc- tion. In: T-PAMI, vol 31 Zhou B, Zhao H, Puig X, Fidler S, Barriuso A, Torralba A (2017) Scene parsing through ADE20K dataset. In: CVPR Zhu Y, Mottaghi R, Kolve E, Lim JJ, Gupta A, Fei-Fei L, Farhadi A (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: ICRA Zitnick C, Vedantam R, Parikh D (2016) Adopting abstract images for semantic scene understanding. T-PAMI 38(4) Zolfaghari M, Oliveira GL, Sedaghat N, Brox T (2017) Chained multi-stream networks ex- ploiting pose, motion, and appearance for action classification and detection. In: ICCV 34 César Roberto de Souza et al. A Appendix In this appendix, we include random frames (Fig- ures 20, 21, 22, 23, and 24) for a subset of the ac- tion categories in PHAV, followed by a table of pixel colors (Table 10) used in our semantic segmentation ground-truth. The frames below show the effect of different variables and motion variations being used (cf. Ta- ble 4). Each frame below is marked with a label in- dicating the value for different variables during the execution of the video, using the legend shown in Figure 19. Environment Phase of the day Variations Urban Day Dawn None Stadium Night Dusk Action blending Lake Weather Muscle weakening Indoors (house) Clear Rainy Random perturbation Green City Cloudy Foggy Objects Middle City Human models Fig. 19: Legend for synthetic action video variations to be used in Figs. 20, 21, 22, 23, and 24. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 35 Fig. 20: Changing environments. Top: kick ball, bottom: synthetic car hit. 36 César Roberto de Souza et al. Fig. 21: Changing phases of the day. Top: run, bottom: golf. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 37 Fig. 22: Changing weather. Top: walk, bottom: kick ball. 38 César Roberto de Souza et al. Fig. 23: Changing motion variations. Top: kick ball, bottom: synthetic car hit. Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical ... 39 Fig. 24: Changing human models. Top: walk, bottom: golf. 40 César Roberto de Souza et al. Table 10: Pixel-wise object-level classes in PHAV. Group Pixel class R G B Group Pixel class R G B Virtual KITTI (Gaidon et al., 2016) Road 100 60 100 Head 220 20 60 CityScapes (Cordts et al., 2016) Building 140 140 140 RightUpperArm 255 255 26 Pole 255 130 0 RightLowerArm 255 215 0 TrafficLight 200 200 0 RightHand 255 140 0 TrafficSign 255 255 0 LeftUpperArm 60 179 113 Vegetation 90 240 0 LeftLowerArm 135 206 235 Parts Terrain 210 0 200 LeftHand 100 149 237 Sky 90 200 255 Chest 248 248 255 Car 255 127 80 RightUpperLeg 102 51 153 Truck 160 60 60 RightLowerLeg 164 89 58 Bus 0 139 139 RightFoot 220 173 116 Misc 80 80 80 LeftUpperLeg 0 0 139 Human Tree 0 199 0 LeftLowerLeg 255 182 193 LeftFoot 255 239 213 Ceiling 240 230 140 Neck 152 251 152 Floor 0 191 255 LeftShoulder 47 79 79 Chair 72 61 139 RightShoulder 85 107 47 Table 255 250 205 LeftElbow 25 25 112 Bed 205 92 92 RightElbow 128 0 0 Lamp 160 82 45 ADE20k (Zhou et al., 2017) LeftWrist 0 255 255 Joints Sofa 128 0 128 Indoors RightWrist 238 130 238 Window 0 128 0 LeftHip 147 112 219 Door 127 255 212 RightHip 143 188 139 Stairs 219 112 147 LeftKnee 102 0 102 Curtain 230 230 250 RightKnee 69 33 84 Fireplace 233 150 122 LeftAnkle 50 205 50 Shelf 153 50 204 RightAnkle 255 105 180 Bench 245 222 179 Screen 218 165 32 Fridge 255 255 240 Ball 178 34 34 Interaction Baseball Bat 210 105 30 objects Gun 255 248 220 Golf Club 173 255 47 Hair Brush 224 255 255 PHAV-only Bow 95 158 160 Pixel-wise object-level classes in PHAV. Some of the classes have been derived from semantic segmentation labels present in other datasets. These include: CityScapes (Cordts et al., 2016), mostly for outdoor object classes; Virtual KITTI (Gaidon et al., 2016), which contains a subset of the class labels in CityScapes; and ADE20k (Zhou et al., 2017), mostly for indoor object classes. The human body has been segmented in 14 parts and 13 joints, for a total of 27 segments. We note that our chosen separation can be combined to recover part separations used in PASCAL-Part (Chen et al., 2018) and J-HMDB (Jhuang et al., 2013) datasets.