Virtual Human In VR

Contents

1. Virtual Reality Overview
2. Virtual Humans in Virtual Environments
3. Modeling of Virtual Humans
4. Animation of Virtual Humans
5. Behavior of Virtual Humans
6. Shared Virtual Environment
7. Summary
8. References

1. Virtual Reality Overview

• What is virtual reality (VR): immersive vs non-immersive, telepresence, augmented reality
• Characterisrics of immersive virtual environments:
• Viewer-centered perspective: head-referenced viewing, free navigation, etc. let you see through your own eyes, you move through virtual the 3D world
• Stereoscopic viewing: enhances the perception of depth and the sense of space
• Multi-sensory: visual, audio, haptic (force feedback), tactile (touch), olfactory
• Realistic interactions: various input and output devices allow for manipulations of virtual objects and communication with the virtual world
• Real-time process: animation, interaction
• VR techniques
• Hardware devices: display, tracking, input, output, computing
• Development software: graphics API, tracking software, integration software, audio, utilities
• VR applications
• Art and entertainment
• Simulation & training
• Scientific visualization
• Engineering & industry
• Medicine - surgery, phobias
• Architecture

2. Virtual Humans in Virtual Environments

• Why virtual human
• Simulation: landscape, ergonomy, emergency
• Education: animated pedagogic agents
• Entertainment: films, games, TV
• Shared virtual environment: virtual teamate
• Evolution and categories
• Avatars
• Representation of the user
• Animation correlated to actual body
• Real-time performace animation
• Requires high-end VR devices (tracking)
• User-guided virtual humans
• A second form of avatar
• User controls motion
• Realistic shape and motion
• Simple device
• Autonomous virtual humans
• Without external intervetion
• Have own goal, rules
• Interactive virtual humans
• Higher level of autonomous virtual human
• Have own behavior, make decisions based on perception systems, memory, and reasoning
• React to environment
• Communicates with other virtual humans and real human
• Virtual crowd
• Based on group behavior, can seek goals
• Flocking motion: following leaders, dispersion and agregation
• Collision avoidance
• Ultimate goals for autonomous virtual humans: should seem to be alive, not just movable
• Shape: no robot, but nice virtual bodies
• Animation: no sliding people, but working people
• Behavior: with perception, mental state, and even intelligence

3. Modeling of Virtul Humans

• Strategy: modeling must include the structure needed for animation
• Face modeling
• Digitally scan a real face using cyberware
• Plaster model: time-consuming, high resolution
• Interactive deformation: local and global deformation, start from template or scratch, real-time
• Interactive texture mapping: interactive texture fitting
• Face cloning: use front & side picture and feature detection technique to reconstruct a cloned face by modifying a generic face model
• Automatic generation: generate a varied geometric models of human faces according to anthropometric statistics for likely face measurements in a population
• Body modeling: multi-layered approach
• Skeleton: hierarycal strcuture - limbs connected with joints of multiple LOF
• Muscles: Deformable grouped volume primitives
• Skin: a spline surface using ray-casting method

4. Animation of Virtual Humans

• Motion control methods
• Predefined animation clips
• Behavior spaces: a library of behavior fragments
• Challenge: how to connect the behavior fragments to ensure visual coherence at        runtime
• Pros and cons: high quality animations, labor intensive to set up,  fixed view point, not flexible
• Generating behavior dynamically
• 3D graphic model of movable body parts and algorithm to generate body motions
• Difficulty: coordinate different body parts to move in concert
• Pros and cons: flexible, but quality hard to achieve
• Degree of control: direct, indirect (guided) and autonomous
• Motion capture and tracking
• For avatars and directly guided virtual humans, need to know the movement of real person as its source of movement
• Facial expression tracking
• Track markers: glue markers on face and reconstruct 3D models from multiple views; need to be discrete and remain in view
• Optical flow: recover image motion parameters corresponding to facial expressions; require high textual detail (high bandwidth)
• Simplified model: track a few feature points/area such as mouth & eye, using anatomy constraints to estimate possible movements; no need of marker, less data transfer, may not be accurate, good for real time
• Body movements tracking
• Physical tracking: sensors on real body, interpolate non-tracked points with constraints by human behavior; need multiple sensors, tracking noise
• Body expression: decompose human activities into action primitives (postures, gestures, and constraints), express action as combination of primitives; higher level of control, need action recognition and reconstruction
• 3D data of movements or interpretated movements: bandwidth, reconstruction
• Animation and deformation
• Skeleton animation
• Direct and inverse kinematics: compute joints angles, positions
• Dynamics: forces, torques, constraints and mass properties
• Body deformation
• Construct a triangle body mesh by connecting cross-section contours of each part, anipulate the cross-section skin contours, interpolate the normals, reduce computaion by limiting the the number of contours to be deformed
• Other deformation methods: multi-resolution mesh morphing, implicit fairing; maybe too complicated to be performed in real time
• Facial animation
• Video-texturing of the face: accurate texture fitting does not work in real time, use simplified head model with attenuated features
• Model-based coding of facial expressions: parameters describing facial expressions are extracted from images; data transfer is reduced
• Lip movement synthesis from speech: extract visual paramters of the lip movement by analyzing audio signals of speech; no need of real person in front of camera
• Multilevel approach: basic motion parameters as minimal perceptible actions (MPAs) at low level, viseme and phoneme at middle level, script of speech and emotions at high level.
• Synchronization of movements from different body parts

5. Behavior of Virtual Humans

• Behavior loop: perception + emotion -> behavior -> action
• Perception: awareness of elements of environment through virtual sensation
• Virtual vision (synthetic vision)
• Actors uses vision for perception of the world and input to behavioural model
• Avoid problems of pattern recognition involved in robotic vision
• Input: description of 3D environment and self position
• Output: 2D array of pixels with: eye-object distance, object id
• Virtual audition
• Actors able to detect sound events
• Able to detect speech in text
• Virtual tactile
• Correspond roughly to collision detection
• Information of the world provided to actor
• Could also be used for self collisions: sensor-body collision test and correction
• Logic state
• Track the current state of the environment
• Pedagogic agents: track problem solving status, students actions and quesitons etc
• Emotion
• Functions
• Psychic and physical reaction of person to perception
• Reaction include body response, facial expression, geustures, or selected behaviors
• Emotion takes place between perceptioin and subsequent reaction
• Different persons can have different emotional response to same perceived event
• Representations
• Emotional state: a normalized value denoting desire to communicate
• Personality matrix: a matrix of state transform possibilities
• Relationship
• Used when there are multiple virtual humans: virtual crowd
• Each actor maintains description of relationship with each other actor
• Represented by a normalized value: good: high value, bad: low value
• Updated according to issue of communication
• Bahavior control
• Rule-based
• E.g. BodyChat, a conversational avatar: maintains knowledge of communicative rules, when receiving an event, can react to the event according to these rules
• Goal-driven
• E.g. virtual crowd, seek goals (interest points), decompose into sub-goals, form a path from source to destination
• Behavior planing
• E.g. PPP Persona, a pedagogical agent: plan a coherent sequence of utterance by searching through alternative sequences until one is found that satisies all coherence constraints
• Adaptive: learn knowledge, past experience

6. Shared Virtual Environments

• Mulitple users at different locations meet in the same virtual world connected by the network
• Client-server system for interface and rendering, or distributed object-oriented model
• Large amounts of global data need to be transfered through network in real time, high bandwith requirements
• Trade-off between accuracy and communication volume, choice also depend on raw data quality, application requirement and number of participants
• With lots of participants, avoid sending informtion to a site when there's little interaction, using filtering and dead-reckoning techniques
• Moving reactive agent behavior from the server to the client, keep the presentation planning capability on the server

7. Summary

• Believable appearance and actions of virtual humans
• Accommodation of virtual humans in collaborated virtual environement
• Intelligence of autonomous virtual humans
• About the Virtual Concert project

Back to contents

8. References

• D. Thalmann, N. Thalmann. Avatars and Autonomous Virtual Humans in Virtual Environments. VR 2000 Tutorial 7 slides, pp. 1-37

• P. Kalra et al. Real-Time Animation of Realistic Virtual Humans. IEEE Computer Graphics and Applications, 1998 Sep./Oct.

• G. Sannier et al. VHD: a System for Directing Real-time Virtual Actors. VR 2000 Tutorial 7, pp. 53.

• S. Musse et al. Crowd Modeling in Collaborative Virtual Environments. VR 2000 Tutorial 7, pp. 65

• L. Emering et al. Body Expression in Virtual Environments. VR 2000 Tutorial 7, pp. 75

• T. K. Capin. Virtual Human Representation and Communication in VLNET Networked Virtul Environment. IEEE Computer Graphics and Applications, Vol. 17, N2, 1997, pp. 42-53.

• H. Seo et. al. Facial Communication in Virtual Enviroments. VR 2000 Tutorial 7, pp. 97

• Desbrun et al. Implicit Fairing of Irregular Meshes using Diffusion and Curvature Flow. Siggraph 1999 Proceedings, pp. 317

• Guskov et al. Multiresolution Signal Processing for Meshes. Siggraph 1999 Proceedings, pp. 325

• Turk and Obrien. Shape Transformation using Variational Implicit Functions. Siggraph 1999 Proceedings, pp. 335

• Lee et al. Multiresolution Mesh Morphing. Siggraph 1999 Proceedings, pp. 343

• Guenter et al. Making Faces. Siggraph 1998 Proceedings, pp. 67

• DeCarlo et al. An Anthropometric Face Model using Variational Techniques. Siggraph 1998 Proceedings, pp. 55

• J. Cassell, H. Vihjamsson. Fully Embodied Conversational Avatars: Making Communicative Behaviors Autonomous. Autonomous Agents and Multi-Agent Systems, 2(1), p. 45-64.

• D. Christianson et al. Declarative Camera Control for Automatic Cinematography. Proceedings of AAAI-96, pp. 148-155.

• W.L. Johnson and J. Rickel. Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments. To appear in International Journal of Artificial Intelligence in Education, 2000.

• P. Karp and S. Feiner. Issues in the Automated Generation of Animated Presentations. Proceedings of Graphics Interface '90, pp. 39-48.

• J. Kolodner and D. Leake. A Tutorial Introduction to Case-Based Reasoning, in Leake, D., Ed. Case-Based Reasoning: Experiences, Lessons, and Future Directions, MIT Press, 1996.