Research Plans

Ying Feng, May 2001

Augmented Reality: AR is a technology in which a user's view of the real world is enhanced with additional information generated from a computer model.

Requirements: precise calibration, registration of sensors and objects in the scene, detailed understanding of the scene

As for illumination: To blend the real world seamlessly with the virtual world, we must solve the problems of common viewing parameters, common visibility and common illumination

Ideally, estimate the parameters of the light sources in the real world: shape, size, direction, position, intensity, color
An accurate estimate is very hard, use simulations for light sources so virtual objects illuminated with the simulated lights will appear as if being illuminated from real world light sources.
Simulate interactions between real & virtaul objects without global illumination: shadow casting, transparencies

Reflectance map: assume known illumination and surface reflectivity, estimate 3D shape from image shading by solving partial differential equations
Assume no knowledge of imaging geometry, deduce light direction and surface orientation by approximating the solution with least square methods and other qualitative analysis

Reshading: illuminate current objects under novel lighting conditions: find the parameters of the lighting and reflectance models that best explain measured values in various pictures, and use these to reshading objects under novel illumination conditions

For lighting calibration in AR systems (in contrast to shape from shading for image processing)

We have the advantage to use objects with known shapes and reflectance models.
Assuming the environment doesn't change much, calibration only needs to be done during system initialization (not in real time)
However, the lighting model we use should not be too complicated, as we need to illuminate virtual objects in real time.

Simulate various lighting situations as the linear sum of multiple point lights (or parallel lights) evenly distributed on a sphere, with different weights as their intensities.
Consider only diffuse lights and reflections, use a sphere with lambertian surface with gray color, assume albedo is known.
Sample light intensities at various points on the sphere, calculate the normals, obtain an array of linear eqations in terms of the intensity weights.
Find the least square solutions to the linear equations.

Will the aggregation of discrete point lights simulate extended lights or spot lights with various shape and size and other properties?
Might need to use a densely distributed model for better approximation, but then the calibration and illuminatino computation will be more complicated,

Impossible to find a perfect sphere with perfect lambertian surface in real world.
The albedo itself needs to be calibrated in advance.

Too sparse: won't reflect the rapid change when the reflectance function is highly sensitive to incident direction.
Too dense: noisy surface, overfitting problem.
Good only for nearly diffuse reflectors.
Dense samples where surface is perpendicular to lighting direction, sparse samples where they are parallel - not known a priori.

Simulated light sources may not cast shadows correctly, which depends on the original position and direction of the real lights.

Consider more complicated models under specific situations (instead of a general model for all)

Indoor situations: mostly extended or spot lights, usually from top, within a certain distance -- simulate with an array of spot lights from above, add attenuation and distance factors into the equations.
Outdoor situation: mostly sunlight, skylight or light from environment -- not good to simulate with discrete point lights, should build models for the sun and sky respectively.
Make calibration interactive, let user indicate/adjust light directions when they are obvious.

One image can only show shading facing the viewer, can not estimate lighting from the back
Need at least a front and a back picture, maybe even one on each side.

Specular reflection is very useful for highlight detection, which gives a direct hint on the light direction.
Hint from the light probe approach: photograph a spherical first-surface mirror placed near the desired location of the synthetic object, obtain the radiance map and trace the ray from camera back to pixel and to light source.
Specular reflection is view-dependent, need to consider camera parameters.

Responsive Information Architects (RIA): engage users in multimodal interaction and respond to their information requests by presenting them with an automatically generated multimedia tour of relevant information.

My part: visual representation and interaction for both the 3D objects and relevant information

Semantic representation of the graphics
According to the current query, retrieve best matches from previous generated graphics and adapt them to current needs.
Involve semantic similarity comparison, information visualization, and automatic 3D graphics generation.

Combine AR and IV: augment real world with IV
RIA long term plan will involve both visual illustration objects and visual actors (avatars)