Construction and Presentation of a Virtual Environment Using
Panoramic Stereo Images of a Real Scene and Computer Graphics Models
Jun Shimamura†, Haruo Takemura‡, Naokazu Yokoya‡, Kazumasa Yamazawa‡
†NTT Cyber Space Laboratory, ‡Nara Institute of Science and Technology JAPAN
Background
Approach
Construction of a large-scale virtualized environment
ex. urban area, natural scenes, …
Requirements in constructing an
immersive virtualized environment
 Full panoramic representation
 Depth information from the real world
 Digitizing and representing dynamic events
• Uses a video-rate omni-directional stereo imaging sensor
• Constructs a full panoramic 3-D model using a cylindrical
panoramic stereo image sequence
• Constructs a prototype system of presenting
a mixed environment ( virtualized, CG )
Omni-directional stereo imaging sensor
virtual center of lens
Composed of twelve CCD cameras and two hexagonal pyramidal mirrors
lens
center
upper
component
• Single viewpoint constraint
• High-resolution image acquisition
• Omni-directional stereoscopic imaging at video-rate
pyramidal
mirror
lower
component
camera
front view
top view
Geometry
Appearance
Generation of panoramic stereo images
A pair of
panoramic images
i. Elimination of geometric distortion in images,
ii. Color adjustment of camera images,
iii. Concatenation of six images for completing
upper and lower omni-directional images of a stereo pair
3006 x 330 pixels
Virtualizing a dynamic real scene
• Layered representation of dynamic real scene
Static scene image generation
• A panoramic image of a static scene is generated
by applying a temporal mode filter to a panoramic
image sequence in a time interval
panoramic image
sequence including
dynamic event
Extraction of moving objects
• Moving objects are extracted by subtracting consecutive image
frames in a sequence
subtracting
static image
majority
filtering
• Depth estimation from static panoramic stereo images and moving objects
Extracted moving object regions
• Stereo matching used to estimate depths of static scene and moving objects
upper panoramic image
• Generation of 3-D model
epipolar line
Apply Delaunay’s triangulation
lower panoramic image
world cylindrical coordinate
Bird’s-eye view of
texture-mapped 3-D static scene model
Panoramic depth map of static scene
Prototype system
Immersive mixed reality system
i. Has a 330-degree cylindrical screen
ii. Merges virtual objects with a virtualized real world scene
iii. Superimposes dynamic event layers onto static scene layer
User’s appearance in mixed environment
using CYLINDRA system
Frame rate:
13 frames/sec.
i. Hardware configuration of immersive mixed reality system
ii. Mixed environment observed from
different viewpoints
6144 x 768 pixels resolution
iii. Superimposing dynamic event layers
onto a static scene layer
Background:
13,400 polygons
CG tree:
41,340 polygons
Total:
54,740 polygons
Original
viewpoint
New higher
viewpoint