Light field – models, processing and compression
Motivation
Conventional image and video capture has proved to be powerful components in numerous applications throughout the years, providing information about how objects in a scene are related in space, time, and colour. Such conventional 2D image captured by a digital camera can be described as sampling of the 7-dimensional plenoptic function that theoretically describe all light reflected of a scene. Advancement in computational power, image sensor resolution and optical system miniaturization in recent years have enabled new image capture systems that are able capture both intensity and direction. Having such information enables new possibilities of post-processing and image enhancements but also require new formats and methods to compress the extensive amount of data.
This tutorial will present the fundamentals of light field, its representation, and the consequences of capturing it by several cameras or a special plenoptic camera. Models describing the light field will be presented that enables optimized selections and prediction of capabilities for different light field systems. The tutorial will also portray fundamental processing of light field data such as interpolation and reconstruction of images with different depth of field, super-resolution and alike. Finally, the tutorial will demonstrate how the light field data can be efficiently compressed by using the high redundancy of the data. Total time of tutorial 3 hours.
In particularly the tutorial will consider the following.
Light field fundamentals and models
- Multiple Pin-hole cameras and plenoptic camera
- Light ray geometrical optics model
- Ray-phase space and Fourier analysis for cameras and displays
- Optimal sampling and antialiasing filter
- SPC-model description and application to plenoptic cameras
Light field processing
- Light field interpolation and reconstruction
- Re-focusing and super-resolution
- Depth extraction
- Sparsification of light field data
Light field image and video compression
- Basics of compression, hybrid coders
- H.264 and HEVC
- Approaches to compress light field data
About the presenters
Robert Bregović received the Dipl.-Ing. and M.Sc. degrees in electrical engineering from the Faculty of Electrical Engineering and Computing, University of Zagreb, Zagreb, Croatia, in 1994 and 1998, respectively, and the D.Sc.(Tech.) (Hons.) degree in information technology from the Tampere University of Technology, Tampere, Finland, in 2003. He was an Assistant with the Department of Electronic Systems and Information Processing, Faculty of Electrical Engineering and Computing, University of Zagreb, Zagreb, Croatia, from 1994 to 1998. In 1999, he was a Visiting Researcher with the Tampere International Center for Signal Processing, Department of Signal Processing, Tampere University of Technology, where he was a Researcher from 2000 to 2003, and he has been a Post-Doctoral Researcher with the Department of Signal Processing since 2003. He has written more than 60 international journal and conference articles. He taught various courses in the area of digital signal processing (e.g., linear filtering and multirate signal processing) and gave tutorials on topics related to visualization of 3-D content at the International Conference on Multimedia and Expo 2013 and International Conference on Visual Communications and Image Processing 2014. His current research interests include the design and implementation of digital filters and filterbanks and topics related to acquisition, processing/modeling, and visualization of 3-D content.
Atanas Gotchev received the M.Sc. degrees in radio and television engineering and applied mathematics and the Ph.D. degree in telecommunications from the Technical University of Sofia, Sofia, Bulgaria, in 1990, 1992, and 1996, respectively, and the D.Sc.(Tech.) degree in information technologies from the Tampere University of Technology, Tampere, Finland, in 2003. He has held research and teaching positions with the Bulgarian Academy of Sciences, Sofia, and Technical University of Sofia. He is currently an Associate Professor (Tenure Track) with the Department of Signal Processing, Tampere University of Technology. His research interests include sampling and interpolation theory, and spline and spectral methods with applications to multidimensional signal analysis. His recent work concentrates on algorithms for multicamera and multisensor 3-D scene capture, transform-domain light field reconstruction, and Fourier analysis of 3-D displays.
Roger Olsson received the M.Sc. degree in Electrical Engineering and the Ph.D. degree in Telecommunication from Mid Sweden University, Sweden, in 1998 and 2010 respectively. He worked in the video compression and distribution industry 1997-2000. He again joined Mid Sweden University as a junior lecturer 2000-2004 where he taught courses in telecommunication, signals- and systems, and signal- and image processing. Since 2010 he is employed as a researcher at Mid Sweden University where his research interest includes plenoptic image capture, processing, and compression; plenoptic system modelling; and depth map capture and processing. www.miun.se/en/personnel/rogerolsson
Mårten Sjöström received the M.Sc. degree in electrical engineering and applied physics from Linköping University, Sweden, in 1992, the Licentiate of Technology degree in signal processing from KTH, Stockholm, Sweden, in 1998, and the Ph.D. degree in modeling of nonlinear systems from EPFL, Lausanne, Switzerland, in 2001. He worked as an Electrical Engineer at ABB, Sweden, from 19931994, was a fellow at CERN from 19941996, and a Ph.D.-student at EPFL, Lausanne, Switzerland during 19972001. In 2001, he joined Mid Sweden University and was appointed Associate Professor and Full Professor in Signal Processing in 2008 and 2013, respectively. He is the head of the subject Computer and System Sciences at Mid Sweden University since 2013. He founded the Realistic 3D research group in 2007. His current research interests are within multidimensional signal processing and imaging, as well as system modelling and identification. www.miun.se/en/personnel/martensjostrom
360° Viewable Digital Holographic Display
Motivation
In the Introduction, we briefly review basic principles of three-dimensional(3D) human vision and motivations for 3D display and imaging technology development. Due to inherent limitations of stereoscopic display technologies, though they are very much mature and available as commercial products, many glasses-free 3D technologies have been developed and holography is regarded as the ultimate technology which can provide naturalness and perfect 3D sensation to human.
Basic concept, brief history and major characteristics of hologram is presented, followed by comparison of analog and digital holography. Whereas analog holography is nothing more than a 3D photography, digital holography enables quite wide range of applications including 3D video communications and tele-presence. Especially, we allot some time to explain various pseudo holograms, which are quoted so frequently in the media and many people confuse with real holograms.
Then technical issues of digital holographic display will be presented as well as approaches on previous display implementations. Concept of space bandwidth product(SBP) and the fact that it is one of major challenges to be overcome for practical holographic display implementations will be explained. Key technical components such as spatial light modulator(SLM) and computer generated hologram(CGH) will be treated in some depth.
Most of previous implementations of holographic displays are “TV (flat panel based)”-type, which we can watch only from front viewpoints with a narrow range of viewing angle and has a small screen size. These limited viewing characteristics are an inherent problem caused by the SBP limitation of the SLM. Perceived quality of reconstructed 3D image of those displays seems yet far from general expectation.
We will present our novel implementation of 360° viewable holographic display. The system overcomes the SBP problem by adopting several multiplexing techniques which are applied on wavelength, time, and space divisions. These multiplexing techniques for achieving full-color, 360° hologram generation, and screen size scaling will be discussed in the tutorial. Exploiting the multiplexing techniques with the very high switching speed of digital micro-mirror device(DMD) as SLM, we have implemented 360° viewable holographic display, by which we can display three inch color object floating on top of a table-type display. More than one person can freely see the holographic 3D image from around the display, thus observing different perspectives of the object depending on their viewpoint. Display architecture and design parameters with technical issues for further enhancement will be explained.
Then we will present our implementation of stereoscopic mobile holographic display based on high resolution liquid crystal display(LCD) panel. Technical issues such as design of efficient back light unit with holographic optical elements, CGH for holographic stereogram, and encoding of the hologram for amplitude SLM will be discussed. One of major target applications of our holographic display is future communication systems like tele-presence system. For this, real time operation should be possible in end-to-end system: from capture, data compression and transmission, up to decoding and display. Real time handling of huge hologram data pose another challenge to the whole signal chain.
Finally, system issues will be briefly covered, including considerations to be given to end-to-end system architecture, compatible data format with other 3D systems, and hologram data compression.
About the presenters
Jinwoong Kim is a Principal Researcher at Electronics and Telecommunications Research Institute(ETRI). He received the B.S. and M.S. degrees in Electronics Engineering from Seoul National University, Seoul, Korea, in 1981 and 1983 respectively, the Ph. D. degree in Electrical Engineering from Texas A&M University, College Station, US, in 1993. He joined ETRI in 1983 and was involved in several big projects in telecommunications and broadcasting area, such as development of TDX-1 and TDX-10 digital telephone switching system, MPEG-2_based HDTV video encoding ASIC chipset and real-time encoder system. Then he carried out, under the name of “SmarTV Project”, several sub-projects on new digital broadcasting technologies including data broadcasting, viewer-customized broadcasting, and digital content protection. He also led projects on MPEG-7 and MPEG-21 core technology development, resulting in several important contributions to the MPEG standards. In 2005, he coordinated a specialist group forum and did a major role in producing a national strategic R&D planning report in the 3D visual technology area, named “3D Vision 2010”. After this, he was granted and led several R&D projects on stereoscopic 3DTV, 3D DMB and multiview 3DTV broadcasting system development. He was chair of 3DTV standard project group of TTA, and had been leading 3DTV broadcasting standardization activity of Korea and participation to ATSC 3DTV standardization. Since 2010, has been leading two big research projects on digital holographic technology development. He was a Far-East Liaison of 3DTV Conference 2007 and 2008, and a plenary speaker of 3DTV-CON 2010. He has been an invited speaker to a number of domestic as well as international workshops including EU-Korea Cooperation Forum workshop on ICT, 3D Fair 2008 in Tokyo and WIO 2013 in Tenerife Spain, ICTC 2013 in Jeju island and DHIP 2013 in Daejeon Korea.
Keehoon Hong is a Senior Researcher at Electronics and Telecommunications Research Institute (ETRI) working in the Digital Holographic Research Section, Giga Service Research Department. He received the B.S. degree in Electrical and Electronic Engineering from Yonsei University, Seoul, Korea, in 2008 and the Ph. D. degree in Electrical Engineering and Computer Science from Seoul National University, Seoul, Korea in 2014. His research interests include autostereoscopic 3D display, digital holographic display, holographic optical elements, and human factors related on 3D displays. He is a co-author of the book, Design and Implementation of Autostereoscopic Displays, SPIE Press, 2016.