Hide menu

The Virtual Photo Set

SSF IIS11-0081 - Data driven scene characterization for realistic rendering

 

The Virtual Photo Set

Grant supported by the Swedish Foundation for Strategic Research (SFF) through the Information Intensive Systems programme.
Project leader: Jonas Unger

 List of Publications

 

Project overview

This demonstrator project is motivated by the rapid growth of image-, sensor- and meta-data available on the internet, through databases, and capture/measurements. These huge data sets and input sources make it possible to analyze the world around us in new ways. The information (pixel data) in these images is, in its raw form, unstructured and contain large redundancies and ambiguities. It is of high importance to develop new theoretical tools for structuring these large data sets, and algorithms for extracting semantic information from visually rich scenes.

Traditional photography is only one modality that can be used to represent reality. In this project we take steps towards a richer documentation of scenes and indeed also provide the tools needed to start to modify and augment the captured reality. The Virtual Photo Set (VPS) will allow users to blend real and virtual objects and generate photo realistic images in which the real and virtual objects are indistinguishable. It will also allow the users to edit the captured reality to alter such things as light sources, object placement and materials etc.  The VPS is a natural step in the on-going paradigm shift of the way in which we capture and document reality.

The project includes the development of novel: new multi-modal scene capture hardware, processing techniques for extraction of semantic scene information and scene reconstruction, as well as algorithms for compression and efficient image synthesis. The demonstrator developed within this project is built on a strong foundation of theoretical research, and encompasses all steps from scene capture and processing, to interactive editing and photo-realistic image synthesis.

In-depth overviews of results and applications with open data and source code from this project can be found at:

http://www.hdrv.org (HDR video capture, scene reconstruction and image based lighting)

Virtual Photo Sets Siggraph Asia 2015 course material (data and extensive course notes)

http://lumahdrv.org (An open source perceptually based HDR video codec)

Project team

This demonstrator project is running during 2012 - 2016, and is carried out within a collaboration between:

The Computer Graphics and Image Processing Group (MIT),   Linköping University

Computer Vision Laboratory (CVL), Linköping University

Sensor Fusion Group, Automatic Control (AC), Linköping University

 

The following researchers and engineers are working within the project:

Senior researchers:

Prof. Anders Ynnerman (MIT)

Prof. Michael Felsberg (CVL)

Prof. Fredrik Gustafsson (AC)

Prof. Reiner Lenz (MIT)

Dr. Jonas Unger (MIT)

Dr. Per-Erik Forsen (CVL)

Gabriel Eilertsen (MIT)

Saghi Hajisharif (MIT)

Joel Kronander (MIT)

Ehsan Miandji (MIT)

Hanna Nyqvist (AC)

Hannes Ovren (CVL)

Andrew Gardner (MIT)

Per Larsson (MIT)

Giulia Meneghetti (CVL)

Apostolia Tsirikoglou (MIT)

 

 

Publications

 

2015
Freddie Åström, George Baravdish, Michael Felsberg, "A Tensor Variational Formulation of Gradient Energy Total Variation", 10th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR 2015), 13-16 January 2015, Hong Kong, China, 2015.
AbstractKeywordsBiBTeX
2014
Vasileios Zografos, Reiner Lenz, Erik Ringaby, Michael Felsberg, Klas Nordberg, "Fast segmentation of sparse 3D point trajectories using group theoretical invariants", Computer Vision -- ACCV 2014, Lecture Notes in Computer Science, Vol. 9003--9007, 2014.
AbstractKeywordsBiBTeX
Takanori Kojima, Rika Mochizuki, Reiner Lenz, Jinhui Chao, "Riemann Geometric Color-Weak Compensationfor Individual Observers", Universal Access in Human-Computer Interaction, Lecture Notes in Computer Science, Vol. 8514, 121-131, 2014.
AbstractKeywordsBiBTeX
Erik Ringaby, Per-Erik Forssén, "A Virtual Tripod for Hand-held Video Stacking on Smartphones", IEEE International Conference on Computational Photography (ICCP 2014), May 2-4, 2014, Intel, Santa Clara, USA, 1-9, 2014.
AbstractKeywordsBiBTeXDOI
Andrew Gardner, Jonas Unger, "Depends: Workflow Management Software for Visual Effects Production", Digital Production Symposium, DigiPro '14, Vancouver, Canada, August, 2014, DigiPro '14, 2014.
AbstractKeywordsBiBTeXFulltext
Joel Kronander, Johan Dahlin, Daniel Jönsson, Manon Kok, Thomas Schön, Jonas Unger, "Real-time video based lighting using GPU raytracing", Proceedings of the 22nd European Signal Processing Conference (EUSIPCO), 2014, 2014.
AbstractKeywordsBiBTeXFulltext
Reiner Lenz, "Generalized Extreme Value Distributions, Information Geometry And Sharpness Functions For Microscopy Images: ", 2014 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), International Conference on Acoustics Speech and Signal Processing ICASSP, 2867-2871, 2014.
AbstractKeywordsBiBTeX
Joel Kronander, Thomas B. Schön, Johan Dahlin, "Backward sequential Monte Carlo for marginal smoothing", Proceedings of the 2014 IEEE Statistical Signal Processing Workshop, 2014.
KeywordsBiBTeX
Joel Kronander, Thomas B. Schön, "Robust auxiliary particle filters using multiple importance sampling", Proceedings of the 2014 IEEE Statistical Signal Processing Workshop, 2014.
KeywordsBiBTeX
Gabriel Eilertsen, Jonas Unger, Robert Wanat, Rafal Mantiuk, "Perceptually based parameter adjustments for video processing operations", ACM SIGGRAPH Talks 2014, 2014.
AbstractKeywordsBiBTeXFulltext
Apostolia Tsirikoglou, Simon Ekeberg, Johan Vikström, Joel Kronander, Jonas Unger, "S(wi)SS: A flexible and robust sub-surface scattering shader", Proceedings of SIGRAD 2014, 2014.
AbstractKeywordsBiBTeXFulltext
Saghi Hajsharif, Joel Kronander, Jonas Unger, "HDR reconstruction for alternating gain (ISO) sensor readout", Eurographics 2014 short papers, 2014.
AbstractKeywordsBiBTeXFulltext
Joel Kronander, Stefan Gustavson, Gerhard Bonnet, Anders Ynnerman, Jonas Unger, "A unified framework for multi-sensor HDR video reconstruction", Signal Processing : Image Communications, 29(2): 203-215, 2014.
AbstractKeywordsBiBTeXDOI
2013
Gabriel Eilertsen, Robert Wanat, Rafal K. Mantiuk, Jonas Unger, "Evaluation of Tone Mapping Operators for HDR-Video", Computer graphics forum (Print), 32(7): 275-284, 2013.
AbstractKeywordsBiBTeXDOIFulltext
Ehsan Miandji, Joel Kronander, Jonas Unger, "Learning Based Compression of Surface Light Fields for Real-time Rendering of Global Illumination Scenes", Proceedings of ACM SIGGRAPH ASIA 2013, 2013.
AbstractKeywordsBiBTeXDOIFulltext
Jonas Unger, Joel Kronander, Per Larsson, Stefan Gustavson, Joakim Löw, Anders Ynnerman, "Spatially varying image based lighting using HDR-video", Computers & graphics, 37(7): 923-934, 2013.
AbstractKeywordsBiBTeXDOIFulltext
Hanna Nyqvist, Fredrik Gustafsson, "A High-Performance Tracking System based on Camera and IMU", 2013 16th International Conference on Information Fusion, 2065-2072, 2013.
AbstractKeywordsBiBTeXFulltext
Jonas Unger, Joel Kronander, Per Larsson, Stefan Gustavson, Anders Ynner, "Temporally and Spatially Varying Image Based Lighting using HDR-video", Proceedings of EUSIPCO '13, 2013.
AbstractKeywordsBiBTeXFulltext
Ehsan Miandji, Joel Kronander, Jonas Unger, "Learning based compression for real-time rendering of surface light fields", Siggraph 2013 Posters, 2013.
AbstractKeywordsBiBTeXDOI
Gabriel Eilertsen, Jonas Unger, Robert Wanat, Rafal Mantiuk, "Survey and Evaluation of Tone Mapping Operators for HDR-video", Siggraph 2013 Talks, 2013.
AbstractKeywordsBiBTeXDOIFulltext
Vasileios Zografos, Reiner Lenz, Michael Felsberg, "The Weibull manifold in low-level image processing: an application to automatic image focusing.", Image and Vision Computing, 31(5): 401-417, 2013.
AbstractKeywordsBiBTeXDOIFulltext
Reiner Lenz, Vasileios Zografos, "Fisher Information and the Combination of RGB channels", 4th International Workshop, CCIW 2013, Chiba, Japan, March 3-5, 2013. Proceedings, Lecture Notes in Computer Science, Vol. 7786, 250-264, 2013.
AbstractKeywordsBiBTeXDOIFulltext
Johan Hedborg, Michael Felsberg, "Fast Iterative Five point Relative Pose Estimation", IEEE Workshop on Robot Vision (WoRV 2013), January 15-17, 2013, Clearwater, FL, USA, 60-67, 2013.
AbstractKeywordsBiBTeXDOIFulltext
Joel Kronander, Stefan Gustavson, Gerhard Bonnet, Jonas Unger, "Unified HDR reconstruction from raw CFA data", Proceedings of IEEE International Conference on Computational Photography 2013, 1-9, 2013.
AbstractKeywordsBiBTeXDOI
Reiner Lenz, Vasileios Zografos, Martin Solli, "Dihedral Color Filtering", Advanced Color Image Processing and Analysis, 119-145, 2013.
AbstractKeywordsBiBTeXDOI
2012
Reiner Lenz, Vasileios Zografos, "RGB Filter design using the properties of the weibull manifold", CGIV 2012 Sixth European Conference on Colour in Graphics, Imaging, and Vision, 200-205, 2012.
AbstractKeywordsBiBTeXFulltext
Saghi Hajisharif, Joel Kronander, Ehsan Miandji, Jonas Unger, "Real-time image based lighting with streaming HDR-lightprobe sequences", Proceedings of SIGRAD 2012, 2012.
AbstractKeywordsBiBTeXFulltext
Satoshi Oshima, Rika Mochizuki, Reiner Lenz, Jinhui Chao, "Color-Weakness Compensation using Riemann Normal Coordinates", 2012 IEEE International Symposium on Multimedia, 175-178, 2012.
AbstractKeywordsBiBTeXDOIFulltext
Joel Kronander, Stefan Gustavson, Jonas Unger, "Real-time HDR video reconstruction for multi-sensor systems", ACM SIGGRAPH 2012 Posters, 65-, 2012.
AbstractKeywordsBiBTeXDOIFulltext
Johan Hedborg, "Motion and Structure Estimation From Video", Linköping Studies in Science and Technology. Dissertations, No. 1449, 2012.
AbstractKeywordsBiBTeXFulltext
Johan Hedborg, Per-Erik Forssén, Michael Felsberg, Erik Ringaby, "Rolling Shutter Bundle Adjustment", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, Computer Vision and Pattern Recognition, 1434-1441, 2012.
AbstractKeywordsBiBTeXDOI

Events at MIT


Page manager: gunnar.host@liu.se
Last updated: Fri Mar 25 19:37:05 CET 2016