Light Field Compression and Streaming

[People] [Goals] [Activities and Findings] [Software] [Publications]

People

Chuo-Ling Chang
Mark Kalman
Prashant Ramanathan
Xiaoqing Zhu
Bernd Girod

Goals

This research project will investigate the compression and streaming of light fields.  The topics that are currently under study are:

• Disparity-compensated lifting for 4-D wavelet decomposition

• Embedded entropy coder for 4-D wavelets

• Geometry refinement

• Theoretical analysis

• Rate-distortion optimized packet scheduling

• Packetization and streaming of scalable light fields bitstreams

• Multiple representation encoding

• Distributed source coding of light fields

• Real-time light-field viewer

Activities and Findings

Disparity-compensated lifting

We have investigated the compression performance and support for scalability of the disparity-compensated lifting scheme for light field compression. Various wavelet kernels for the wavelet transform across the views have been implemented with disparity compensation incorporated into the lifting framework. The effects of different wavelet kernels and different number of resolution levels have been studied. Rate-distortion performance and scalability of the lifting scheme have been compared with previous work on light field compression. The publications [Girod_ICASSP2003] and [Chang_VCIP2003] document these experiments.

Disparity-compensated lifting significantly outperforms previous techniques for light field compression in terms of both rate-distortion performance and support for scalability. The effective exploitation of the correlation across individual views provides a gain typically around 4dB for the same quality of the reconstructed views. The lifting framework also enables view-point scalability, image-size scalability and reconstruction scalability for light fields.
 

Embedded entropy coder for 4-D wavelets

We have implemented the SPIHT algorithm for wavelet coefficient coding in the disparity-compensated lifting scheme. The SPIHT algorithm has been modified to work in a blockwise manner. Rate-distortion optimized bit-stream truncation is adopted to optimally combine the bitstreams from different blocks to a scalable representation. A new shape adaptation and encoding technique has been developed and incorporated into the light field coder. It has also been beneficially incorporated into an existing DCT-based light-field coder. Shape-adaptation is reported in the publication [Chang_ICIP2003], and other aspects of the entropy coder are reported in [Girod_ICASSP2003] and [Chang_VCIP2003].

Blockwise coding of the wavelet coefficients provides more flexibility for access of the compressed data and reduces the computational complexity. Combined with rate-distortion optimized truncation, blockwise coding results in better bit-allocation in different parts of the image. Shape-adaptive coding of light fields greatly improves compression efficiency as well as visual quality of the reconstructed views. The results from disparity-compensated lifting, as well as the entropy coder, are reported in [Girod_ICASSP2003] [Chang_VCIP2003] and [Chang_ICIP2003]. [Chang_ICIP2003] specifically discusses shape-adaptation.
 

Geometry refinement

Building upon some of our previous work, we have developed an algorithm to automatically refine inaccurate geometry such that compression efficiency is maximized. In order to identify and mitigate the effects of outlier data on the algorithm, we have used a weighted least squares approach and experimented with various weighting functions. We also investigated a multi-resolution approach to geometry refinement. This work has been reported in [Sebe_VMV2002].

Our geometry refinement algorithm further enhances compression efficiency. It is more stable than a previous algorithm and requires only one-tenth of the computation time. Experiments on synthetic data sets show that the new algorithm achieves nearly the best possible results, given the constraints on the geometry model used. The results can be found in [Sebe_VMV2002].
 

Theoretical analysis

We have proposed a theoretical framework to analyze the rate-distortion performance of light field coding, as reported in [Ramanathan_ICIP2002]. We have built upon this work in [Ramanathan_ICIP2004] to also the incorporate random access capabilities of a particular coding scheme into the rate-distortion framework. We have further extended the framework and used it to study the rate-distortion performance of a streaming system in [Ramanathan_PCS2004].

Our theoretical framework [Ramanathan_ICIP2002] provides a way of studying the effects of the various parameters of a light field coding system on compression efficiency, which may be difficult or impossible to study with real light field coder. More recently, our rate-distortion analysis in [Ramanathan_ICIP2004], that considers view-trajectory-dependent distortion, shows that random access is very important when accessing only a part of the light field. This analysis is extended and applied to a streaming framework in [Ramanathan_PCS2004]. Random access turns out to be extremely important for interactive streaming. In our theoretical model, depending on the parameters, independent encoding of light fields images is often found to be more efficient for streaming because of its easy random access.
 

Rate-distortion optimized packet scheduling

We have extended the rate-distortion optimized packet scheduling framework proposed by Chou et al. to interactive light field streaming. We achieved this by introducing the following new concepts:

• view-dependent distortions

• multiple deadlines for a data unit

• distortions based on client-buffer state

We have developed a system to calculate the view-dependent distortions for any desired user view trajectory. We implemented a simulator for our model of packet delays and losses, as well as packet scheduling based on the above concepts. We document this in [Ramanathan_ICIP2003], and there is related work in [Kalman_ICIP2003]. We have investigated experimentally how the dependencies and random access of different coding schemes affect streaming performance. Additionally, the streaming framework has been extended to allow for multiple representation coding (see Multiple representation coding). This work has been submitted as [Ramanathan_PV2004].

Our experiments in streaming of conventional prediction-based encodings show that our rate-distortion optimized packet scheduling can perform significantly better than even a fairly sophisticated heuristic scheme that does not consider per-data-unit distortion contributions. This is both in terms of visual quality over various trajectories, and in measured rate-distortion performance. These results are reported in [Ramanathan_ICIP2003]. In our experiments with different prediction dependency structures, we find that the random access of a particular scheme is as important as the compression efficiency. Independent coding of images often gave the best rate-distortion performance for the streaming system. Some of these results can be found in a submitted paper [Ramanathan_PV2004].
 

Packetization and streaming of scalable light fields bitstreams

We have considered the streaming of scalable light field bitstreams where each light field image is independently encoded using wavelet techniques. We have proposed two rate-distortion optimized interactive streaming frameworks, where the focus is on assembling and packetizing the bits that are relevant to the remote user. A sender-based system is proposed in [Chang_VCIP2004] where, in response to a receiver request, the sender allocates the bitstreams to the outgoing packets in a manner such that the distortion of the novel view at the receiver is minimized. A corresponding receiver-based system is also proposed in [Chang_ICME2004], where the computational burden is shifted to the receiver, to allow the sender to serve many clients.

For streaming scalable bitstreams of light fields, experiments show that the proposed rate-distortion optimized framework performs consistently better than a heuristic scheme that does not consider the rate-distortion characteristics of the encoded light field dataset. To achieve the same rendering quality, the optimized framework can save up to 25% of the transmission rate over the heuristic scheme. We also find that the sender-based framework in general outperforms the receiver-based one, though not significantly. Nevertheless, the receiver-based framework removes the computational burden at the server, hence allows the server to simultaneously serve a large number of receivers. The results for the sender-based system are reported in [Chang_VCIP2004]. The results for the receiver-based system, and its comparison with the sender-based system, are reported in [Chang_ICME2004].
 

Multiple representation encoding

We have proposed a multiple representation encoding of light fields that uses prediction for efficient compression, while providing random access to images. This work has been reported in [Ramanathan_MMSP2004]. We have also proposed an extension to the rate-distortion optimized streaming framework to work with this encoding (see Theoretical analysis).

We found that our proposed multiple representation encoding of a light field provided random access that is comparable to independent coding of images, but with better compression efficiency since prediction is used between images. These results have been reported in [RamanathanMMSP_2004]. Experiments show that using multiple representations for interactive streaming can outperform independent coding and conventional predictive encoding, depending on the data set and encoding [Ramanathan_PV2004].
 

Distributed source coding of light fields

We have investigated Wyner-Ziv coding of light fields to provide a low-complexity encoding system, reported in [Zhu_SSP2003]. Wyner-Ziv coding has also been used to provide random access to images and good compression efficiency, as reported in [Aaron_MMSP2004].

In related work, reported in [Aaron_MMSP2004], we find that Wyner-Ziv coding of light fields can also be used to provide random access, with better compression efficiency than independent coding of images. We also show that Wyner-Ziv coding can provide low-complexity encoding of light field images, which may be important during light field acquisition. This aspect of Wyner-Ziv light field coding is described in [Zhu_SSP2003].
 

Real-time light field viewer

We have implemented a real-time light field viewer which runs on a Linux platform. The source code can be downloaded below. We hope to eventually integrate this viewer into a working light field streaming test-bed.

Software

Our implementation of a real-time light field viewer can be downloaded from here.

Publications

[Sebe_VMV2002] Ismail Oner Sebe, Prashant Ramanathan and Bernd Girod, "Multi-view Geometry Estimation for Light Field Compression," Proc. Vision, Modeling, and Visualization, VMV-2002, Erlangen, Germany, Nov. 2002, pp. 265-272. [pdf]

[Ramanathan_ICIP2002_Geometry] Prashant Ramanathan, Eckehard Steinbach, Peter Eisert and Bernd Girod, "Geometry Refinement for Light Field Compression," Proc. IEEE International Conference on Image Processing, ICIP-02, Rochester, NY, September 2002, vol. 2, pp. 225-228. [pdf]

[Ramanathan_ICIP2002_Theory] Prashant Ramanathan and Bernd Girod, "Theoretical Analysis of Geometry Inaccuracy for Light Field Compression," Proc. IEEE International Conference on Image Processing, ICIP-02, Rochester, NY, September 2002, vol. 2, pp. 229-232. [pdf]

[Flierl_ICIP2002] Markus Flierl and Bernd Girod, "Video Coding with Motion Compensation for Groups of Pictures," Proc. IEEE International Conference on Image Processing, ICIP-02, Rochester, NY, September 2002, vol. 1, pp. 69-72. [pdf]

This material is based upon work supported by the National Science Foundation under Grant No. 0225315. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the  views of the National Science Foundation (NSF). This work has also been supported, in part, by Intel Corporation and Okawa Foundation.

Last Modified: 09/01/2004