ut ut

Laboratory for Image & Video Engineering

Virtual Reality

The Future Projects that LIVE is targeting are problems of great significance for which there currently exist no adequate solution. Virtual Reality, which is currently seen as among The Next Big Things, presents several such problems. Of foremost interest to LIVE are the questions of Virtual Reality Picture Quality, Visual Comfort, and most importantly, Safety. Since at least the time of Jaron Lanier researchers in LIVE have been fascinated by the topic of virtual reality, or VR. It would seem that the reality of VR as a mainstream consumer technology may finally be approaching, an event we view with both excitement and reservation.

VR Picture Quality: We have begun studying VR picture quality in a very limited way, using our own resources and those of others. Building on our background in pictyre quality, we envision develop models and algorithms for automatically detecting and assessing the perceptual impact of distortions, both “just noticeable” and supra-threshold, on the viewing of immersive VR content. Doing this in the right way will be a rather immense task, requiring us to capture a vast database of immersive 3D picture and video content, introducing realistic distortions into the content in a perceptually separable manner, conducting large-scale human studies on the perceived quality of the content, and developing algorithm(s) that can effectively predict the quality of distorted immersive pictures. The task will be challenged by the scaling of the pictures, the dependence of quality perception on gaze direction / saliency, and the durations required to capture human judgements of an entire immersive scene that requires head and eye movements to experience in full.

VR Visual Discomfort: While picture quality is important, a more worrisome topic is visual discomfort experienced when viewing VR content. This can include nausea, disorientation, headache, eye strain, and more. We have done significant work on non-VR visual discomfort prediction and avoidance, and hope to build on this strong background going forward. The work going forward will involve interesting the neuroscience of vision and of the oculomotor control system that controls the focusing and movements of the eyes when viewing 3D, as well as theories of neuro- statistical models of 3D shape, depth, and disparity that heretofore have not been used for visual discomfort modeling. Our specific goal would be to develop simple, fast algorithms to predict the degree of experienced visual discomfort, as well as algorithms that can be used to mediate the modification of 3-D content during content creation or as the content is being viewed.

VR Safety: Most significant of all in our view are safety issues connected with VR viewing. These have been largely unaddressed and unmodeled by both the scientific community and by the VR industry. Yet it is a problem of potential global consequence as VR unfolds into mass commercialization, that goes far beyond worries regarding visual discomfort. The main concern is that when viewers, particularly younger viewers, watch immersive 3-D VR content for extended periods (e.g., hours), adaptations of their visual systems may occur. Importantly, this could happen even while not causing feelings of discomfort, causing stealthy damage to the visual brain, rewiring it to reflect an unreal 3D world, while losing the ability to correctly perceive the real 3D world. The human visual system is surprisingly plastic, and very sensitive to the statistics of the visual content that is being watched. While it is known that excessive “screen time” can adversely affect emotion and personality, it can also powerfully affect they way we see. While viewing 3-D VR can be constructive, for example assisting visually impaired (stereo-deficient) patients in learning to see depths better, it may also be destructive, and cause viewers to adapt to incorrect statistical distributions of depths, or reduced depth distributions. This could result in reduced visual proprioception, which could affect the ability to navigate, drive, and be active in the world. While this would be of particular concern during the earlier developmental years, the neural plasticity of the visual systems engages this as a concern for anyone who might over-use VR. This is a difficult problem to approach, since conducting studies of the long-term effects of possibly damaging immersive 3D VR content is unrealistic. Instead, we would like to leverage our expertise and experience in 3D viewing, 3D neuroscience, and 3D natural scene modeling to develop automatic or semi-automatic algorithms for 3D VR content creation or modification, towards forcing the 3-D statistics of created VR content to adhere to the known statistical laws of the visual environment over 3D space and time. This very large problem will require deep innovations in immersive VR perceptual modeling, experimentation, and algorithm design. In our view, given the rapidity in technical advances that are being made, but without a theory, guidance, or plan regarding the relevant health issues, this is the best path to take. We think that this is among the most pressing problems in information science and neuroscience, the solution to which could affect hundreds of millions of viewers. Some of the papers we have published on topics relevant to 3D natural scene statistics, experienced 3D quality and visual discomfort (but not in immersion settings), follow:

Y. Liu, A.C. Bovik and L.K. Cormack, “Statistical modeling of 3D natural scenes with application to Bayesian stereopsis,” IEEE Transactions on Image Processing, vol. 20, no. 9, pp. 2515-2530, September 2011.
C.-C. Su, A.C. Bovik and L.K. Cormack, “Color and depth priors in natural images,” IEEE Transactions on Image Processing, vol. 22, no. 6, pp. 2259-2274, June 2013.
M.-J. Chen, L.K. Cormack and A.C. Bovik, “No-reference quality assessment of natural stereopairs,” IEEE Transactions on Image Processing, vol. 22, no. 9, pp. 3379-3391, September 2013.
A.K. Moorthy, C.-C. Su, A. Mittal and A.C. Bovik, “Subjective evaluation of stereoscopic image quality,” Signal Processing: Image Communication, vol. 28, no. 9, pp. 870-883, September 2013.
M.-J. Chen, A.C. Bovik and L.K. Cormack, “Study of distortion conspicuity on stereoscopically viewed 3D images,” Journal of the Society for Information Display, vol. 21 no. 11, pp. 491-503, November 2013.
K. Lee, A.K. Moorthy, S. Lee and A.C. Bovik, “3D visual activity assessment based on natural scene statistics,” IEEE Transactions on Image Processing, vol. 23, no. 1, pp. 450- 465, January 2014.
T. Kim, J. Kang, S. Lee and A.C. Bovik, “Interactive continuous quality evaluation of subjective 3D video quality of experience,” IEEE Transactions on Multimedia, vol. 16. no. 2, pp. 387-402, February 2014.
H. Kim, S. Lee and A.C. Bovik, “Saliency measurement on stereoscopic videos,” IEEE Transactions on Image Processing, vol. 23, no. 4, pp. 1476-1490, April 2014.
J. Park, S. Lee and A.C. Bovik, “3D visual discomfort prediction: Vergence, foveation, and the physiological optics of accomodation,” IEEE Journal on Selected Topics in Signal Processing, vol. 8, no. 3, pp. 415-427, June 2014.
J. Park, H. Oh, S. Lee and A.C. Bovik, “3D visual discomfort predictor: Analysis of disparity and neural activity statistics,” IEEE Transactions on Image Processing, vol. 24, no. 3, pp. 1101-1114, March 2015.
T. Kim, S. Lee and A.C. Bovik, “Transfer function model of physiological mechanisms underlying temporal visual comfort experienced when viewing stereoscopic 3D images,” IEEE Transactions on Image Processing, vol 24, no. 11, pp. 4335-4347, November 2015.
H. Oh, S. Lee and A.C. Bovik, “3D visual discomfort prediction: A dynamic accommodation and vergence interaction model,” IEEE Transactions on Image Processing, vol. 25, no. 2, pp. 615-629, February 2016.
K.J. Chen, J. Zhou, J. Sun and A.C. Bovik, “3D visual discomfort prediction using low complexity disparity algorithms,” EURASIP Journal on Image and Video Processing, DOI: 10.1186/s13640-016- 0127-4, vol. 2016, no. 1, pp. 1-10, August 2016.
K.J. Chen and A.C. Bovik, “Visual discomfort prediction on stereoscopic 3D images without explicit disparities,” Signal Processing: Image Communication, vol. 51, pp. 50-60, February 2017.
T. Kim, S. Lee and A.C. Bovik, “Enhancement of visual comfort and sense of presence on stereoscopic 3D images,” IEEE Transactions on Image Processing, vol. 26, no. 8, pp. 3789- 3801, August 2017.