Virtual Reality
The Future Projects that LIVE is targeting are problems of great significance for
which there currently exist no adequate solution. Virtual Reality, which is currently
seen as among The Next Big Things, presents several such problems. Of foremost
interest to LIVE are the questions of Virtual Reality Picture Quality, Visual
Comfort, and most importantly, Safety.
Since at least the time of Jaron Lanier researchers in LIVE have been fascinated by
the topic of virtual reality, or VR. It would seem that the reality of VR as a
mainstream consumer technology may finally be approaching, an event we view
with both excitement and reservation.
VR Picture Quality: We have begun studying VR picture quality in a very limited
way, using our own resources and those of others. Building on our background in
pictyre quality, we envision develop models and algorithms for automatically
detecting and assessing the perceptual impact of distortions, both “just noticeable”
and supra-threshold, on the viewing of immersive VR content. Doing this in the
right way will be a rather immense task, requiring us to capture a vast database of
immersive 3D picture and video content, introducing realistic distortions into the
content in a perceptually separable manner, conducting large-scale human studies on
the perceived quality of the content, and developing algorithm(s) that can effectively
predict the quality of distorted immersive pictures. The task will be challenged by
the scaling of the pictures, the dependence of quality perception on gaze direction /
saliency, and the durations required to capture human judgements of an entire
immersive scene that requires head and eye movements to experience in full.
VR Visual Discomfort: While picture quality is important, a more worrisome topic
is visual discomfort experienced when viewing VR content. This can include nausea,
disorientation, headache, eye strain, and more. We have done significant work on
non-VR visual discomfort prediction and avoidance, and hope to build on this strong
background going forward. The work going forward will involve interesting the
neuroscience of vision and of the oculomotor control system that controls the
focusing and movements of the eyes when viewing 3D, as well as theories of neuro-
statistical models of 3D shape, depth, and disparity that heretofore have not been
used for visual discomfort modeling. Our specific goal would be to develop simple,
fast algorithms to predict the degree of experienced visual discomfort, as well as
algorithms that can be used to mediate the modification of 3-D content during
content creation or as the content is being viewed.
VR Safety: Most significant of all in our view are safety issues connected with VR
viewing. These have been largely unaddressed and unmodeled by both the scientific
community and by the VR industry. Yet it is a problem of potential global
consequence as VR unfolds into mass commercialization, that goes far beyond
worries regarding visual discomfort. The main concern is that when viewers,
particularly younger viewers, watch immersive 3-D VR content for extended periods
(e.g., hours), adaptations of their visual systems may occur. Importantly, this could
happen even while not causing feelings of discomfort, causing stealthy damage to
the visual brain, rewiring it to reflect an unreal 3D world, while losing the ability to
correctly perceive the real 3D world. The human visual system is surprisingly
plastic, and very sensitive to the statistics of the visual content that is being watched.
While it is known that excessive “screen time” can adversely affect emotion and
personality, it can also powerfully affect they way we see. While viewing 3-D VR
can be constructive, for example assisting visually impaired (stereo-deficient)
patients in learning to see depths better, it may also be destructive, and cause
viewers to adapt to incorrect statistical distributions of depths, or reduced depth
distributions. This could result in reduced visual proprioception, which could affect
the ability to navigate, drive, and be active in the world. While this would be of
particular concern during the earlier developmental years, the neural plasticity of the
visual systems engages this as a concern for anyone who might over-use VR.
This is a difficult problem to approach, since conducting studies of the long-term
effects of possibly damaging immersive 3D VR content is unrealistic. Instead, we
would like to leverage our expertise and experience in 3D viewing, 3D neuroscience,
and 3D natural scene modeling to develop automatic or semi-automatic algorithms
for 3D VR content creation or modification, towards forcing the 3-D statistics of
created VR content to adhere to the known statistical laws of the visual environment
over 3D space and time. This very large problem will require deep innovations in
immersive VR perceptual modeling, experimentation, and algorithm design. In our
view, given the rapidity in technical advances that are being made, but without a
theory, guidance, or plan regarding the relevant health issues, this is the best path to
take. We think that this is among the most pressing problems in information science
and neuroscience, the solution to which could affect hundreds of millions of viewers.
Some of the papers we have published on topics relevant to 3D natural scene
statistics, experienced 3D quality and visual discomfort (but not in immersion
settings), follow:
Y. Liu, A.C. Bovik and L.K. Cormack, “Statistical modeling of 3D natural scenes with application to Bayesian stereopsis,” IEEE Transactions on Image Processing, vol. 20, no. 9, pp. 2515-2530, September 2011.
C.-C. Su, A.C. Bovik and L.K. Cormack, “Color and depth priors in natural images,” IEEE Transactions on Image Processing, vol. 22, no. 6, pp. 2259-2274, June 2013.
M.-J. Chen, L.K. Cormack and A.C. Bovik, “No-reference quality assessment of natural stereopairs,” IEEE Transactions on Image Processing, vol. 22, no. 9, pp. 3379-3391, September 2013.
A.K. Moorthy, C.-C. Su, A. Mittal and A.C. Bovik, “Subjective evaluation of stereoscopic image quality,” Signal Processing: Image Communication, vol. 28, no. 9, pp. 870-883, September 2013.
M.-J. Chen, A.C. Bovik and L.K. Cormack, “Study of distortion conspicuity on stereoscopically viewed 3D images,” Journal of the Society for Information Display, vol. 21 no. 11, pp. 491-503, November 2013.
K. Lee, A.K. Moorthy, S. Lee and A.C. Bovik, “3D visual activity assessment based on natural scene statistics,” IEEE Transactions on Image Processing, vol. 23, no. 1, pp. 450- 465, January 2014.
T. Kim, J. Kang, S. Lee and A.C. Bovik, “Interactive continuous quality evaluation of subjective 3D video quality of experience,” IEEE Transactions on Multimedia, vol. 16. no. 2, pp. 387-402, February 2014.
H. Kim, S. Lee and A.C. Bovik, “Saliency measurement on stereoscopic videos,” IEEE Transactions on Image Processing, vol. 23, no. 4, pp. 1476-1490, April 2014.
J. Park, S. Lee and A.C. Bovik, “3D visual discomfort prediction: Vergence, foveation, and the physiological optics of accomodation,” IEEE Journal on Selected Topics in Signal Processing, vol. 8, no. 3, pp. 415-427, June 2014.
J. Park, H. Oh, S. Lee and A.C. Bovik, “3D visual discomfort predictor: Analysis of disparity and neural activity statistics,” IEEE Transactions on Image Processing, vol. 24, no. 3, pp. 1101-1114, March 2015.
T. Kim, S. Lee and A.C. Bovik, “Transfer function model of physiological mechanisms underlying temporal visual comfort experienced when viewing stereoscopic 3D images,” IEEE Transactions on Image Processing, vol 24, no. 11, pp. 4335-4347, November 2015.
H. Oh, S. Lee and A.C. Bovik, “3D visual discomfort prediction: A dynamic accommodation and vergence interaction model,” IEEE Transactions on Image Processing, vol. 25, no. 2, pp. 615-629, February 2016.
K.J. Chen, J. Zhou, J. Sun and A.C. Bovik, “3D visual discomfort prediction using low complexity disparity algorithms,” EURASIP Journal on Image and Video Processing, DOI: 10.1186/s13640-016- 0127-4, vol. 2016, no. 1, pp. 1-10, August 2016.
K.J. Chen and A.C. Bovik, “Visual discomfort prediction on stereoscopic 3D images without explicit disparities,” Signal Processing: Image Communication, vol. 51, pp. 50-60, February 2017.
T. Kim, S. Lee and A.C. Bovik, “Enhancement of visual comfort and sense of presence on stereoscopic 3D images,” IEEE Transactions on Image Processing, vol. 26, no. 8, pp. 3789- 3801, August 2017.