Subjective Image and Video Quality Assessment 

LIVE Quality Assessment Database

The Quality Assessment research at LIVE is being conducted in collaboration with CPS

New: Release 2 of the database has more distortion types and images! Scroll down to download.

Introduction

Quality Assessment research strongly depends upon subjective experiments to provide calibration data as well as a testing mechanism. After all, the goal of all QA research is to make quality predictions that are in agreement with subjective opinion of human observers. In order to calibrate QA algorithms and test their performance, a data set of images and videos whose quality has been ranked by human subjects is required. The QA algorithm may be trained on part of this data set, and tested on the rest.

The need for extensive calibration, testing and validation of a QA algorithm cannot be overemphasized. For the last three decades, QA researchers have considered it sufficient to calibrate, test and report performance on a limited set of images and videos. However, in light of the recent VQEG Phase-I testing report, which reports that the performance of a number of state-of-the-art video quality assessment algorithms was "statistically indistinguishable" from PSNR, the importance of extensive subjective QA experiments has been reasserted.

At LIVE (in collaboration with The Department of Psychology at the University of Texas at Austin), an extensive experiment was conducted to obtain scores from human subjects for a number of images distorted with different distortion types. These images were acquired in support of a research project on generic shape matching and recognition.

We have decided to make the data set available to the research community free of charge. If you use these images in your research, we kindly ask that you reference this website and our work listed below that makes use of them.

1. H.R. Sheikh, Z.Wang, L. Cormack and A.C. Bovik, "LIVE Image Quality Assessment Database Release 2", http://live.ece.utexas.edu/research/quality.
2. H.R. Sheikh and A.C. Bovik, "Image information and visual quality," Image Processing, IEEE Transactions on, vol.15, no.2pp. 430- 444, Feb. 2006.
3. H.R. Sheikh, M.F. Sabir and A.C. Bovik, "A statistical evaluation of recent full reference image quality assessment algorithms", Image Processing, IEEE Transactions on, vol. 15, no. 11, pp. 3440-3451, Nov. 2006.
4. Z. Wang, A.C. Bovik, H.R. Sheikh and E.P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," Image Processing, IEEE Transactions on , vol.13, no.4pp. 600- 612, April 2004.
5. Z. Wang and A.C. Bovik, "A universal image quality index," Signal Processing Letters, IEEE , vol.9, no.3pp.81-84, Mar 2002.

Please scroll to the end of the page to download the data set.

Subjective experiments are cumbersome to design and expensive. Although great care was taken to ensure that the testing environment was as close to the "real-world" as possible, we cannot claim that our subjective experiments were exhaustive, comprehensive, or precise in every respect. Hence, any research conducted with the database should consider the limitations imposed by the scope and methodology of our experiments.

Please contact Hamid Rahim Sheikh (hamid.sheikh@ieee.org) if you have any questions.
This investigators on this research were:
Hamid Rahim Sheikh (hamid.sheikh@ieee.org) -- Department of ECE at UT Austin
Dr. Alan C. Bovik (bovik@ece.utexas.edu) -- Department of ECE at UT Austin
Dr. Lawrence Cormack (cormack@psy.utexas.edu) -- Department of Psychology at UT Austin
Dr. Zhou Wang (zhouwang@ieee.org)

What's new in Release 2

Release 2 has the following differences from Release 1:

1. More distortion types: 

• JPEG compresses images (169 images). More subjects than in Release 1

• JPEG2000 compressed images (175 images)

• NEW! Gaussian blur (145 images)

• NEW! White noise (145 images)

• NEW! Bit errors in JPEG2000 bit stream (145 images)

1. More subjects for JPEG distortion

2. DMOS values instead of MOS values for distorted images

3. Different processing of raw scores than in Release 1 

4. Individual subject scores and source code for processing raw scores will be released later. 

Please read the readme.txt in the database release for more details about the database.

Download Release 2

Subjective database Release 2. Please email Hamid R. Sheikh (hamid dot sheikh at ieee dot org) for password request (mentioning Release 1 or 2), briefly describing the intended use of the database and your affiliation. If you download the database, it is assumed that you agree to the copyright notice

Update to Release 2

Download realigned subjective quality data here. This data was obtained by running realignment experiments on Release 2 data. The details of the experiment can be found in the paper: H. R. Sheikh, M. F. Sabir, A. C. Bovik, "A Statistical Evaluation of Recent Full Reference Quality Assessment Algorithms", Image Processing, IEEE Transactions on, vol. 15, no. 11, pp. 3440-3451, Nov. 2006. Download in the "Publications" section of the LIVE website.

Download Release 1

Subjective database for JPEG2000 - README.TXT. Please email Hamid R. Sheikh (hamid dot sheikh at ieee dot org)  for password request (mentioning Release 1 or 2), briefly describing the intended use of the database and your affiliation. If you download the database, it is assumed that you agree to the copyright notice

Subjective database for JPEG - README.TXT. Please email Hamid R. Sheikh (hamid dot sheikh at ieee dot org) for password request, briefly describing the intended use of the database and your affiliation. If you download the database, it is assumed that you agree to the copyright notice

Back to Quality Assessment Research page