Foundations and Trends® in Computer Graphics and Vision > Vol 15 > Issue 3

Towards Better User Studies in Computer Graphics and Vision

By Zoya Bylinskii, Adobe Research, USA, bylinski@adobe.com | Laura Herman, Adobe Inc., USA, lherman@adobe.com | Aaron Hertzmann, Adobe Research, USA, hertzman@adobe.com | Stefanie Hutka, Adobe Inc., USA, stefanie.hutka@gmail.com | Yile Zhang, University of Washington, USA, yz278@uw.edu

 
Suggested Citation
Zoya Bylinskii, Laura Herman, Aaron Hertzmann, Stefanie Hutka and Yile Zhang (2023), "Towards Better User Studies in Computer Graphics and Vision", Foundations and Trends® in Computer Graphics and Vision: Vol. 15: No. 3, pp 201-252. http://dx.doi.org/10.1561/0600000106

Publication Date: 08 May 2023
© 2023 Z. Bylinskii et al.
 
Subjects
Evaluation,  Design and evaluation,  Interdisciplinary influence,  Perception and the user interface,  Theory
 

Free Preview:

Download extract

Share

Download article
In this article:
1. Introduction
2. The Start of User Studies in Graphics and Vision
3. User Research Methods for Graphics and Vision
4. The Broader Landscape of User Research Methodologies
5. Reporting and Conducting Ethical Research
6. Limitations and Problems of User Studies
7. Conclusion
Acknowledgements
References

Abstract

Online crowdsourcing platforms have made it increasingly easy to perform evaluations of algorithm outputs with survey questions like “which image is better, A or B?”, leading to their proliferation in vision and graphics research papers. Results of these studies are often used as quantitative evidence in support of a paper’s contributions. On the one hand we argue that, when conducted hastily as an afterthought, such studies lead to an increase of uninformative, and, potentially, misleading conclusions. On the other hand, in these same communities, user research is underutilized in driving project direction and forecasting user needs and reception. We call for increased attention to both the design and reporting of user studies in computer vision and graphics papers towards (1) improved replicability and (2) improved project direction. Together with this call, we offer an overview of methodologies from user experience research (UXR), human-computer interaction (HCI), and applied perception to increase exposure to the available methodologies and best practices. We discuss foundational user research methods (e.g., needfinding) that are presently underutilized in computer vision and graphics research, but can provide valuable project direction. We provide further pointers to the literature for readers interested in exploring other UXR methodologies. Finally, we describe broader open issues and recommendations for the research community.

DOI:10.1561/0600000106
ISBN: 978-1-63828-172-6
64 pp. $55.00
Buy book (pb)
 
ISBN: 978-1-63828-173-3
64 pp. $150.00
Buy E-book (.pdf)
Table of contents:
1. Introduction
2. The Start of User Studies in Graphics and Vision
3. User Research Methods for Graphics and Vision
4. The Broader Landscape of User Research Methodologies
5. Reporting and Conducting Ethical Research
6. Limitations and Problems of User Studies
7. Conclusion
Acknowledgements
References

Towards Better User Studies in Computer Graphics and Vision

Most research in computer graphics and image synthesis produces outputs for human consumption. In many cases, these algorithms operate largely automatically; in other cases, interactive tools allow professionals or everyday users to author or edit images, video, textures, geometry, or animation.

Online crowdsourcing platforms have made it increasingly easy to perform evaluations of algorithm outputs with survey questions like “which image is better, A or B?”, leading to their proliferation in vision and graphics research papers. Results of these studies are often used as quantitative evidence in support of a paper’s contributions. When conducted hastily as an afterthought, such studies can lead to an increase of uninformative, and, potentially, misleading conclusions. On the other hand, in these same communities, user research is underutilized in driving project direction and forecasting user needs and reception.

Increased attention is needed in both the design and reporting of user studies in computer vision and graphics papers towards (1) improved replicability and (2) improved project direction. This monograph focusses on these aspects, and an overview of methodologies from user experience research (UXR), human-computer interaction (HCI), and applied perception to increase exposure to the available methodologies and best practices are also presented. Foundational user research methods are included, (e.g., need finding) that are presently underutilized in computer vision and graphics research, but can provide valuable project direction. Also, further pointers to the literature for readers interested in exploring other UXR methodologies are given, and broader open issues and recommendations for the research community are described.

 
CGV-106