Foundations and Trends® in Information Retrieval > Vol 3 > Issue 1–2

Methods for Evaluating Interactive Information Retrieval Systems with Users

By Diane Kelly, School of Information and Library Science, University of North Carolina at Chapel Hill, USA, dianek@email.unc.edu

 
Suggested Citation
Diane Kelly (2009), "Methods for Evaluating Interactive Information Retrieval Systems with Users", Foundations and Trends® in Information Retrieval: Vol. 3: No. 1–2, pp 1-224. http://dx.doi.org/10.1561/1500000012

Publication Date: 28 Apr 2009
© 2009 D. Kelly
 
Subjects
User modelling and user studies for IR
 

Free Preview:

Download extract

Share

Download article
In this article:
1 Introduction 
2 What is Interactive Information Retrieval? 
3 Background 
4 Approaches 
5 Research Basics 
6 Experimental Design 
7 Sampling 
8 Collections 
9 Data Collection Techniques 
10 Measures 
11 Data Analysis 
12 Validity and Reliability 
13 Human Research Ethics 
14 Outstanding Challenges and Future Directions 
15 Conclusion 
References 

Abstract

This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users. The primary goal of this article is to catalog and compile material related to this topic into a single source. This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies. This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR). Finally, this article concludes with a discussion of outstanding challenges and future research directions.

DOI:10.1561/1500000012
ISBN: 978-1-60198-224-7
236 pp. $99.00
Buy book (pb)
 
ISBN: 978-1-60198-225-4
236 pp. $150.00
Buy E-book (.pdf)
Table of contents:
1: Introduction
2: What is IIR
3: Background
4: Approaches
5: Research Basics
6: Experimental Design
7: Sampling
8: Instruments and Data Collection
9: Data Collection Techniques
10: Measures
11: Data Analysis
12: Validity and Reliability
13: Human Research Ethics
14: Outstanding Challenges and Future Directions
15: Conclusion
Acknowledgements
References

Methods for Evaluating Interactive Information Retrieval Systems with Users

Methods for Evaluating Interactive Information Retrieval Systems with Users provides a foundation on which those new to Interactive Information Retrieval (IIR) can make more informed choices about how to design and conduct IIR evaluations with human subjects. The primary goal is to catalog and compile material related to the IIR evaluation method into a single source. Methods for Evaluating Interactive Information Retrieval Systems with Users begins by providing historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems. It proceeds to describe the major components of interactive information retrieval system evaluation and goes on to describe different experimental designs and sampling strategies. From there, it presents core instruments and data collection techniques and measures, explains basic data analysis techniques and reviews and discusses previous studies. It also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval. It concludes with a discussion of outstanding challenges and future research directions.

 
INR-012