Date/Time
Date(s) - 18/02/2015
2:00 pm - 3:00 pm
Location
Meston Building
Users versus Models: What observations tells us about effectiveness metrics
Abstract:
Retrieval system effectiveness can be measured in two quite different ways: by monitoring the behaviour of users and gathering data about the ease and accuracy with which they accomplish certain specified information-seeking tasks; or by using numeric effectiveness metrics to score system runs in reference to a set of relevance judgements. In the second approach, the effectiveness metric is chosen in the belief that it predicts ease or accuracy.
This work explores that link, by analysing the assumptions and implications of a number of effectiveness metrics, and exploring how these relate to observable user behaviours. Data recorded as part of a user study included user self-assessment of search task difficulty; gaze position; and click activity. Our results show that user behaviour is influenced by a blend of many factors, including the extent to which relevant documents are encountered, the stage of the search process, and task difficulty. These insights can be used to guide development of batch effectiveness metrics.
Biography
Paul Thomas is a research scientist at the Commonwealth Scientific and Industrial Research Organisation (CSIRO). The CSIRO is Australia’s national science agency and is charged with carrying out research for Australian government, industry, and communities. He also holds an adjunct Professorship at the Research School of Computer Science at the Australian National University.
Dr Thomas’s research considers how people interact with information, in particular information retrieval systems such as web search or digital libraries. His expertise covers federated search systems, models of user behaviour, and evaluating search systems with explicit or implicit user feedback. Applications have included web-based systems for general use, personal and workplace search systems, digital libraries for professional groups, and search on mobile devices.
Dr Thomas’s work has been published in top-ranked journals and conferences including the Journal of the Association for Information Science and Technology (JASIST), the Information Retrieval Journal (IR), and the SIGIR and CIKM conferences. He has served on programme committees for all major conferences and journals in information retrieval; and amongst other service roles he has been invited to coordinate or be a mentor for the SIGIR Doctoral Consortium across three years.
Dr Thomas has significant expertise in industrial research, and experience in translating research into practise in industry and government. For example, he has led or worked on projects which analyse web usage and suggest improvements for some of Australia’s largest sites, and which monitor social media to improve government communications and policy-making. He has also designed and implemented digital libraries and novel information systems to support industry in Australia and internationally.
http://es.csiro.au/people/paul
The Host of this DIVF is Dr Leif Azzopardi, University of Glasgow and Dr Martin Halvey, Glasgow Caledonian University is the co-host.
