Abstracting Domain-Specific Information Retrieval and Evaluation (ADmIRE)

Information Retrieval (IR) on theWorld WideWeb functions very effectively and efficiently. These search tools are specifically designed as multi-purpose tools, applicable in as wide an array of situations as possible. Nevertheless, all information is not equal. There are areas for which these tools are conceived too broadly to be useful: health or biomedical information, intellectual property information, social science publications, blogs, press photographs, etc. A search in one of these domains is called a domain-specific search. Such a search is specific in terms of the collection of documents indexed, search refinements arising from the domain characteristics, domain coverage specificity, types of multimodal data (e.g. images, chemical formulae) present in the documents, and the end users and their tasks. Even though many domains have similar characteristics and challenges, there is as yet no general framework for developing domain-specific search solutions, and no way of characterising a domain to allow the best approach or tools to be chosen. ADmIRE will contribute to the following areas of IR: 1. Domain-specific IR methodologies and 2. IR evaluation. For domain-specific IR, we will develop a framework consisting of a classification scheme for domain-specific problems and a protocol giving the optimal evidence-based approach to solving the problem based on the characterisation. The latter will be obtained by doing systematic reviews of IR papers. In fact, the creation of systematic reviews and hence of guidelines for domain-specific IR is an example of a (rather difficult) domain- specific search problem. We will use this problem as the central scenario for ADmIRE, working on the research and development to create semi-automated tools for IR researchers to create systematic reviews effectively. This scenario therefore simultaneously serves as a model of a domain-specific problem and as a means to find the evidence-based solution. For IR evaluation, ADmIRE will develop guidelines for future IR evaluation campaigns and publications to allow easier and more effective use of the evaluation results to guide decisions in domain-specific search system design, and develop a framework for component-based evaluation based on the workflow paradigm. The framework will being IR evaluation closer to the e-Science paradigm, allowing better sharing of experimental setups between researchers, and more flexibility in performing IR experiments with multiple configurations of components. IR (and potentially other empirically-based areas of computer science) has the advantage of already starting from a position where experimental studies are the cornerstone of the domain. ADmIRE will put together all of this experimental information, organizing it to make use of it in a specific domain, which is a very challenging and yet absolutely necessary task for the development of the field of IR.

The list of search results returned by search engines and the order in which they are presented can have huge impact on the outcomes of peoples decisions. A doctor searching for new treatments for a disease may not find a treatment if the document describing it is not ranked highly enough in the result list, leading to a patient potentially not being given the optimal treatment. An examiner in a patent office missing some prior art during a patent search and hence granting a patent may trigger legal proceedings between two companies. For this reason, it is important that search engines, particularly those used in such professional environments, are free from bias. In the ADmIRE project, we identified various search engine biases and developed methods to overcome these biases. Search engines use various heuristic equations for matching documents to queries. One potential bias is the findability bias, in which a document has particular characteristics that make the likelihood of it being returned for most queries rather small. It is possible to measure the findability bias experimentally by querying a document with a huge collection of queries and seeing how often it is returned. Here we proposed an analytical approach to measuring findability which reduces the extensive experimentation needed. A second potential bias is document length. If a search engine counts how often a query word is present in a document, then longer documents likely have the word present more often. Through recognising that a document can be long either because it talks about many topics (e.g. a collection of short stories), or because it talks about a single topic in a long- winded way (e.g. some textbooks), we were able to propose a method to reduce document length bias in search engine results. The final bias addressed is in evaluating search engines. In order to know how well a search engine functions, it is tested with a number of queries, for which people judge which documents are relevant or not knowing which documents are relevant and retrieved by the search engine means that measures of the effectiveness of a search engine can be made. However, with typically millions of documents searched, it is infeasible for the relevance of every document for every query to be manually judged. Not having all documents manually judged means that the effectiveness measures are biased in some situations. To reduce this bias, we developed approaches to automatically select a collection of documents that should be manually judged in order to reduce the bias as much as possible. Furthermore, approaches that can reduce the bias in the effectiveness measurements given an existing test collection (collection of queries, documents, and manual relevance judgements) were also developed, which allows the hundreds of existing test collections to be used to give less biased measurements.