Knowledge Delta based improvement and continuous evaluation

Attending industry conferences for information systems (e.g. medical, news, intellectual property domains), it is easy to observe, in the last 2-3 years, a surge in semantic search systems, using artificial intelligence to produce the best results for a variety of work tasks that have an underlying search application. End -users of such systems have no means of assessing the value of these systems, but have to trust the companies offering them. At the same time, companies developing these search based applications have no reliable tools to integrate effectiveness evaluation as part of their testing procedures. The challenge here lies in the fact that while there are numerous benchmarks available in the academic community, there is no quantification of the differences between them. Such a benchmark is typically constituted of a set of documents to be indexed by the search engine (the document collection), a set of queries that simulate user information needs (the query set), and a set of relevance judgements (the qrel set). Changes in any of these, in order for a search system to maintain optimal performance, need to be reflected in changes in the system parameters. But while changes in effectiveness and changes in system parameters are typically easy to observe or measure, changes in the benchmark are currently difficult, if not impossible to measure. Building on the state-of-the-art in representation learning, KoDicare investigates methods to understand changes in the benchmarks beyond the simple term statistics. Significant changes in the document collection or query set need to be quantified at a semantic level. Using such a quantification, which we denote as the Knowledge Delta, we will be able to run ablation studies tests where we change, in a controlled environment, units of Knowledge and observe differences in performance of the search system. The ability to do so has a significant impact both on academic research (providing the means for more controlled experiments in information retrieval) as well as on industry (providing the means to update the search engine if and only if the environment has significantly changed). KoDicare brings together the Research Studios Austria Forschungsgesellschaft, the Laboratoire dInformatique de Grenoble, and Qwant SAS to develop the fundamental theory needed to integrate effectiveness evaluation into future ( semantic) search systems.

Evaluating search systems requires setting up an environment: select a paradigm, metrics, a dataset, etc. The choice of an environment is rarely motivated objectively, and the impact of its variations (choosing a dataset against another, altering one) is rarely measured. Such objectivity comes from a quantifiable understanding of the differences between datasets, documents, or test queries. In Kodicare, we generically call such difference "knowledge delta". Evaluation of several environments, knowing their knowledge deltas, leads to measuring and qualifying "results deltas". Online systems require continuous evaluation with a stable and meaningful environment; which guarantees the reproducibility and explainability of systems results. The environment and result deltas will be able to support such continuous evaluation, and to provide explanations. The theoretical results will be confronted to real cases defined by a French company that deploys a web search engine (Qwant). Scientific and technical challenges: To our knowledge, no such framework dedicated to real continuous evaluation of information retrieval systems exist, due to the numerous parameters that must be handled. The deltas proposed by Kodicare are then a sensible way to tackle this problem. Continuous evaluation is only possible with real cases, which are often difficult to define without the help of web search companies. The large implication of Qwant helped the project define usable scenarios and to test them on a real-life serach engine.