Embedded Human Computation for Knowledge Extraction and Evaluation

The rapid growth and fragmented character of social media and publicly available structured data challenges established approaches to knowledge extraction. Many algorithms fail when they encounter noisy, multilingual and contradictory input. Efforts to increase the reliability and scalability of these algorithms face a lack of suitable training data and gold standards. Given that humans excel at interpreting contradictory and context-dependent evidence, the uComp project will address the above mentioned shortcomings by merging collective human intelligence and automated methods in a symbiotic fashion. The project will build upon the emerging field of Human Computation (HC) in the tradition of games with a purpose and crowdsourcing marketplaces. It will advance the field of Web Science by developing a scalable and generic HC framework for knowledge extraction and evaluation, delegating the most challenging tasks to large communities of users and continuously learning from their feedback to optimise automated methods as part of an iterative process. A major contribution is the proposed foundational research on Embedded Human Computation (EHC), which will advance and integrate the currently disjoint research fields of human and machine computation. EHC goes beyond mere data collection and embeds the HC paradigm into adaptive knowledge extraction workflows. An open evaluation campaign will validate the accuracy and scalability of EHC to acquire factual and affective knowledge. In addition to novel evaluation methods, uComp will also provide shared datasets and benchmark EHC against established knowledge processing frameworks. While uComp methods will be generic and evaluated across domains, climate change was chosen as the main use case for its challenging nature, subject to fluctuating and often conflicting interpretations. Collaborating with international organisations such as EEA, NOAA and NASA will increase impact, provide a rich stream of input data, attract and retain a critical mass of users, and promote the adoption of EHC among a wide range of stakeholders.

Information systems that utilize artificial intelligence and semantic Web technologies are increasingly common, with a plethora of application domains. Such systems typically require a critical mass of training data to infer patterns and optimize their results. For providing such training data and to verify data generated by machine algorithms, human effort is still needed in many cases. Using human domain experts is typically very expensive and not scalable, therefore crowdsourcing techniques are used to tap into the collective intelligence of large user communities.Within the uComp project, the WU research team at WU worked on the integration of humans into knowledge creation and knowledge verification work flows. We designed, implemented and evaluated a plugin for the Protégé ontology editor, which we then used to measure the aspects of feasibility and scalability of using crowdsourcing in knowledge acquisition. We studied various setups to make effective use of crowd workers, and compared the results from crowd workers to results of human experts. To obtain generalizable results, we operated in multiple domains of knowledge, and utilized data sources in various languages. Furthermore, WU improved the underlying system to create knowledge structures by adding support for learning from heterogeneous sources of data (for example unstructured data from news media and social media) and storing the results in a semantic knowledge base. We conducted research into optimizing and balancing systems that process redundant and potentially conflicting input data, including quality control processes for content acquisition and human computation.