Embedded Human Computation for Knowledge Extraction and Evaluation

The rapid growth and fragmented character of social media and publicly available structured data challenges established approaches to knowledge extraction. Many algorithms fail when they encounter noisy, multilingual and contradictory input. Efforts to increase the reliability and scalability of these algorithms face a lack of suitable training data and gold standards. Given that humans excel at interpreting contradictory and context-dependent evidence, the uComp project will address the above mentioned shortcomings by merging collective human intelligence and automated methods in a symbiotic fashion. The project will build upon the emerging field of Human Computation (HC) in the tradition of games with a purpose and crowdsourcing marketplaces. It will advance the field of Web Science by developing a scalable and generic HC framework for knowledge extraction and evaluation, delegating the most challenging tasks to large communities of users and continuously learning from their feedback to optimise automated methods as part of an iterative process. A major contribution is the proposed foundational research on Embedded Human Computation (EHC), which will advance and integrate the currently disjoint research fields of human and machine computation. EHC goes beyond mere data collection and embeds the HC paradigm into adaptive knowledge extraction workflows. An open evaluation campaign will validate the accuracy and scalability of EHC to acquire factual and affective knowledge. In addition to novel evaluation methods, uComp will also provide shared datasets and benchmark EHC against established knowledge processing frameworks. While uComp methods will be generic and evaluated across domains, climate change was chosen as the main use case for its challenging nature, subject to fluctuating and often conflicting interpretations. Collaborating with international organisations such as EEA, NOAA and NASA will increase impact, provide a rich stream of input data, attract and retain a critical mass of users, and promote the adoption of EHC among a wide range of stakeholders.

The rapid growth and fragmented character of social media and publicly available structured data challenges established approaches to knowledge extraction. Many algorithms fail when they encounter noisy, multilingual and contradictory input. Efforts to increase the reliability and scalability of these algorithms face a lack of suitable training data and gold standards. Given that humans excel at interpreting contradictory and context-dependent evidence, the uComp project addresses the above-mentioned shortcomings by merging collective human intelligence and automated methods in a symbiotic fashion. The project advanced the field of Human Computation (HC), using a combination of games with a purpose and crowdsourcing marketplaces such as CrowdFlower. At the same time, uComp contributed to Web Science by developing a scalable and generic HC framework for knowledge extraction and evaluation, delegating the most challenging tasks to large user communities and continuously learning from their feedback to optimise automated methods as part of an iterative process.uComp developed an Embedded Human Computation (EHC) framework, which goes beyond mere data collection and embeds the HC paradigm into adaptive knowledge extraction workflows. The project validated the accuracy and scalability of EHC to acquire factual and affective knowledge - while uComp methods are generic and applicable across domains, the topic of climate change was chosen as the main use case for its challenging nature, subject to fluctuating and often conflicting interpretations.Active collaboration between MODUL University Viennas Department of New Media Technology and international partner organisations had a positive impact on the achievable scientific progress, and helped to increase the international visibility of the project. The list of partner organisations include the core partners of the uComp project itself, the consortium of DecarboNet.eu, a European research initiative within the EU 7th Framework Programme, the Climate Program Office of the U.S. National Oceanic & Atmospheric Administration (NOAA), and the United Nations Environment Programme (UNEP).Three public Web applications showcased the results of this collaborative work: The uComp Language Quiz to acquire multilingual language resources, the (2) Climate Challenge to gather social perceptions and create collective awareness of environmental issues, and the (3) Media Watch on Climate Change as a central knowledge repository including a visual dashboard to analyse the extracted factual and affective knowledge. These applications provided a rich stream of input data, attracted and engaged a critical mass of participants, and promoted the adoption of EHC among a wide range of stakeholders.