Story Comprehension Using Crowdsourced Commonsense Knowledge

Abstract

This thesis examines the problem of commonsense knowledge acquisition and the application of this knowledge to automated story understanding. Lately, a number of researchers and institutions focused their efforts to gather commonsense knowledge as an essential component for developing "intelligent" machines. The approach taken is that knowledge appropriate for story understanding can be gathered by sourcing the task to the crowd, using both intrinsic and extrinsic methods for knowledge acquisition. The proposed methodology centers on breaking this task into a sequence of more specific tasks, so that human participants not only identify relevant knowledge, but also convert it into a machine-readable form and evaluate its applicability to story understanding tasks, such as question answering. We propose and investigate methods for the acquisition and application of commonsense knowledge, employing techniques for the representation, reasoning and retrieval of commonsense knowledge established by other researchers in the field.

The work in this thesis begins with the presentation of a literature review on the current state of affairs on automated story understanding, commonsense knowledge acquisition and appropriate representations of the acquired knowledge. A number of systems are presented, focusing on the ones that use human computation or crowdsourcing as a method for acquiring knowledge. The reader is also introduced to computational argumentation which is an appropriate substrate for representing knowledge. Argumentation semantics are used for representing knowledge and reasoning with it in the internal mechanisms of all the developed tools.

We present a tool for helping users to encode a story and to manually add knowledge rules in a way that machines can understand them. This tool is a Web-based Integrated Development Environment called "Web-STAR", that helps both expert and non-expert users in encoding stories in symbolic form and adding background knowledge. The tool also provides a number of embedded utilities for converting natural language stories to symbolic format, visually adding knowledge using a directed graph editor and promoting user collaboration. The output is presented both textually and graphically in a timeline format, where users can follow the comprehension model of a story and track changes in the story timeline. The IDE was evaluated for its ease of use both by expert and non-expert users, following user experience measurement methodologies and it received a high score in its evaluation.

Next we present a novel framework and platform we have developed for implementing crowdsourcing applications (e.g., Games with a Purpose or language learning applications) that can be used by human workers for gathering commonsense knowledge. We designed and executed two experiments that examine whether fully automated or hybrid crowdsourcing techniques, i.e., techniques that benefit from both manually, crowd-contributed and automatic acquisition of knowledge, can be used to gather commonsense knowledge. The first application, a Game With A Purpose (GWAP) called "Knowledge Coder" relied only on crowdsourcing approaches to acquire knowledge. The second application, again a GWAP called \enquote{Robot Trainer}, was designed using a hybrid methodology for gathering background knowledge, generalizing it and evaluating its appropriateness in answering questions on unseen stories. The acquired knowledge was tested on story comprehension tasks such as question answering and the results show that the gathered knowledge is useful in answering story questions on new unseen stories, since the gathered knowledge is applicable in different domains.

We also study the problem of inferring the geographic focus of a story at a country level, i.e., the geographic location that the story is related to. We developed an application for inferring the geographic focus of stories using crowdsourced knowledge bases, contributing in understanding the "Where" a story takes place type of question. This application, called "Geo-Mantis" retrieves knowledge from popular crowdsourced knowledge bases, such as ConceptNet and YAGO and returns a prediction of the country of focus. Furthermore, an expansion of this application was developed to apply a crowdsourced strategy for this task. Crowd-workers evaluated the usefulness of the arguments supporting a specific country on identifying the geographic focus of a document and the evaluated arguments were tested for identifying the geographic focus.

The thesis concludes with a discussion of the outcome of the conducted experiments on the Web-STAR IDE, the GWAPs for acquiring commonsense knowledge and the application of crowdsourced knowledge for geographic focus identification, highlighting the different contributions in the area of commonsense knowledge acquisition and its application in automated story understanding.