A Study on the Behavior of Crowd Workers when Reporting Forecasts under Proper Scoring Rules

Carvalho, Arthur and Kroon, Denise (2018) A Study on the Behavior of Crowd Workers when Reporting Forecasts under Proper Scoring Rules. American Conference on Information Systems.

[img]
Preview
Text
A Study on the Behavior of Crowd Workers when Reporting Forecasts.pdf

Download (404kB) | Preview

Abstract

The quality of data sources is one of the biggest concerns regarding (big) data analytics. For example, humans can deliberately lie or behave strategically when reporting their beliefs/opinions in surveys and opinion polls, which results in data sets of low reliability. The issue of honest reporting of subjective data can, in theory, be tackled by incentive-compatible methods such as proper scoring rules. We report a study conducted on the crowdsourcing platform Amazon Mechanical Turk that investigates the efficacy of proper scoring rules in inducing workers to honestly report forecasts. Our novel experimental design is able to detect several strategies employed by crowd workers other than truth-telling. Our experimental results, hence, cast a shadow on the usefulness of incentive-compatible methods, such as proper scoring rules, as a way of inducing honest reporting of subjective data and, thus, to ensure data quality.

Item Type: Article
Subjects: H Social Sciences > HB Economic Theory
Divisions: Faculty of Industrial Engineering and Informatics > Information System
Depositing User: staff repository 2
Date Deposited: 31 Aug 2018 18:45
Last Modified: 31 Aug 2018 18:45
URI: http://repository.ittelkom-pwt.ac.id/id/eprint/4573

Actions (login required)

View Item View Item