Please use this identifier to cite or link to this item:
Title: Probabilistic matrix factorization with active learning for quality assurance in crowdsourcing systems
Authors: Dr. YUEN Man-Ching, Connie 
King, Irwin 
Prof. LEUNG Kwong Sak 
Issue Date: 2015
Publisher: IADIS
Source: Proceedings of the 14th International Conference WWW/Internet , 2015 , pp. 127-138
Conference: Proceedings of the 14th International Conference WWW/Internet 2015 
Abstract: In crowdsourcing systems, tasks are distributed to networked people to complete such that a company's production cost can be greatly reduced. Currently, quality assurance in crowdsourcing systems highly relies on redundancy of answers provided by multiple workers with varying expertise; however massive redundancy is very expensive and time-consuming. Active learning is a learning approach to achieve certain accuracy with a very low cost, and thus a number of previous works adopted active learning in crowdsourcing systems for quality assurance. However, previous works do not consider the varying expertise of workers for various task categories in real crowdsourcing scenarios. In our paper, we propose ActivePMF (Probabilistic Matrix Factorization with Active Learning) for TaskRec (a task recommendation framework) to learn a factor analysis model for quality assurance in crowdsourcing systems. PMF is the state-of-the-art approach for recommendation systems. Our model considers not only worker task selection preference (how often a worker accepting similar tasks), but also worker performance history (how often a worker's work done on similar tasks getting accepted). It first randomly assigns all new tasks to the most reliable worker in the task category. Next, it actively selects the most uncertain task for the most reliable workers to work on to retrain the classification model. Complexity analysis shows that our model is efficient and is scalable to large datasets. Based on experiments on real-world datasets, compared with random selection on task and worker to the PMF learning model, our ActivePMF can greatly improve both MAE and RMSE performance (up to 38% improvement in MAE and up to 25% improvement in RMSE). To the best of our knowledge, we are the first one to use PMF with active learning to recommend tasks for quality assurance in crowdsourcing systems. © 2015.
Type: Conference Paper
ISBN: 978-989853344-9
Appears in Collections:Publication

Show full item record

Page view(s)

checked on Jan 3, 2024

Google ScholarTM

Impact Indices



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.