Probabilistic matrix factorization with active learning for quality assurance in crowdsourcing systems

Abstract

In crowdsourcing systems, tasks are distributed to networked people to complete such that a company's production cost can be greatly reduced. Currently, quality assurance in crowdsourcing systems highly relies on redundancy of answers provided by multiple workers with varying expertise; however massive redundancy is very expensive and time-consuming. Active learning is a learning approach to achieve certain accuracy with a very low cost, and thus a number of previous works adopted active learning in crowdsourcing systems for quality assurance. However, previous works do not consider the varying expertise of workers for various task categories in real crowdsourcing scenarios. In our paper, we propose ActivePMF (Probabilistic Matrix Factorization with Active Learning) for TaskRec (a task recommendation framework) to learn a factor analysis model for quality assurance in crowdsourcing systems. PMF is the state-of-the-art approach for recommendation systems. Our model considers not only worker task selection preference (how often a worker accepting similar tasks), but also worker performance history (how often a worker's work done on similar tasks getting accepted). It first randomly assigns all new tasks to the most reliable worker in the task category. Next, it actively selects the most uncertain task for the most reliable workers to work on to retrain the classification model. Complexity analysis shows that our model is efficient and is scalable to large datasets. Based on experiments on real-world datasets, compared with random selection on task and worker to the PMF learning model, our ActivePMF can greatly improve both MAE and RMSE performance (up to 38% improvement in MAE and up to 25% improvement in RMSE). To the best of our knowledge, we are the first one to use PMF with active learning to recommend tasks for quality assurance in crowdsourcing systems. © 2015.