Anomaly detection in consumer review analytics for idea generation in product innovation: Comparing machine learning and deep learning techniques

Abstract

With the development of big data analytics, consumers' online reviews are becoming increasingly useful for product innovation with hidden innovative ideas that can be extracted. However, these ideas may be only hidden in a small part of the massive reviews. This study aims to investigate the potential of using anomaly detection technology to identify unique reviews for more effective innovation generation. Three classical anomaly detection approaches (including both machine and deep learning) were explored, namely, isolation forest, density-based cluster analysis, and autoencoder methods. Using the consumer reviews on Dyson Vacuum cleaner from Xiaohongshu (one of the most popular social media platforms in China), we tested and compared the application of these three approaches in detecting innovation-relevant reviews. The results indicate that the two machine learning approaches, aka., density-based cluster analysis and isolation forest are too sensitive to the length of the reviews. The deep learning method, autoencoder, on the contrary, shows good stability and capability to detect the unique reviews from the whole dataset. Furthermore, the experts’ rating also confirms the effectiveness of autoencoder in identifying innovation-relevant reviews. Future studies and implications are then discussed.