Perancangan dan Implementasi Arsitektur Data Pipeline Otomatis untuk Analisis Sentimen Ulasan Aplikasi E-Commerce menggunakan Apache Airflow, Docker, dan Ensemble Learning

Abstract

The growth of user reviews on e-commerce platforms in Indonesia is growing much faster than conventional analysis capacity that relies on manual review. This study designs and implements an automated data pipeline based on an ETL (Extract, Transform, Load) architecture that integrates data acquisition through dynamic web scraping (Selenium), Indonesian text preprocessing, sentiment classification using Ensemble Learning models (Multinomial Naive Bayes and Random Forest), workflow orchestration using Apache Airflow, containerization with Docker, and storage of results in a PostgreSQL database. The system is tested using a case study of reviews of three popular e-commerce applications in Indonesia (Shopee, Tokopedia, and Blibli) with a total of more than 30,000 rows of raw review data. The test results show that the pipeline runs successfully end-to-end and automatically, with an idempotency mechanism that successfully maintains data integrity from the risk of duplication. The Ensemble model achieves an overall accuracy of 77.94% using the SMOTE (Synthetic Minority Over-sampling Technique) technique to address class imbalance. However, a per-class evaluation analysis revealed that SMOTE was only effective in improving performance on minority classes with relatively sufficient source data, while failing to provide significant improvements on classes with extreme imbalance. This finding provides a methodological contribution regarding the limitations of oversampling techniques' effectiveness in very limited data and emphasizes the importance of evaluating per-class metrics rather than relying solely on overall accuracy.