Optimasi Klasifikasi Hate Speech dan Offensive Language melalui Frozen RoBERTa Feature Extraction dan Random Forest
DOI:
https://doi.org/10.47065/bulletincsr.v6i4.1157Keywords:
HASOC 2021; Hate Speech; Offensive Content; RoBERTa; Random Forest; Handcrafted FeaturesAbstract
Hate speech and offensive content detection on social media remains a significant challenge in Natural Language Processing (NLP) due to the characteristics of Twitter data, which are typically short, informal, and contain various elements such as mentions, URLs, hashtags, and emotional expressions that complicate the classification process. End-to-end Transformer fine-tuning approaches generally require substantial computational resources; therefore, this study explores a more computationally efficient approach by utilizing RoBERTa as a frozen feature extractor combined with Random Forest as the classifier. This approach enables the exploitation of contextual representations generated by Transformer models without requiring full model retraining.The study employs the HASOC 2021 English Track dataset, which consists of two classification tasks: Task A for binary classification (HOF and NOT) and Task B for multi-class classification (HATE, OFFN, PRFN, and NONE). The classification pipeline is optimized through the incorporation of handcrafted features, oversampling, Random Forest hyperparameter tuning, and threshold tuning in specific scenarios. Model performance is evaluated using accuracy, precision, recall, and F1-macro, with F1-macro serving as the primary metric due to class imbalance. The best-performing model achieved an F1-macro score of 0.80 on Task A and 0.64 on Task B. These results indicate that the combination of frozen RoBERTa representations and Random Forest provides strong performance for binary hate speech and offensive content classification. However, the performance on Task B highlights the difficulty of distinguishing linguistically similar categories, such as HATE, OFFN, and PRFN, suggesting that fine-grained multi-class classification remains a challenging task. Overall, the findings indicate that RoBERTa-based frozen feature extraction constitutes a computationally efficient alternative for hate speech detection on English Twitter data, although further improvements are required to enhance performance in multi-class classification settings.
Downloads
References
S. Dharmawan, V. C. Mawardi, and N. J. Perdana, “Klasifikasi Ujaran Kebencian Menggunakan Metode FeedForward Neural Network (IndoBERT),” Jurnal Ilmu Komputer dan Sistem Informasi, vol. 11, no. 1, Jun. 2023, doi: 10.24912/jiksi.v11i1.24066.
M. O. Ibrohim and I. Budi, “Hate speech and abusive language detection in Indonesian social media: Progress and challenges,” Aug. 01, 2023, Elsevier Ltd. doi: 10.1016/j.heliyon.2023.e18647.
E. W. Pamungkas, D. G. P. Putri, and A. Fatmawati, “Hate Speech Detection in Bahasa Indonesia: Challenges and Opportunities,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 6, 2023, doi: 10.14569/IJACSA.2023.01406125.
M. Zampieri et al., “SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020),” in Proceedings of the Fourteenth Workshop on Semantic Evaluation, Stroudsburg, PA, USA: International Committee for Computational Linguistics, Sep. 2020, pp. 1425–1447. doi: 10.18653/v1/2020.semeval-1.188.
P. Alonso, R. Saini, and G. Kovacs, “TheNorth at SemEval-2020 Task 12: Hate Speech Detection Using RoBERTa,” in Proceedings of the Fourteenth Workshop on Semantic Evaluation, Stroudsburg, PA, USA: International Committee for Computational Linguistics, 2020, pp. 2197–2202. doi: 10.18653/v1/2020.semeval-1.292.
B. Ray and A. Garain, “JU at HASOC 2020: Deep Learning with RoBERTa and Random Forest for Hate Speech and Offensive Content Identification in Indo-European Languages,” 2020. [Online]. Available: http://ceur-ws.org
F. Barbieri, J. Camacho-Collados, L. Espinosa Anke, and L. Neves, “TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification,” in Findings of the Association for Computational Linguistics: EMNLP 2020, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 1644–1650. doi: 10.18653/v1/2020.findings-emnlp.148.
G. Kaplun, A. Gurevich, T. Swisa, M. David, S. Shalev-Shwartz, and E. Malach, “Less is More: Selective Layer Finetuning with SubTuning,” Jul. 2023. [Online]. Available: http://arxiv.org/abs/2302.06354
M. Mosbach, T. Pimentel, S. Ravfogel, D. Klakow, and Y. Elazar, “Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation,” May 2023. [Online]. Available: http://arxiv.org/abs/2305.16938
O. Galal, A. H. Abdel-Gawad, and M. Farouk, “Federated Freeze BERT for text classification,” J. Big Data, vol. 11, no. 1, p. 28, Feb. 2024, doi: 10.1186/s40537-024-00885-x.
U. Iftikhar, S. F. Ali, G. Mustafa, N. Bahar, and K. Ishaq, “Beyond words: a hybrid transformer-ensemble approach for detecting hate speech and offensive language on social media,” PeerJ Comput. Sci., vol. 11, p. e3214, Oct. 2025, doi: 10.7717/peerj-cs.3214.
Y. Zhang, V. Hangya, and A. Fraser, “A Study of the Class Imbalance Problem in Abusive Language Detection,” in Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), Stroudsburg, PA, USA: Association for Computational Linguistics, 2024, pp. 38–51. doi: 10.18653/v1/2024.woah-1.4.
U. Iftikhar, S. F. Ali, G. Mustafa, N. Bahar, and K. Ishaq, “Beyond words: a hybrid transformer-ensemble approach for detecting hate speech and offensive language on social media,” PeerJ Comput. Sci., vol. 11, p. e3214, Oct. 2025, doi: 10.7717/peerj-cs.3214.
J. A. Ortiz-Zambrano, C. H. Espín-Riofrío, and A. Montejo-Ráez, “Deep Encodings vs. Linguistic Features in Lexical Complexity Prediction,” Neural Comput. Appl., vol. 37, no. 3, pp. 1171–1187, Jan. 2025, doi: 10.1007/s00521-024-10662-9.
I. Mollas, Z. Chrysopoulou, S. Karlos, and G. Tsoumakas, “ETHOS: a multi-label hate speech detection dataset,” Complex & Intelligent Systems, vol. 8, no. 6, pp. 4663–4678, Dec. 2022, doi: 10.1007/s40747-021-00608-2.
S. Modha et al., “Overview of the HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages and Conversational Hate Speech,” in Forum for Information Retrieval Evaluation, New York, NY, USA: ACM, Dec. 2021, pp. 1–3. doi: 10.1145/3503162.3503176.
M. I. Prasetiyowati, N. U. Maulidevi, and K. Surendro, “Feature selection to increase the random forest method performance on high dimensional data,” International Journal of Advances in Intelligent Informatics, vol. 6, no. 3, p. 303, Nov. 2020, doi: 10.26555/ijain.v6i3.471.
B. W. Lee, Y. S. Jang, and J. Lee, “Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2021, pp. 10669–10686. doi: 10.18653/v1/2021.emnlp-main.834.
M. Mujahid et al., “Data oversampling and imbalanced datasets: an investigation of performance for machine learning and feature engineering,” J. Big Data, vol. 11, no. 1, p. 87, Jun. 2024, doi: 10.1186/s40537-024-00943-4.
T. Mandla et al., “Overview of the HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages,” Aug. 2021, doi: 10.1007/978-3-030-73696-5_6.
M. Sivakumar, S. Parthasarathy, and T. Padmapriya, “Trade-off between training and testing ratio in machine learning for medical image processing,” PeerJ Comput. Sci., vol. 10, p. e2245, Sep. 2024, doi: 10.7717/peerj-cs.2245.
Y. Xu and R. Goodacre, “On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning,” J. Anal. Test., vol. 2, no. 3, pp. 249–262, Jul. 2018, doi: 10.1007/s41664-018-0068-2.
CardiffNLP, “cardiffnlp/twitter-roberta-base-offensive,” CardiffNLP. [ Online]. Available: https://huggingface.co/cardiffnlp/twitter-roberta-base-offensive
F. Barbieri, J. Camacho-Collados, L. Espinosa Anke, and L. Neves, “TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification,” in Findings of the Association for Computational Linguistics: EMNLP 2020, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 1644–1650. doi: 10.18653/v1/2020.findings-emnlp.148.
K. Gémes, Á. Kovács, M. Reichel, and G. Recski, “Offensive text detection on English Twitter with deep learning models and rule-based systems,” 2021. [Online]. Available: https://github.com/GKingA/tuw-inf-hasoc2021.
H. Ma’rifah, A. P. Wibawa, and M. I. Akbar, “Klasifikasi Artikel Ilmiah Dengan Berbagai Skenario Preprocessing,” Sains, Aplikasi, Komputasi dan Teknologi Informasi, vol. 2, no. 2, p. 70, Apr. 2020, doi: 10.30872/jsakti.v2i2.2681.
Arif Bijaksana Putra Negara, “The Influence Of Applying Stopword Removal And Smote On Indonesian Sentiment Classification,” Lontar Komputer?: Jurnal Ilmiah Teknologi Informasi, vol. 14, no. 03, pp. 172–185, Oct. 2025, doi: 10.24843/LKJITI.2023.v14.i03.p05.
Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” Jul. 2019, [Online]. Available: http://arxiv.org/abs/1907.11692
M. I. Alfarizi, L. Syafaah, and M. Lestandy, “Emotional Text Classification Using TF-IDF (Term Frequency-Inverse Document Frequency) And LSTM (Long Short-Term Memory),” JUITA?: Jurnal Informatika, vol. 10, no. 2, p. 225, Nov. 2022, doi: 10.30595/juita.v10i2.13262.
N. Arifin, U. Enri, and N. Sulistiyowati, “Penerapan Algoritma Support Vector Machine (SVM) dengan TF-IDF N-Gram untuk Text Classification,” STRING (Satuan Tulisan Riset dan Inovasi Teknologi), vol. 6, no. 2, p. 129, Dec. 2021, doi: 10.30998/string.v6i2.10133.
R. M. O. Cruz, W. V. de Sousa, and G. D. C. Cavalcanti, “Selecting and combining complementary feature representations and classifiers for hate speech detection,” Jan. 2022, [Online]. Available: http://arxiv.org/abs/2201.06721
S. F. Taskiran, B. Turkoglu, E. Kaya, and T. Asuroglu, “A comprehensive evaluation of oversampling techniques for enhancing text classification performance,” Sci. Rep., vol. 15, no. 1, p. 21631, Jul. 2025, doi: 10.1038/s41598-025-05791-7.
A. Demircio?lu, “Applying oversampling before cross-validation will lead to high bias in radiomics,” Sci. Rep., vol. 14, no. 1, p. 11563, May 2024, doi: 10.1038/s41598-024-62585-z.
A. Glazkova, “A Comparison of Synthetic Oversampling Methods for Multi-class Text Classification,” Aug. 2020. [Online]. Available: http://arxiv.org/abs/2008.04636
S. Maldonado, C. Vairetti, A. Fernandez, and F. Herrera, “FW-SMOTE: A feature-weighted oversampling approach for imbalanced classification,” Pattern Recognit., vol. 124, p. 108511, Apr. 2022, doi: 10.1016/j.patcog.2021.108511.
R. Li, M. Liu, D. Xu, J. Gao, F. Wu, and L. Zhu, “A Review of Machine Learning Algorithms for Text Classification,” 2022, pp. 226–234. doi: 10.1007/978-981-16-9229-1_14.
P. Probst, M. N. Wright, and A. Boulesteix, “Hyperparameters and tuning strategies for random forest,” WIREs Data Mining and Knowledge Discovery, vol. 9, no. 3, May 2019, doi: 10.1002/widm.1301.
S. Szeghalmy and A. Fazekas, “A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning,” Sensors, vol. 23, no. 4, p. 2333, Feb. 2023, doi: 10.3390/s23042333.
I. A. Rahma and L. H. Suadaa, “Penerapan Text Augmentation untuk Mengatasi Data yang Tidak Seimbang pada Klasifikasi Teks Berbahasa Indonesia,” Jurnal Teknologi Informasi dan Ilmu Komputer, vol. 10, no. 6, pp. 1329–1340, Dec. 2023, doi: 10.25126/jtiik.2023107325.
Febiana Anistya and Erwin Budi Setiawan, “Hate Speech Detection on Twitter in Indonesia with Feature Expansion Using GloVe,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 5, no. 6, pp. 1044–1051, Dec. 2021, doi: 10.29207/resti.v5i6.3521.
S. Riyanto, I. S. Sitanggang, T. Djatna, and T. D. Atikah, “Comparative Analysis using Various Performance Metrics in Imbalanced Data for Multi-class Text Classification,” 2023. [Online]. Available: http://gcancer.org/pdr
S. Saseendran and S. Giri, “Classification of Hate Speech and Offensive Content using an approach based on DistilBERT,” 2021. [Online]. Available: https://github.com/sharan0276
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Optimasi Klasifikasi Hate Speech dan Offensive Language melalui Frozen RoBERTa Feature Extraction dan Random Forest
ARTICLE HISTORY
How to Cite
Issue
Section
Copyright (c) 2026 Marsha Cahyani Dwisyakilla, Surya Agustian, Novriyanto Novriyanto, Muhammad Affandes
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
