The Investigation of Convolution Layer Structure on BERT-C-LSTM for Topic Classification of Indonesian News Headlines

Dzakira Fabillah, Rizka Auliarahmi, Siti Dwi Setiarini, Trisna Gelar


An efficient and accurate method for classifying news articles based on their topics is essential for various applications, such as personalized news recommendation systems and market research. Manual classification methods are tedious, prompting the use of deep learning techniques in this study to automate the process. The developed model, BERT-C-LSTM, combines BERT, the convolutional layer from CNN, and LSTM, leveraging their individual strengths. BERT excels at transforming text into context-dependent vector representations, The design of the classification model employs a blend of convolutional layers and LSTM, referred to as C-LSTM. The convolutional layer possesses the capability to extract salient elements, including keywords and phrases, from input data. On the other hand, the Long Short-Term Memory (LSTM) model exhibits the ability to comprehend the temporal context present in sequential data. This study aims to investigate the influence of the convolutional layer structure in BERT-C-LSTM on the classification of Indonesian news headline categorized into eight topics. The results indicate that there are no significant differences in accuracy between BERT-C-LSTM model architectures with a single convolutional layer and multiple parallel convolutional layers and the models using various filter sizes. Furthermore, the BERT-C-LSTM model achieves an accuracy that is not much different from the BERT-LSTM and BERT-CNN models, with accuracies reaching 92.6%, 92.1%, and 92.7%, respectively.


BERT; Convolution Layer; IndoBERT; LSTM; News; Word Embedding

Full Text:



Chen, Y., Perozzi, B., Al-Rfou, R., & Skiena, S. (2013). The Expressive Power of Word Embeddings. 28.

Chowdhury, P., Eumi, E. M., Sarkar, O., & Ahamed, M. F. (2022). Bangla News Classification Using GloVe Vectorization, LSTM, and CNN. Lecture Notes on Data Engineering and Communications Technologies, 95(December 2021), 723–731.

Dong, J., He, F., Guo, Y., & Zhang, H. (2020). A commodity review sentiment analysis based on BERT-CNN model. 2020 5th International Conference on Computer and Communication Systems, ICCCS 2020, 143–147.

Ezen-Can, A. (2020). A Comparison of LSTM and BERT for Small Corpus. 1–12.

Fauzi, M. A. (2018). Automatic Complaint Classification System Using Classifier Ensembles. Telfor Journal, 10(2), 123–128.

González-Carvajal, S., & Garrido-Merchán, E. C. (2020). Comparing BERT against traditional machine learning text classification. Ml.

Ingole, P., Bhoir, S., & Vidhate, A. V. (2018). Hybrid Model for Text Classification. Proceedings of the 2nd International Conference on Electronics, Communication and Aerospace Technology, ICECA 2018, Iaeac, 450–458.

James Mutinda, Mwangi, W., & Okeyo, G. (2023). Sentiment Analysis of Text Reviews Using Lexicon-Enhanced Bert Embedding (LeBERT) Model with Convolutional Neural Network. Appl. Sci, 13(1445).

Kandhro, I. A., Jumani, S. Z., Kumar, K., Hafeez, A., & Ali, F. (2020). Roman Urdu Headline News Text Classification Using RNN, LSTM and CNN. Advances in Data Science and Adaptive Analysis, 12(02), 2050008.

Kaur, K., & Kaur, P. (2023). BERT-CNN: Improving BERT for Requirements Classification using CNN. Procedia Computer Science, 218(2022), 2604–2611.

Koto, F., Rahimi, A., Lau, J. H., & Baldwin, T. (2020). IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP. 757–770.

Li, W., Gao, S., Zhou, H., Huang, Z., Zhang, K., & Li, W. (2019). The automatic text classification method based on bert and feature union. Proceedings of the International Conference on Parallel and Distributed Systems - ICPADS, 2019-Decem, 774–777.

Maslennikova, E. (2019). ELMo Word Representation For News Protection.

Murfi, H., Gowandi, T., Ardaneswari, G., & Nur-, S. (2022). BERT-Based Combination of Convolutional and Recurrent Neural Network for Indonesian Sentiment Analysis. 1–15.

Pogiatzis, A. (2019). NLP: Contextualized word embeddings from BERT.

Safaya, A., Abdullatif, M., & Yuret, D. (2020). KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media. 14th International Workshops on Semantic Evaluation, SemEval 2020 - Co-Located 28th International Conference on Computational Linguistics, COLING 2020, Proceedings, 2054–2059.

Wang, J., Wang, Z., Zhang, D., & Yan, J. (2017). Combining knowledge with deep convolutional neural networks for short text classification. IJCAI International Joint Conference on Artificial Intelligence, 0, 2915–2921.

Widhiyasana, Y., Semiawan, T., Gibran, I., Mudzakir, A., & Noor, M. R. (2021). Penerapan Convolutional Long Short-Term Memory untuk Klasifikasi Teks Berita Bahasa Indonesia (Convolutional Long Short-Term Memory Implementation for Indonesian News Classification). Jurnal Nasional Teknik Elektro Dan Teknologi Informasi |, 10(4), 354–361.

Zhang, Z., Zhao, H., & Wang, R. (2020). Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond.

Zhou, C., Sun, C., Liu, Z., & Lau, F. C. M. (2015). A C-LSTM Neural Network for Text Classification. October 2017.

Zhou, H. (2022). Research of Text Classification Based on TF-IDF and CNN-LSTM. Journal of Physics: Conference Series, 2171(1).



  • There are currently no refbacks.

Copyright (c) 2023 Journal of Software Engineering, Information and Communication Technology (SEICT)

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Journal of Software Engineering, Information and Communicaton Technology (SEICT), 
2774-1699 | p-ISSN:2744-1656) published by Program Studi Rekayasa Perangkat Lunak, Kampus UPI di Cibiru.

 Indexed by.