Use of Bidirectional Encoder Representations from Transformers (BERT) and Robustly Optimized Bert Pretraining Approach (RoBERTa) for Nepali News Classification

Abstract

News classification is a technique of classifying news documents into predefined groups. One of the earliest problems in Natural Language Processing was the categorization of news. Huge number of news is generated from different news portals each day and it is difficult to consign the specific types of news from that portal. News must be assigned into respective appropriate classes as users want to read certain type of news automatically as per the need. Text classification has been done by using different machine learning algorithm like Support Vector Machine (SVM), Long Short-Term Memory (LSTM). However, Bidirectional Encoder Representations from Transformers (BERT) and Robustly Optimized Bert Pretraining Approach (RoBERTa) have not been fully scrutinized for Nepali news classification tasks. This research develops two models for Nepali news classification namely BERT and RoBERTa by collecting news data from various national news portals. Precision, Recall, F1 score and accuracy are used to evaluate the effectiveness of the model. Both models are trained and tested with AdamW optimizer with learning rate 1e-5 i.e., 0.0001. While comparing both models, RoBERTa found to be better than BERT model with accuracy 95.3 percent.