What is a RoBERTa model?

Key takeaways:

• RoBERTa is an optimized variant of BERT developed by Facebook and the University of Washington in 2019.

• Built on the transformer model, it uses a similar framework to BERT but incorporates enhancements in training methods.

• Key improvements

  • Dynamic masking: It uses different masked tokens in each training epoch, unlike BERT’s static masking.

  • No NSP task: It eliminates the next sentence prediction task for better performance.

  • Large batch size: It is trained with a mini-batch size of 8000 sequences, compared to BERT's 256.

  • Increased training data: It is pretrained on around 160 GB of text, significantly more than BERT's 16 GB.

  • BBPE tokenizer: It utilizes a byte-level byte pair encoding with a 50,000 vocabulary size, larger than BERT's 30,000.

• RoBERTa outperforms BERT on various NLP tasks, including text classification and sentiment analysis, due to its optimized training techniques.

The researchers from Facebook and the University of Washington discovered that the BERT model is not fully optimized, which indicated a potential for a better-performing model by improving the pretraining methods. As a result, Facebook developed the Robustly Optimized BERT Pretraining Approach (RoBERTa), a popular variant of Bidirectional Encoder Representations from Transformers (BERT) in 2019. This state-of-the-art language model is built on the architecture of BERT with slight differences in the training procedures.

Working of RoBERTa

RoBERTa is built on the transformer model architecture. Let’s see the architecture diagram of RoBERTa to see how it works.

• Step 1: Our input sentence is converted into the tokens and a CLS token is added at the start of the sentence to represent the beginning of the sentence.

• Step 2: Tokens are further converted into embeddings using the embedding models.

• Step 3: These embeddings are then passed as input to the RoBERTa model, which returns the result.

Pretraining the RoBERTa model

The following are techniques used to pretrain the RoBERTa model:

• Dynamic masking

• Removing NSP task

• Large batch size

• Increased training data

• BBPE tokenizer

Now let’s see in detail how RoBERTa’s training process differs from that of BERT, which optimizes the RoBERTa model and makes it outperform BERT on many NLP tasks such as text classification, sentiment analysis, question-answering, etc.

Dynamic masking

Static masking is a technique that trains the model to predict the same masked token of a sentence in every epoch. Unlike static masking, which is used by the BERT model, the RoBERTa model uses dynamic masking, which works by masking different tokens of the sentence in every epoch.

Removing NSP task

The BERT model is trained on the next sentence prediction (NSP) task, which uses the pair of sentences to predict whether the first sentence is a follow-up of the second sentence. Researchers observed that training the BERT without the NSP task slightly improves the model performance. Hence, RoBERTa is pretrained using the FULL-SENTENCESThe FULL-SENTENCES technique for pretraining a model involves training the model using complete sentences that are sampled continuously from one or more documents, rather than using sentence pairs. without the NSP task. The maximum token length is 512, and once we reach the end of a document, we continue sampling from the next document.

Large batch size

RoBERTa is pretrained on a mini-batch size of 8000 sequences for three million steps. This is larger than that of BERT, which uses a batch size of 256 sequences for one million steps to increase the model performance and speed.

Increased training data

RoBERTa is pretrained on the following five datasets:

• Toronto BookCorpus

• English Wikipedia

• Common Crawl-News (CC-News)

• Open WebText

• Stories (a subset of Common Crawl)

This training data is 10 times larger (around 160 GB of text data) than that of BERT (16 GB), which results in a better performance of the model.

BBPE tokenizer

BERT uses the character-level BPE with a 30,000 vocabulary size, whereas RoBERTa uses a byte-level byte pair encoding (BBPE) with a vocabulary size of 50,000, which is larger than BERT.

Quiz

Let's take a short quiz to test your understanding of the RoBERTa model.

Conclusion

In conclusion, RoBERTa model is a powerful and optimized variant of BERT, achieving state-of-the-art results in various NLP tasks. The introduction of dynamic masking, removal of the NSP task, larger batch size, increased training data and BBPE tokenizer—all of these improvements contributed to its outstanding performance on various NLP tasks.