Bert

BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model introduced by Google's research team. It has significantly advanced the state-of-the-art in various NLP tasks. Unlike traditional language models, BERT is bidirectional, meaning it considers both left and right contexts of words. The core architecture of BERT is based on the Transformer model, using self-attention mechanisms to capture long-range dependencies.

BERT is pre-trained on two tasks: masked language modeling (MLM) and next sentence prediction (NSP). This approach allows BERT to excel in tasks like text classification, named entity recognition, question answering, and sentiment analysis.

The architecture of BERT along with overall pretraining and finetuning procedures
The architecture of BERT along with overall pretraining and finetuning procedures.

Data

The following table summarizes the data used to train BERT:

TypeCount
Total words2.2 billion
Unique words22.94 million
Total sentences181.45 million
Total documents17.52 million

Training

BERT was trained using the following hyperparameters:

Hyper-parameterValue
vocab_size50,000
hidden_size768
num_hidden_layers12
num_attention_heads12
learning_rate0.00005
batch_size110

Results

After training, BERT achieved the following results:

MetricValue
Training loss1.8633
Evaluation loss1.4681
Perplexity4.3826
NER Precision0.8475
NER Recall0.7390
NER Macro F10.7786
POS Precision0.8838
POS Recall0.8543
POS Macro F10.8611

Deliverable Files

The deliverables for BERT include the following files:

  • config.json - JSON file with model configurations
  • pytorch_model.bin - Model weights in PyTorch format
  • special_tokens_map.json - Token information for the tokenizer
  • tokenizer_config.json - Tokenizer configurations
  • vocab.txt - Vocabulary file