09. BERT

---

BERT

---

Prerequisites

1. Transformer

Transformer is very important turning point of model with good performance. But at the moment, the focus of task is just translation.

---

What is BERT

1. What is BERT?

• A Model that use encoder-only of transformer model</span>. The main idea is bidirectional method can adapt the new task like self-supervised learning.

• Structure

  \[Input \rightarrow ENCODER \rightarrow Softmax \rightarrow Output\]
• From tasks of Masked Language Model(MLM), Next Sentence Prediction(NSP), it works for training.
• With bidirectional, word embedding is not fixed embedding and to be contextualizal embedding data.

2. Why use BERT?

1. Transformer Encoder
2. Token + Segment + Position Embedding
3. Self-Supervised Learning

Encoder: Bidirectional contextualization

Thanks to Token, OOV issue is almost solved and even though there’re typo issue, it will be tokenalization of almost word or use UNK token that can guess contextual what kind of a word.

3. How use Transformer?

1. Token + Segment + Position Embedding 
2. Transformer Encoder

$Input$

• $Token\; embedding$: a pre-trained word embedding (WordPiece)
  • With special token: [CLS], [SEP]
    • [CLS]: Classificaiton toekn, put always at the beginning. Final hidden state for this token is used as the aggregate sequence representation for classification tasks
    • [SEP]: Seperator token, used to mark the end of a sentence.
• $Segment\; embedding$: A learned embedding indicating which sentence each token belongs to
• $Position\; embedding$: a learned embedding for each position

$Transformer\; Encoder$

BERT uses the Transformer encoder to learn bidirectional language representations that function as contextual embeddings. And the main idea of how to make contextualizaion functions. With the MLM task, the BERT learn embedding and contextualization weights. It can use not only by itself but also fine tunning with another models.

The model is not difference via transformer. But the contribution of BERT is that this perspective change the NLP paradigm. Especially MLM task is very efficient and easily collect the data from anywhere without human annotation. In other word, Not a new architecture BUT a new training framwork with transformer.