07. Attention

---

Attention

---

Prerequisites

1. Recurrent Neural Network
2. Seq2Seq

Recurrent Neural Network have a problem that have rough 1:1 aligment. But machine translation breaks these assumptions. input/output lengths differ, different order, a target word may depend on a far-away source word.

Seq2Seq have a problem that have bottleneck of encoder. If encoder have long sequences, it is not sufficient. It still suffer from treating extremly long sequences.

---

What is Attention

1. What is Attention?

• A Model that use Seq2Seq model but additional method that take into account hidden states at all input steps with closer attention on more relevant input tokens</span>.

• Structure

  \[Input \rightarrow ENCODER \rightarrow DECODER \rightarrow Softmax \rightarrow Output\]
• Can be free bottleneck of decoder hidden state
• Instead of compressing everything into one vector, Decoder attends to all encoder hidden states.

2. Why use Attention?

All Encoder Hidden States

Each output token focuses on different input.

Attention map visualizes alignment.

3. How use Attention?

1. Query(context)
2. Value(references)
3. Key(references)
4. Attention value

$ Query $: the decoder hidden state at the t

$ Keys $: the encoder hidden state at all times

$ Values $: the encoder hidden state at all times

$ Attention Value $: the result of Attention(Q,K,V)

\(\text{Attention}(Q, K, V) = \text{weighted sum of } V\)

Step 1: Define Hidden States

Encoder hidden states: $ h_1, h_2, …, h_T \in \mathbb{R}^d $,
Decoder state at time t: $ s_t \in \mathbb{R}^d $

“Decoder state at time t ($s_t$)” is Query

Step 2: Compute Attention Scores

Score between decoder state and each encoder state: $ e_{t,i} = s_t^T h_i $

Vector form:

\[e_t = \begin{bmatrix} s_t^T h_1 \\ s_t^T h_2 \\ \vdots \\ s_t^T h_T \end{bmatrix} \in \mathbb{R}^T\]

“Each encoder state ($h_i$)” is Key

Step 3: Softmax → Attention Coefficients

\[\alpha_{t,i} = \frac{\exp(e_{t,i})}{\sum_{j=1}^{T} \exp(e_{t,j})}\]

where: $ \sum_{i=1}^{T} \alpha_{t,i} = 1 $

Step 4: Compute Attention Vector

Weighted sum:

\[a_t = \sum_{i=1}^{T} \alpha_{t,i} h_i\]

This is convex combination of encoder states.

“Each encoder state ($h_i$)” is Value

“Each encoder state ($a_t$)” is Attention Value

Step 5: Combine with Decoder State

Concatenate:

\[\tilde{s}_t = \tanh(W_c [a_t ; s_t])\]

where: $[a_t ; s_t]$ is concatenated vector

Output probability:

\[P(y_t | y_{<t}, x) = \text{softmax}(W_o \tilde{s}_t)\]

Attention Process

Relation Token from attention score

Component	Meaning
Query	Decoder state
Key	Encoder states
Value	Encoder states
Attention weight	Similarity measure
Output	Weighted average