Deep Learning::Attention is all you need paper (Transformer Model)

The Transformer model relies solely on self-attention mechanisms.

The Transformer model handles input and output sequences through a specific architecture that includes encoder and decoder components, each employing self-attention mechanisms. Here’s how each part operates:

1. **Encoder**:
- The encoder processes the input sequence of symbol representations, mapping it to a sequence of continuous representations. It is composed of a stack of identical layers (6 in the base model), each containing:
- **Multi-head Self-Attention Layer**: This allows each position in the input sequence to attend to all other positions, capturing dependencies irrespective of their distance.
- **Feed-Forward Neural Network**: A position-wise fully connected feed-forward network is applied to the output of the attention mechanism.
- Each layer employs residual connections and normalization to stabilize and enhance learning.

2. **Decoder**:
- The decoder generates the output sequence symbol by symbol. Like the encoder, it also consists of a stack of identical layers. However, it includes an additional multi-head attention layer that attends over the encoder’s output. Its layers consist of:
- **Masked Multi-head Self-Attention Layer**: This ensures that predictions for a certain position can only depend on known outputs (those generated before it), maintaining the auto-regressive property.
- **Encoder-Decoder Attention Layer**: Queries come from the previous decoder layer, while keys and values come from the encoder’s output. This allows the decoder to utilize all positions in the input sequence.
- **Feed-Forward Neural Network**: Similar to the encoder, each layer in the decoder also has a feed-forward component.

3. **Positional Encoding**:
- Since the Transformer does not have recurrence or convolutions, positional encodings are added to the input embeddings to provide information about the order of tokens in the sequences. These encodings allow the model to understand the relative positions within the input and output.

These components work together to efficiently process sequences in parallel, as opposed to the sequential processing seen in traditional recurrent networks, enhancing the model's ability to learn contextual relationships.

The Transformer uses positional encodings added to the input embeddings to provide information about the order of the sequence.

The models were trained on machines with 8 NVIDIA P100 GPUs.

# Regularization Techniques in the Transformer Model

During the training of the Transformer model, the following regularization techniques were employed:

1. **Residual Dropout**:
- Dropout was applied to the output of each sub-layer before it was added to the sub-layer input and normalized. This helps to prevent overfitting by randomly setting a fraction of the input units to zero during training.
- Additionally, dropout was applied to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks.

**Dropout Rate**: For the base model, a dropout rate of Pdrop=0.1 was used.

2. **Label Smoothing**:
- Label smoothing is a technique where the model's confidence is reduced by assigning a small probability to incorrect classes. This discourages the model from becoming overly confident about its predictions.
- The label smoothing value used was ϵls=0.1.

These regularization techniques helped improve the model's accuracy and BLEU scores while avoiding overfitting during the training process.

The Transformer implements masking in the decoder to prevent positions from attending to subsequent positions.

Label smoothing helps to improve accuracy and BLEU score by making the model less confident in its predictions.

The training data for the machine translation tasks was preprocessed in the following ways:

Sentence Pair Selection: The datasets used included the standard WMT 2014 English-German dataset with about 4.5 million sentence pairs and a significantly larger WMT 2014 English-French dataset consisting of 36 million sentence pairs.

Token Encoding: Sentences were encoded using byte-pair encoding, which helps manage the vocabulary by breaking words into subword units. This method results in a shared source-target vocabulary of about 37,000 tokens for the English-German dataset and a vocabulary of 32,000 word-pieces for the English-French dataset.

Batching by Sequence Length: To optimize training, sentence pairs were batched together based on approximate sequence length. Each training batch contained a set of sentence pairs containing approximately 25,000 source tokens and 25,000 target tokens.

This preprocessing enables the Transformer to train efficiently on the data, leveraging a manageable vocabulary while maintaining the context needed for effective translation.