Transformer Model Architecture Overview
The original Transformer model architecture is divided into an encoder and a decoder, each composed of a stack of 6 identical layers. Here are the key components:
- Multi-Head Self-Attention: Allows the encoder to consider other words in the input when encoding a particular word.
- Position-wise Feed-Forward Networks: A simple fully connected neural network applied to each position separately and identically.
- Residual Connection & Layer Normalization: Around each of the two sub-layers.
- Multi-Head Self-Attention: Similar to the encoder but also includes a masking mechanism to prevent future information from affecting the current and past states.
- Encoder-Decoder Attention: Allows every position in the decoder to attend over all positions in the encoder.
- Position-wise Feed-Forward Networks: Same as in the encoder.
- Residual Connection & Layer Normalization: Same as in the encoder.
Both the encoder and decoder use multi-head attention mechanisms, where the attention function is run in parallel multiple times and the results are concatenated.
- Output of each sub-layer and the embedding layers: dmodel=512
- Inner-layer of the feed-forward networks: dff=2048
- Positional Encoding: Added to the input embeddings to give the model information about the position of each word.
This architecture leverages solely attention mechanisms, dispensing with recurrence and convolutions entirely. It's designed for high parallelization and low training time while achieving state-of-the-art performance on sequence transduction tasks.
Would you like to know more about any specific part?
The most important citations in the "Model Architecture" section of the original Transformer paper are:
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation by Kyunghyun Cho et al. (2014)
- Focus: Introduces the encoder-decoder structure commonly used in sequence-to-sequence models.
Neural Machine Translation by Jointly Learning to Align and Translate by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio (2014)
- Focus: Introduces attention mechanisms in the context of neural machine translation.
Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, Oriol Vinyals, and Quoc V. Le (2014)
- Focus: Another foundational paper on sequence-to-sequence learning, emphasizing the use of LSTMs.
Generating Sequences with Recurrent Neural Networks by Alex Graves (2013)
- Focus: Discusses the auto-regressive nature of generating sequences, a property also present in the Transformer model.
Deep Residual Learning for Image Recognition by Kaiming He et al. (2016)
- Focus: Introduces residual connections, which are used in the Transformer to facilitate the training of deep networks.
Layer Normalization by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton (2016)
- Focus: Introduces layer normalization, used in the Transformer to stabilize the activations of the network.
These citations are foundational to the architecture and mechanisms employed in the Transformer model. Would you like to dive deeper into any of these papers?