Tiny Transformers Explained

Abstract

We study tiny transformers and show attention A suffices for toy tasks.

1 Introduction

Transformers are neural networks. We define attention as s throughout this paper.

Figure 1: The tiny attention block.
Figure 2: An SVG figure, skipped in v1.

2 Method

Our method stacks two attention layers with residual connections.

References

[1] Someone et al. 2017.