Alex’s novel approach was to parallelize the computation of his neural networks, allowing them to be wider and deeper than ever before. 2 But how did he train his network? That’s all down to ...

In the formal paper, "data2vec: A General Framework for Self-supervised Learning ... in each block as target," where a "block" is the Transformer equivalent of a neural network layer.