LLMs-from-scratch/ch03/02_bonus_efficient-multihead-attention

More Efficient Multi-Head Attention Implementations

• mha-implementations.ipynb contains and compares different implementations of multi-head attention

Summary

The figures below summarize the performance benchmarks (lower is better).

 

Forward pass only

 

Forward and backward pass

 

Forward and backward pass after compilation