Weight tying reuses the token embedding weights in the output layer, so the model shares one large vocabulary-sized matrix instead of learning two separate ones. section 4.6
In the GPT pipeline, it sits at the input/output ends of the model: token embeddings build the hidden representation, and the output head maps back to vocabulary logits.
Implementation details to notice:
- The shared matrix also lowers memory footprint and computational complexity. section 4.6
Parameter-count check
A model instance reports 163,009,536 parameters before tying, then 124,412,160 trainable parameters when the output head is excluded from the count.
The treatment explains the idea from the shape of the layers through the parameter-count effect, but stops short of implementing tied weights in the shown GPTModel and notes that separate layers are used here. section 4.6
