AIModels.fyi

AIModels.fyi

Share this post

AIModels.fyi
AIModels.fyi
Bye tokens, hi patches

Bye tokens, hi patches

Meta announces a better way to scale LLMs

aimodels-fyi's avatar
aimodels-fyi
Dec 14, 2024
∙ Paid
3

Share this post

AIModels.fyi
AIModels.fyi
Bye tokens, hi patches
1
Share

Do we really need to break text into tokens, or could we work directly with raw bytes?

First, let’s think about how do LLMs currently handle text. They first chop it up into chunks called tokens using rules about common word pieces. This tokenization step has always been a bit of an odd one out. While the rest of the model learns and adapts during training, tokenization stays fixed, based on those initial rules. This can cause problems, especially for languages that aren’t well-represented in the training data or when handling unusual text formats.

AIModels.fyi is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Keep reading with a 7-day free trial

Subscribe to AIModels.fyi to keep reading this post and get 7 days of free access to the full post archives.

Already a paid subscriber? Sign in
© 2025 AIModels.fyi
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share