An unconventional approach to language understanding

Can gzip be a language model?

Based on research by Nathan

2024

02

The Unexpected Idea

Rethinking language models through compression

The Core Premise

01

What is gzip? A compression algorithm that finds patterns in data to reduce file size
02

Compression = Prediction Better compression requires better understanding of patterns
03

Language Modeling Predicting the next token based on context and patterns
04

The Insight Can compression ratios serve as similarity metrics for text?

How It Works

1

Input Text

Receive the target text for classification or prediction

→

2

Concatenate & Compress

Combine with reference texts and apply gzip compression

→

3

Measure Ratios

Compare compression ratios to find the most similar texts

→

4

Make Prediction

Select the category with the smallest compression difference

Performance Comparison

73%

gzip+kNN

78%

BERT

75%

GPT-2

Accuracy on text classification benchmarks

Gzip vs Neural Language Models

Gzip Approach

No training required
Highly interpretable
Extremely fast
Zero parameters
Limited to classification

Neural Models Current Standard
Requires massive training
Black box complexity
Computationally expensive
Billions of parameters
Full generation capability

Sometimes the simplest solutions reveal profound truths.
Compression might just be another form of understanding.

Reflection on the gzip LM discovery

Thank You

86 Hacker News points • Sparking conversations about language and compression

nathan.rs/posts/gzip-lm/