Attention, KV Cache, and Prompt Caching Explained From First Principles

What Is an Attention Mechanism, and What Does It Solve?

A large language model cannot store the word “cat”, only numbers. So every word has to become a list of numbers, which we’ll call a vector, looked up from a big table. But here’s the catch, that lookup is context-free, meaning “it” in the sentence “the cat eats because it was hungry” has the same vector representation “it” gets in every other sentence. On its own, it knows nothing about the cat.