https://yapingyang.github.io/posts/Recent content in Posts on yapingyangHugoen-usSat, 01 Aug 2026 16:54:12 -0400https://yapingyang.github.io/posts/attention-kv-cache-and-prompt-caching-explained-from-first-principles/Sat, 01 Aug 2026 16:54:12 -0400https://yapingyang.github.io/posts/attention-kv-cache-and-prompt-caching-explained-from-first-principles/<h2 id="what-is-an-attention-mechanism-and-what-does-it-solve">What Is an Attention Mechanism, and What Does It Solve?</h2> <p>A large language model cannot store the word &ldquo;cat&rdquo;, only numbers. So every word has to become a list of numbers, which we&rsquo;ll call a vector, looked up from a big table. But here&rsquo;s the catch, that lookup is context-free, meaning &ldquo;it&rdquo; in the sentence &ldquo;the cat eats because it was hungry&rdquo; has the same vector representation &ldquo;it&rdquo; gets in every other sentence. On its own, it knows nothing about the cat.</p>