https://learning-notes-8ef.pages.dev/tags/nlp/Recent content in Nlp on Learning JournalHugoenThu, 18 Jun 2026 00:00:00 +0000https://learning-notes-8ef.pages.dev/posts/machine-learning/understanding-attention/Thu, 18 Jun 2026 00:00:00 +0000https://learning-notes-8ef.pages.dev/posts/machine-learning/understanding-attention/<h2 id="the-core-idea">The Core Idea</h2> <p>Attention lets each token in a sequence &ldquo;look at&rdquo; every other token and decide how much to weight its contribution.</p> <h2 id="query-key-value">Query, Key, Value</h2> <p>Three projections of the input:</p> <ul> <li><strong>Query (Q):</strong> what this token is looking for</li> <li><strong>Key (K):</strong> what each token offers</li> <li><strong>Value (V):</strong> what each token contributes if selected</li> </ul> <p>$$\text{Attention}(Q, K, V) = \text{softmax}!\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$</p> <h2 id="why-it-works">Why It Works</h2> <p>The dot product of Q and K measures compatibility. Dividing by $\sqrt{d_k}$ prevents softmax from saturating in high dimensions.</p>