{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Transformer Architecture & Self-Attention\n",
        "Xenors practice notebook. Run each cell and compare the output with the tutorial page.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Calculate scaled dot-product attention\n",
        "Scaling prevents dot products from growing too large as key dimension increases.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import torch, math\n",
        "Q=torch.tensor([[1.,0.],[0.,1.]])\n",
        "K=Q.clone(); V=torch.tensor([[10.,0.],[0.,20.]])\n",
        "W=torch.softmax(Q@K.T/math.sqrt(2),dim=-1)\n",
        "print(W.round(decimals=3)); print((W@V).round(decimals=2))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Expected/checkpoint**\n",
        "```text\ntensor([[0.670, 0.330], [0.330, 0.670]])\n...\n```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Verify causal masking\n",
        "A causal mask prevents a decoder token from reading future tokens.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "scores=torch.zeros(4,4)\n",
        "mask=torch.triu(torch.ones(4,4),diagonal=1).bool()\n",
        "scores=scores.masked_fill(mask,float('-inf'))\n",
        "print(scores)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Expected/checkpoint**\n",
        "```text\nupper triangle becomes -inf\n```\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.x"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}