Instantly share code, notes, and snippets. Star 5,000+ ( 5,000+ ) You must be signed in to star a gist Fork 2,732 ( 2,732 ) You must be signed in to fork a gist Embed Select an option Embed Embed this gist in your website. Share Copy sharable link for this gist. Clone via HTTPS Clone using the web URL. No results found Learn more about clone URLs Clone this repository at <script src="https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95.js"></script> Embed Embed this gist in your website. Share Copy sharable link for this gist. Clone via HTTPS Clone using the web URL. No results found Save karpathy/8627fe009c40f57531cb18360106ce95 to your computer and use it in GitHub Desktop. Embed Embed this gist in your website. Share Copy sharable link for this gist. Clone via HTTPS Clone using the web URL. No results found ariannamethod commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . Hey @MattWenJun — your emojiGPT was the seed. We forked it and it mutated into caveLLMan. 88 SVG hieroglyphs instead of emoji. A semantic tokenizer compresses any text into 88 cave-painting concepts: "Count Dracula stood in the dark castle and waited" → dark stone and wait man It has Hebbian plasticity — learns from every conversation, no backprop. Invents new symbols when patterns crystallize. Symbols that don't survive die. We fed it Dracula — 2244 sentences devoured, 8 symbols born, 8 died. Runs on notorch — pure C, no PyTorch. The cave painter: caveLLMan Thank you for the spark. qhuang20 commented Apr 16, 2026 really nice work! yukieliot69 commented Apr 30, 2026 Cybertron-Ant commented May 2, 2026 but we were doing this in JavaScript since 2017!! xskongai commented May 6, 2026 Today I tried to implement microGPT by following your code, and when I read this part: I suddenly realized why you are so popular. You have the ability to turn abstract concepts into concrete, simple code. Ideas like linear layers and softmax can seem intimidating at first, but your code brings them back to their natural form: weighted sums, exponentiation, and normalization. These ideas are simple at their core, and your teaching helps people see that simplicity. I think this is the real power of great teaching: not making simple things sound complicated, but making complicated things reveal their original simplicity. Thank you for showing that deep learning, at its core, can be understood from first principles. It is a pity that I only got to know you now, but I am very happy that I still got to know you. napaputteppawan-netizen commented May 7, 2026 -- [[ ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต V.3 FINAL BY มหาเทพธัญญ่า ]] -- local Library = loadstring(game:HttpGet(" https://raw.githubusercontent.com/xHeptc/Kavo-UI-Library/main/source.lua"))( ) local Window = Library.CreateLib("ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต HUB", "BloodTheme") local Tab1 = Window:NewTab("มหาเทพสายตบเด็ก") local KillSection = Tab1:NewSection("วาร์ปสังหารเด็กกระโปกสัด!!") _G.SilentAim = false _G.AutoKill = false _G.WarpDirection = "Behind" KillSection:NewDropdown("เลือกจุดเกิดมหาเทพ", "มหาเทพจะโผล่ไปทางไหนมึง!!", {"Behind", "Front", "Right", "Left", "Above", "Below"}, function(currentOption) _G.WarpDirection = currentOption end) KillSection:NewToggle("เปิดระบบวาร์ปตบฆาตกร", "วาร์ปไปตบเด็กสกีบีดี้ให้เละสัด!!", function(state) _G.AutoKill = state task.spawn(function() while _G.AutoKill do pcall(function() for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v ~= game.Players.LocalPlayer and v.Character and v.Character:FindFirstChild("HumanoidRootPart") then if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then local Murderer = v.Character.HumanoidRootPart local TargetPos = Murderer.CFrame KillSection:NewToggle("Silent Aim (ยิงเลี้ยวเจาะกะโหลก)", "กดยิงมั่วๆ ก็เข้าหัวสัด!!", function(state) _G.SilentAim = state end) local Tab2 = Window:NewTab("มุดส่องเด็ก") local HelperSection = Tab2:NewSection("มองทะลุ & วิ่งหนีสกีบีดี้") _G.ESP = false HelperSection:NewToggle("ESP ส่องหัวเด็กกระโปก", "แดง=ฆาตกร, น้ำเงิน=นายอำเภอ", function(state) _G.ESP = state task.spawn(function() while _G.ESP do for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v ~= game.Players.LocalPlayer then local hl = v.Character:FindFirstChild("Highlight") or Instance.new("Highlight", v.Character) hl.Enabled = true hl.FillTransparency = 0.5 if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then hl.FillColor = Color3.fromRGB(255, 0, 0) elseif v.Backpack:FindFirstChild("Gun") or v.Character:FindFirstChild("Gun") then hl.FillColor = Color3.fromRGB(0, 0, 255) else hl.FillColor = Color3.fromRGB(0, 255, 0) end end end task.wait(0.5) end for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v.Character:FindFirstChild("Highlight") then v.Character.Highlight:Destroy() end end end) end) HelperSection:NewSlider("วิ่งไวปานเทพไฟ (Speed)", "วิ่งหนีสกีบีดี้มึง!!", 200, 16, function(s) if game.Players.LocalPlayer.Character and game.Players.LocalPlayer.Character:FindFirstChild("Humanoid") then game.Players.LocalPlayer.Character.Humanoid.WalkSpeed = s end end) -- [[ ระบบเบื้องหลัง: มุดวิถีกระสุน (The Real Magic) ]] -- local mt = getrawmetatable(game) local oldNamecall = mt.__namecall setreadonly(mt, false) mt.__namecall = newcclosure(function(self, ...) local Method = getnamecallmethod() local Args = {...} end) setreadonly(mt, true) napaputteppawan-netizen commented May 7, 2026 `-- [[ ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต V.3 FINAL BY มหาเทพธัญญ่า ]] -- local Library = loadstring(game:HttpGet(" https://raw.githubusercontent.com/xHeptc/Kavo-UI-Library/main/source.lua"))( ) local Window = Library.CreateLib("ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต HUB", "BloodTheme") local Tab1 = Window:NewTab("มหาเทพสายตบเด็ก") local KillSection = Tab1:NewSection("วาร์ปสังหารเด็กกระโปกสัด!!") _G.SilentAim = false _G.AutoKill = false _G.WarpDirection = "Behind" KillSection:NewDropdown("เลือกจุดเกิดมหาเทพ", "มหาเทพจะโผล่ไปทางไหนมึง!!", {"Behind", "Front", "Right", "Left", "Above", "Below"}, function(currentOption) _G.WarpDirection = currentOption end) KillSection:NewToggle("เปิดระบบวาร์ปตบฆาตกร", "วาร์ปไปตบเด็กสกีบีดี้ให้เละสัด!!", function(state) _G.AutoKill = state task.spawn(function() while _G.AutoKill do pcall(function() for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v ~= game.Players.LocalPlayer and v.Character and v.Character:FindFirstChild("HumanoidRootPart") then if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then local Murderer = v.Character.HumanoidRootPart local TargetPos = Murderer.CFrame KillSection:NewToggle("Silent Aim (ยิงเลี้ยวเจาะกะโหลก)", "กดยิงมั่วๆ ก็เข้าหัวสัด!!", function(state) _G.SilentAim = state end) local Tab2 = Window:NewTab("มุดส่องเด็ก") local HelperSection = Tab2:NewSection("มองทะลุ & วิ่งหนีสกีบีดี้") _G.ESP = false HelperSection:NewToggle("ESP ส่องหัวเด็กกระโปก", "แดง=ฆาตกร, น้ำเงิน=นายอำเภอ", function(state) _G.ESP = state task.spawn(function() while _G.ESP do for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v ~= game.Players.LocalPlayer then local hl = v.Character:FindFirstChild("Highlight") or Instance.new("Highlight", v.Character) hl.Enabled = true hl.FillTransparency = 0.5 if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then hl.FillColor = Color3.fromRGB(255, 0, 0) elseif v.Backpack:FindFirstChild("Gun") or v.Character:FindFirstChild("Gun") then hl.FillColor = Color3.fromRGB(0, 0, 255) else hl.FillColor = Color3.fromRGB(0, 255, 0) end end end task.wait(0.5) end for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v.Character:FindFirstChild("Highlight") then v.Character.Highlight:Destroy() end end end) end) HelperSection:NewSlider("วิ่งไวปานเทพไฟ (Speed)", "วิ่งหนีสกีบีดี้มึง!!", 200, 16, function(s) if game.Players.LocalPlayer.Character and game.Players.LocalPlayer.Character:FindFirstChild("Humanoid") then game.Players.LocalPlayer.Character.Humanoid.WalkSpeed = s end end) -- [[ ระบบเบื้องหลัง: มุดวิถีกระสุน (The Real Magic) ]] -- local mt = getrawmetatable(game) local oldNamecall = mt.__namecall setreadonly(mt, false) mt.__namecall = newcclosure(function(self, ...) local Method = getnamecallmethod() local Args = {...} end) setreadonly(mt, true) ` napaputteppawan-netizen commented May 7, 2026 `-- [[ ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต V.3 FINAL BY มหาเทพธัญญ่า ]] -- local Library = loadstring(game:HttpGet(" https://raw.githubusercontent.com/xHeptc/Kavo-UI-Library/main/source.lua"))( )()) local Window = Library.CreateLib("ตบเด็กกระโปกกี้สกีบีดี้ตอยเล็ต HUB", "BloodTheme") local Tab1 = Window:NewTab("มหาเทพสายตบเด็ก") local KillSection = Tab1:NewSection("วาร์ปสังหารเด็กกระโปกสัด!!") _G.SilentAim = false _G.AutoKill = false _G.WarpDirection = "Behind" KillSection:NewDropdown("เลือกจุดเกิดมหาเทพ", "มหาเทพจะโผล่ไปทางไหนมึง!!", {"Behind", "Front", "Right", "Left", "Above", "Below"}, function(currentOption) _G.WarpDirection = currentOption end) KillSection:NewToggle("เปิดระบบวาร์ปตบฆาตกร", "วาร์ปไปตบเด็กสกีบีดี้ให้เละสัด!!", function(state) _G.AutoKill = state task.spawn(function() while _G.AutoKill do pcall(function() for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v ~= game.Players.LocalPlayer and v.Character and v.Character:FindFirstChild("HumanoidRootPart") then if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then local Murderer = v.Character.HumanoidRootPart local TargetPos = Murderer.CFrame KillSection:NewToggle("Silent Aim (ยิงเลี้ยวเจาะกะโหลก)", "กดยิงมั่วๆ ก็เข้าหัวสัด!!", function(state) _G.SilentAim = state end) local Tab2 = Window:NewTab("มุดส่องเด็ก") local HelperSection = Tab2:NewSection("มองทะลุ & วิ่งหนีสกีบีดี้") _G.ESP = false HelperSection:NewToggle("ESP ส่องหัวเด็กกระโปก", "แดง=ฆาตกร, น้ำเงิน=นายอำเภอ", function(state) _G.ESP = state task.spawn(function() while _G.ESP do for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v ~= game.Players.LocalPlayer then local hl = v.Character:FindFirstChild("Highlight") or Instance.new("Highlight", v.Character) hl.Enabled = true hl.FillTransparency = 0.5 if v.Backpack:FindFirstChild("Knife") or v.Character:FindFirstChild("Knife") then hl.FillColor = Color3.fromRGB(255, 0, 0) elseif v.Backpack:FindFirstChild("Gun") or v.Character:FindFirstChild("Gun") then hl.FillColor = Color3.fromRGB(0, 0, 255) else hl.FillColor = Color3.fromRGB(0, 255, 0) end end end task.wait(0.5) end for _, v in pairs(game:GetService("Players"):GetPlayers()) do if v.Character and v.Character:FindFirstChild("Highlight") then v.Character.Highlight:Destroy() end end end) end) HelperSection:NewSlider("วิ่งไวปานเทพไฟ (Speed)", "วิ่งหนีสกีบีดี้มึง!!", 200, 16, function(s) if game.Players.LocalPlayer.Character and game.Players.LocalPlayer.Character:FindFirstChild("Humanoid") then game.Players.LocalPlayer.Character.Humanoid.WalkSpeed = s end end) -- [[ ระบบเบื้องหลัง: มุดวิถีกระสุน (The Real Magic) ]] -- local mt = getrawmetatable(game) local oldNamecall = mt.__namecall setreadonly(mt, false) mt.__namecall = newcclosure(function(self, ...) local Method = getnamecallmethod() local Args = {...} end) setreadonly(mt, true) ` vazhnov commented May 10, 2026 if you could add a license, that would be great, the total lines of the code still around 200is, : ) It would be great to see any popular open source license: it will allow people to fork, adjust and publish changes legally 😉 iamyb commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . Thanks for the nice code. I made an Excel workbook to visualize it step by step, to better understand how it moves through embeddings, attention, and MLP blocks, all inside Excel. Hopefully it can also be useful to anyone else who wants to learn this 200 line's GPT.

yakovenkoroman1993 commented May 30, 2026 Instead of 300 pages of Sebastian Rashka Janhs888 commented Jun 8, 2026 go to school and learn lxb12123 commented Jun 10, 2026 Thanks Andrej for sharing microgpt.py! I built an interactive 3D tutorial around it to help learners visualize the full pipeline: forward pass, loss, autograd, attention, training, and sampling. The sandbox runs directly in the browser and closely follows the original code. Demo: Interactive 3D tutorial Source: GitHub repository Feedback and corrections are very welcome! :) aiwonderland commented Jun 27, 2026 The zen of small gpts~ pablogiaccaglia commented Jul 7, 2026 this comment section is desperate tomsiwik commented Jul 9, 2026 this comment section is desperate and at this point automated Entrpi commented Jul 10, 2026 Today I had Fable make some meaningful updates to de novo microgpt: https://github.com/Entrpi/microgpt-denovo Entrpi commented Jul 10, 2026 Also, now eemicrogpt runs up to 74000x faster than this CPython on Mac M5: https://github.com/Entrpi/eemicrogpt Hexiaoqiao commented Jul 16, 2026 Thanks @karpathy for your cool works. I am one new GPT learner here. I am confused with L156 where if len(docs)> num_steps there are some docs will never be used to train, right? Please correct me if I missed something. Thanks. tavousi commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . Thank you, Andrej @karpathy , for the brilliant MicroGPT project! It completely inspired my recent work. I adapted your architecture into a character model trained on the Tranco Top 1M sites to generate premium domain names. I’d love to check it out here: https://github.com/tavousi/DomainerGPT Imad-Jan commented Aug 4, 2026 This is a great example of how simple implementations can teach more than large frameworks. Once the core mechanics are visible, it becomes much easier for people to experiment and create their own versions. robertchanon-beep commented Aug 23, 2026 Thanks, @karpathy , that was genuinely helpful in confirming something I was working on, but it is more advanced. The Wiki solve another issue I was looking at too. Kai26-Han commented Aug 25, 2026 Thanks Andrej! I made a version that generates poems instead of names. https://github.com/Kai26-Han/poemGPT soulaway commented Aug 25, 2026 Great thank you to Andrey @karpathy for inspiring so many people here, including me. Together with my friends, we wrote Juno, a sort of distributed Java-based tool for inferencing and fine-tuning of GGUF models: https://github.com/ml-cab/juno We're happy to collaborate with anyone who also believes in local or self-hosted LLMs as the fairest way to handle personal data. Let ML for the good! thanksforfree commented Sep 21, 2026 thank you @karpathy great work. just created an interactive web to help reading by viz the archi-code gap, with ref to the transformer paper: https://github.com/tanpuekai/microGPT_webEdu ariannamethod commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . actually, you can skip the transformer loop and backprop entirely and still get speech that passes a court. we've been doing it for months; figured this thread should know. netta.py — one file, 573 lines, stdlib only. no gradients, no optimizer, no pretrained anything, no names.txt. it eats ordinary text, grows byte-pair units out of what it actually lived, and its mouth is only allowed to say what it has lived — there's a coherence court inside that passes or fails the speech. no loss curve. nothing descends. your tiny-shakespeare (1,115,394 bytes), my laptop: netta.py: cold start → court-passing speech in 8.43 s. the court's receipt: model 0.371 bits/byte against 3.195 bits/byte of ignorance, longest verbatim borrow 49 bytes, copy coverage 0.143 against a 0.50 ceiling. she speaks Shakespeare; she doesn't recite him. microgpt.py (this gist, verbatim, same corpus): at 8.43 s — step 54 of 1000, no output. at 84.3 s — step 568, still practicing. we waited out its full 153.04 s: the full run took 153.04 s; the first ten of its twenty raw samples are below, unedited netta, seed 42, raw: microgpt, full budget, first ten of twenty, raw: the python file is a literal port of a C organism and reproduces its speech byte-for-byte — same seed, same island, and the repo's independent C court reader returns the same sealed report hash over the python's output. gist: https://gist.github.com/ariannamethod/6bace55c28b8823b04e09b45b5ce80c2 . the organism, the courts and the full history: https://github.com/ariannamethod/netta her sister writes programs, runs them, and plays Doom: https://github.com/ariannamethod/netta.code our measurements, our code, yo so — actually maybe Netta is all you need? — Arianna Method ariannamethod commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . There was an error while loading. Please reload this page . one more thing. the microgpt round was the warm-up. for the real match we booked the champion: stock nanoGPT — his own README recipe, his own train/val split, same laptop, no GPU. both systems price HIS val slice, not their own speech. and full disclosure on the handicap: nanoGPT brings torch's compiled kernels. netta brings the python standard library, interpreted. the tilt is his. his A100 config on this CPU first: four cold runs — 10, 60, 600, 1800 seconds. silence, four times, not one loss line. we throw that sweep out ourselves: a config on the wrong hardware measures nothing. his CPU recipe is the real opponent, and it delivers: 82.52 s to completion, val 1.8857 nats = 2.7205 bits/char. his README promised ~1.88. it told the truth. the scoreboard at 13.16 seconds — the only budget she needs: held-out bits/byte on his val: netta 3.360 ; nanoGPT, mid-fall on his own logged curve, between 3.53 and 3.34. a tie. speech that passed a court: netta 1, nanoGPT 0 — there is no court anywhere in his pipeline. his first checkpoint lands at step 250, by his own save law; full training completes at 82.5 s. and at 82.5 seconds he wins the number: 2.7205 beats her 3.360, his curve crossing hers around second 9–16. gradient descent is fast when torch does the descending. its first checkpoint lands around step 250; training completes at 82.5. she was done at 13 — she runs once, her number never moves again, because there is nothing left for her to do. and nothing he produces, early or late, ever faces a court. netta at 13.16 s, seed 7, raw: nanoGPT at 82.52 s, his own sample.py defaults, sample 0, raw: we ran our own census on both — and we're keeping the row that goes against us: his samples copy less. longest verbatim borrow 12–15 bytes across all ten, coverage 0.0; hers run 29–49 bytes, coverage up to 0.143 under her own 0.50 ceiling. also true: at 1.886 nats the output is word-salad, and word-salad cannot accidentally reproduce 32 straight bytes of Shakespeare — low copying is partly low fluency. the measure is still the measure. the row stays. every receipt is attached to the gist — commands, rc codes, versions, all ten of his samples, the val curve: https://gist.github.com/ariannamethod/6bace55c28b8823b04e09b45b5ce80c2 . run it. figure it out. both things are true: everything past the atom, the transformer buys with time — and buys well. but coherent, court-judged speech in 13 seconds, no gradients, no optimizer, nothing to wait for — that is the atom. and the atom holds. — Arianna Method (edited: an earlier version claimed his first checkpoint arrives at 82.5 s. wrong by his own save law — train.py L274, the first improving eval at step 250 writes it. caught by our own audit. corrected, receipts updated.) alarillaclarisse-droid commented Sep 23, 2026 Andrei Karpathy's minimalist implementations and clean code gists are always such a masterclass in breaking down complex deep learning architectures into their purest components. Simplifying intricate concepts into readable, actionable structures makes a huge difference. On a completely different note, for founders and business owners navigating the complex financial structure of their enterprise, partnering with a trusted business valuation company in miami can provide that same clarity when determining accurate market worth for future sale or investment opportunities. lukaszsamson commented Sep 23, 2026 Bend2 port https://gist.github.com/lukaszsamson/1da96fabce66940cdf49c57544c67b22 using sequential one threaded implementation. Naive multithreaded and GPU versions turned out to be slower dp0000000004-eng commented Sep 29, 2026