LLMs have no memory

Understand it one step at a time

The short runs these in order in about 28 seconds. Here they are written out — pick any step to jump the short straight to it.

1. 1

   Copilot does not remember you

   The model is stateless. Between two requests it holds nothing — not your repo, not your last question.

   model idle · zero state

2. 2

   The model is a frozen function

   Tokens in, one token out. Weights are read-only at inference, so nothing you type is ever written back.

   inference · weights read-only

3. 3

   So the editor rebuilds the world

   Every suggestion is a fresh prompt your IDE assembles: the file, the tabs, repo rules, retrieved code, the chat so far.

   assembling the prompt

4. 4

   It reads around your cursor

   Your file is split into a prefix and a suffix, and the model is asked to fill in only the middle.

   fill-in-the-middle

5. 5

   Your open tabs are the context

   Copilot scans nearby files you have open, scores the snippets that look like your code, and pastes the winners in.

   scanning neighbouring tabs

6. 6

   Chat searches your repo index

   Ask about #codebase and your workspace is chunked, embedded and searched — again on every turn.

   retrieving from the workspace index

7. 7

   Everything fights for one budget

   The context window is a fixed number of tokens. Rules, code, retrieved snippets, history and the answer all share it.

   packing the context window

8. 8

   Chat memory is the transcript, resent

   Every turn ships the whole conversation again. When it stops fitting, the oldest turns fall out — and it "forgets".

   replaying the conversation

9. 9

   So give it what it needs to know

   Instruction files, the right tabs open, explicit #file references and a fresh chat when it drifts. That is the real memory.

   you own the context