CPU caches
The conceptWhy your processor waits 100 times longer for RAM than its cache
A CPU cache is a small, very fast memory on the processor chip that holds copies of recently used bytes. Reading the closest cache (L1) takes about 1 nanosecond; reading main memory (RAM) takes about 100.
The CPU checks the cache before RAM. A hit means the bytes are there and come back at once. A miss means the CPU stalls, fetches them from RAM and keeps a copy, so the next use is fast.
Caches are stacked in levels: L1 is tiny (tens of kilobytes) and fastest, L2 is larger and slower, and L3 is megabytes in size and shared by all cores. Each miss falls through to the next level, and RAM is the last stop.
What it’s forEstimates how long a memory read takes on average once the cache's hits and misses are counted.
average access time = hit time + miss rate × miss penalty
- hit time
- time to read from the cache, e.g. 1 ns for L1
- miss rate
- fraction of reads not found in the cache, often a few percent
- miss penalty
- extra time to fetch from the slower level, e.g. 100 ns for RAM
What it givesThe typical cost of one memory read, which shows how much a small miss rate matters.
- Hit time 1 ns, miss rate 5%, miss penalty 100 ns
- 1 + 0.05 × 100 = 1 + 5 = 6 ns
- With a 1% miss rate: 1 + 0.01 × 100 = 2 ns