The Tensor array in TSU doesn't differ from matrices, which comes with the benefit of using each memory address for both layer loading and KV caching. GPU waste a lot of resources because they can only do sequential processing as they have no buffers other than going down the tree in memory tree. At least that is my understanding.

A very simple ascii art representation of both shows the difference

GPU Processing including KV cache:

Compute Units (Active)   Wasted Idle Resources
+------------------+     +---------------------+
|  ████  ████  ██  |     | ░░░░░░░░░░░░░░░░░░░ |
|  ███   ████  ██  |     | ░░░░░░░░░░░░░░░░░░░ |
|  ██    ████  ██  |     | ░░░░░░░░░░░░░░░░░░░ |
|  ██    ████      |     | ░░░░░░░░░░░░░░      |
+------------------+     +---------------------+

█ = Compute Active (warps/threads executing matrix ops)
░ = Waste (memory reserved but unused)

TSU including KV cache:

TSU Compute Units (Active)       Wasted Idle Resources
+------------------------------+------------------------+
|  ██████████████████████████  | ░░░░░░░                |
|  ██████████████████████████  | ░░░░░░░                |
|  ██████████████████████████  |                        |
|  ██████████████████████████  |                        |
+--------------------------------+----------------------+

█ = Compute Active (highly utilized tensor streaming & fused KV cache)
░ = Waste (minimal idle resources)

I hope this highlights how, despite powerful parallel compute units, GPUs often suffer from memory over-allocation, fragmentation, and bandwidth bottlenecks that leave many resources underutilized in LLM inference workloads.

The streamlined, tiled, and instruction-controlled design of TSUs achieves much higher hardware utilization with less waste compared to the layered and often over-provisioned GPU KV cache systems.

Waste in idle resources but still using a lot of power versus highly condensed access writes without loss of resources will deliver substantial energy savings of watts vs token processing (joules).