kv-cache : support attention rotation for heterogeneous iSWA by ggerganov · Pull Request #21513 · ggml-org/llama.cpp

Posted by jacek2023@reddit | LocalLLaMA | View on Reddit | 16 comments

tl;dr: Fixes KV-cache rotation for hybrid-attention models like Gemma 4

[-]

BigYoSpeck@reddit

I've tested it with both the UD Q6_K_XL and bartowski Q8_0 of Gemma 4 31B

For general logic, reasoning, instruction following and creativity it seems broadly a match for none quantised KV. But for coding it's been just slightly off in the details that completely blow it

One of the tests I do is getting the model to make a Micro Machines game

Gemma 4 does a really good job of this. AI cars that drive the track, collisions, sliding physics, track limits, lap counts and race position all handled producing a perfectly playable game

With -ctk and -ctv q8_0 it gets the details just wrong enough that it all falls apart. AI driving in circles, acceleration physics off so the car zooms off screen instantly, track graphics not aligned

I've no doubt a clearer prompt could work around it, but the point of the test is as basic a prompt as the base config can handle not behaving quite as well with this

[-]

jacek2023@reddit (OP)

could you share the game? :)

[-]

BigYoSpeck@reddit

Top-down racing game (like Micro Machines).

Single index.html file.
Web Canvas API.
Physics (acceleration, friction, dynamic steering, collisions with walls and opponent cars, and "drifting" to capture the feel of the classic Micro Machines games). Micro Machines cars don't just turn. They have some slide. Implement "forward velocity" and "side velocity" to replicate the sliding feel.
AI drivers using a Waypoint System, meaning the computer drivers navigate by targeting a series of invisible nodes placed around the track.
Track defined as a path or a set of boundary points. A simple road visual with chicane obstacles placed on the track. Have red curbing like real race course. The start/finish line should be at the end of a straight, not on a corner and should be a black and white chequerboard
Lap Logic to prevent "cheating" (just driving in circles at the finish line), player must pass the midpoint of the track before the finish line will count as a completed lap.
Lap count. Set a lap limit of 5 laps, cars should stop when reaching this limit and a winner be declared.
Terrain detection. The cars should slow down slightly when driving off track as though on grass. Apply a multiplier of 0.95 when the car leaves the race track onto grass.
The camera should follow the player smoothly with viewport transformation to render the world relative to the camera position.
The track should be larger that the screen view port so that not all is visible without driving around the circuit.
The track should be wide enough to allow the cars to take racing lines around the corners.
Win/lose conditions with final results.