System

So for people here, it’s neither a pure {-1, 0, +1} representation nor strictly {-1, +1} in the usual sense — it’s something slightly different. In a standard model, weights look like real-valued numbers, for example: 0.753453, -1.1757562, 0.005435344, ext.. These are typically stored in FP16, meaning each weight uses 16 bits. So for 128 weights: 128 × 16 bits = 2048 bitsIn Bonsai, weights are quantized differently. Each weight is approximated as:+scale or -scale /// Instead of storing full-precision values, we store:

-the sign of each weight (positive or negative)

-a shared scale factor for a group of weights

For a group of 128 weights: Each weight is represented by 1 bit (its sign: + or -) → 128 weights = 128 bits

Instead of 128 different values, we store one shared scale value in FP16 → 1 scale = 16 bits

So total storage becomes:

128 bits (signs)

+ 16 bits (scale)

= 144 bits

Per weight:

144 / 128 = 1.125 bits per weight

👉 That’s why Bonsai is effectively a 1.125-bit model, compared to 16 bits per weight in FP16:

144 bits vs 2048 bits

Finally, the scale is not arbitrary — it is chosen to minimize the approximation error between the original weights and their quantized version.

In practice (simplified case), the optimal scale is:

the average of the absolute values of the weights

So the idea is:

keep only the direction (sign) of each weight

approximate its magnitude using a shared scale

drastically reduce memory while preserving overall structureSo for people here, it’s neither a pure {-1, 0, +1} representation nor strictly {-1, +1} in the usual sense — it’s something slightly different. In a standard model, weights look like real-valued numbers, for example: 0.753453, -1.1757562, 0.005435344, ... These are typically stored in FP16, meaning each weight uses 16 bits. So for 128 weights: 128 × 16 bits = 2048 bits In Bonsai, weights are quantized differently. Each weight is approximated as: +scale or -scale Instead of storing full-precision values, we store: the sign of each weight (positive or negative) a shared scale factor for a group of weights For a group of 128 weights: Each weight is represented by 1 bit (its sign: + or -) → 128 weights = 128 bits Instead of 128 different values, we store one shared scale value in FP16 → 1 scale = 16 bits So total storage becomes: 128 bits (signs) + 16 bits (scale) = 144 bits Per weight: 144 / 128 = 1.125 bits per weight 👉 That’s why Bonsai is effectively a 1.125-bit model, compared to 16 bits per weight in FP16: 144 bits vs 2048 bits Finally, the scale is not arbitrary — it is chosen to minimize the approximation error between the original weights and their quantized version. In practice (simplified case), the optimal scale is: the average of the absolute values of the weights So the idea is: keep only the direction (sign) of each weight approximate its magnitude using a shared scale drastically reduce memory while preserving overall structure

so for people here, it's nether fully -1/0/1 or fully -1/1 , it's something different : for a normal model his weight is something like that : ex: 0.753453, -1.1757562, 0.005435344, etc. in FP16 so for 128 weight 128 × 16 bits = 2048 bits , for bonsai it's + scale or - scale for scale defining a common value , basically bonsai is 128 bits + 1 FP16 , every weight is nether +1 (+scale) or -1 (-scale) so 128 weight =128 bit then instead of having 128 different value we store 1 common value in fp16 , 1 scale = 16 bits , so for 128 weight 128 bits (- , + )

+ 16 bits (scale)

= 144 bits 144 / 128 = 1.125 bits per weight , so bonsai is a 1.125 bit models, we have 144 bits instead of 2048 bit , the scale optimal is choosen for minimising error so for the moment we do the Average of absolute values