Could High Bandwidth Flash be Local Inference's saviour?

I worked with a device using Flash as its "memory" in the past. First, let's look at the cost/complexity of Flash vs DRAM: Fundamentally, DRAM is a lot more complex to deal with than Flash simply due to the physics: You may know that DRAM stores its information inside capacitors with a "high" charge being a 1 and no charge a zero. The issue you run into when scaling DRAM density is that as you make capacitors smaller, their capacitance drops, meaning you need to compensate via deep trench capacitors (which are exactly what they sound like) and stacking them in complex 3d shapes to maximize density while keeping the capacitors as large as possible. This means you consume a massive amount of vertical space and add a ton of process complexity: A single DRAM cell is already very "high" compared to a flash cell. Flash instead uses no separate capacitors and therefore does not have the same scaling/refresh constraints as DRAM (you can further improve this by having transistors share diffusion regions, which is what is done in traditional NAND flash already). Modern flash (e.g. V-NAND) wills stack hundreds of layers vertically, which improves the density even further. You can't do this for DRAM since the capacitors are already positioned vertically. This also goes further into the lithography: Flash can deal with large voltage variations and comparatively high error rates (the density is high enough to just have error correcting bits to account for process uncertainty). DRAM does not have that luxury: You have extremely tight sense margins and need to guarantee very low leakage. This is before we even account for the fact that you can store multiple bits inside a single flash cell by using multiple threshold levels. This is also why there are so few DRAM manufacturers, despite the actual architecture not being that complex: The issue you have in DRAM is that you can't just scale your process with the node size since you have a lot of "non transistor" components, which don't scale the way you would expect them to from a physics level. The issue with Flash from a performance POV is that the reading/writing process is slower on a physical level: NAND reads bits by first applying a voltage to the word line, then measuring the current through a series string, and finally determining whether/how much the transistor conducts by comparing to threshhold(s). Writing is also physically slow since flash uses electron tunneling for writing, which needs rather high voltages (cells locally reach anywhere from 10-20V) and verification cycles afterwards (this electron tunneling is also what slowly degrades the cells). Essentially you "kick" electrons over an oxide barrier, from which they can't escape. However, to do that you neend high voltage. A lot of the complexity in Flash though arises due to its need to be non-volatile: You could use floating gate storage with very thin oxides, which has poorer retention but should program faster (you essentially build a weird version of SRAM with one transistor per cell). However, this doesn't matter: You can use current flash technology already as a backbone for computation. Fundamentally, Flash is not that slow (especially if you heavily multiplex it), it just has high latency. For DL where the access patterns are somewhat predictable, using a high-bandwidth, high-latency mass storage might already be worth it (i.e. you have SRAM->DRAM->flash). You technically only need one "hot" layer and a "loading" layer assuming you have enough throughput since you can pipeline from one layer to the next. The complexity is just in how to properly deal with such an additional source of memory. The actual principle is pretty well understood: You use the same trick as with HBM by connecting multiple (flash) cells via microbumps. The throughput is similar to DRAM, just the latency is horrible. In short: Flash will always be orders of magnitude cheaper per gigabyte and you don't need any special engineering for Flash to become a high bandwidth, high capacity source. The difficulty is dealing with latency, but you can mask this easily if you have static computation graphs and are careful with prefetch (the golden ticket with "low latency, high bandwidth, high capacity" would be FeFETs/RRAM or MRAM but those are not mass market ready).