RAM Memory Bandwidth measurement numbers (for both Intel and AMD with instructions on how to measure your system)

Posted by jd_3d@reddit | LocalLLaMA | View on Reddit | 60 comments

Reply to Post

60 Comments

[-]

vlmtdev@reddit

8x64GB DDR4-2666 LRDIMM ECC (Samsung M386A8K40BM2-CTD7Y), 8 channel mode 1x EPYC 7402 Supermicro H11SSL-i Intel(R) Memory Latency Checker - v3.12 Measuring idle latencies for random access (in ns)... Numa node Numa node 0 0 121.8 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads :118238.7 3:1 Reads-Writes :118079.9 2:1 Reads-Writes :120224.6 1:1 Reads-Writes :122288.7 Stream-triad like:121910.3 All NT writes : 84892.8 1:1 Read-NT write:123180.1 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 0117691.1 Measuring Loaded Latencies for the system Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type InjectLatencyBandwidth Delay(ns)MB/sec ========================== 00000709.60118085.8 00002716.20117912.1 00008658.64117351.9 00015681.89117359.7 00050594.19117899.3 00100222.93120606.7 00200160.23 77033.9 00300189.81 51357.2 00400156.96 46146.0 00500142.44 37553.5 00700151.26 27441.9 01000136.16 19662.0 01300134.02 15317.4 01700133.98 11841.7 02500133.66 8260.9 03500134.68 6038.3 05000134.66 4378.3 09000141.63 2618.4 20000135.77 1450.3 Measuring cache-to-cache transfer latency (in ns)... Local Socket L2->L2 HIT latency20.5 Local Socket L2->L2 HITM latency20.7

[-]

twisted_nematic57@reddit

I've been looking for a tool like the one you linked to for half a damn year, Thanks so much for sharing it

[-]

MLDataScientist@reddit

8x 32GB 3200Mhz ECC DDR4 with AMD EPYC 7532 (32 cores). ``` Intel(R) Memory Latency Checker - v3.11 Measuring idle latencies for random access (in ns)... Numa node Numa node 0 0 120.3 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 142360.9 3:1 Reads-Writes : 136401.2 2:1 Reads-Writes : 138671.6 1:1 Reads-Writes : 138050.0 Stream-triad like: 137832.5 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 0 141531.3 Measuring Loaded Latencies for the system Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Inject Latency Bandwidth Delay (ns) MB/sec ========================== 00000 336.56 141470.7 00002 338.78 141004.8 00008 345.68 140769.2 00015 343.10 141687.5 00050 340.43 141820.5 00100 230.58 142649.0 00200 160.36 107313.9 00300 155.70 76437.6 00400 148.30 59313.4 00500 142.39 48673.2 00700 136.40 35597.9 01000 132.25 25398.1 01300 131.14 19810.0 01700 132.63 15328.2 02500 129.23 10681.3 03500 128.88 7807.8 05000 129.66 5625.8 09000 130.75 3350.5 20000 131.98 1772.8 Measuring cache-to-cache transfer latency (in ns)... Local Socket L2->L2 HIT latency 21.8 Local Socket L2->L2 HITM latency 22.3 ```

[-]

Turbo_mafia@reddit

Dual 9654 with 24 channel ddr5 4800 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 800632.9 3:1 Reads-Writes : 755604.7 2:1 Reads-Writes : 742822.2 1:1 Reads-Writes : 723441.3 Stream-triad like: 742547.5 https://preview.redd.it/357p0su9ya4e1.png?width=2576&format=png&auto=webp&s=2257aa2981bb5de3d21e1b1d3234cc6bd16ea157

[-]

jd_3d@reddit (OP)

Wow, 800MB/sec is GPU territory. How fast is it for CPU inference on the larger models?

[-]

Turbo_mafia@reddit

llm\_load\_print\_meta: [general.name](http://general.name)= Qwen2.5 Coder 32B system\_info: n\_threads = 312 (n\_threads\_batch = 312) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | LLAMAFILE = 1 | AARCH64\_REPACK = 1 | write a poem on the theme of "maya" in devanagari script Here's a poem on the theme of "Maya" (illusion) in Devanagari script: माया विश्वास बनाये जग में, लोगों के दिल में झूठी पहचान, भ्रमित जीवन में हम लड़ते हैं, आलोक में परिवर्� llama\_perf\_sampler\_print: sampling time = 20.70 ms / 131 runs ( 0.16 ms per token, 6330.03 tokens per second) llama\_perf\_context\_print: load time = 17454.02 ms llama\_perf\_context\_print: prompt eval time = 227.98 ms / 3 tokens ( 75.99 ms per token, 13.16 tokens per second) llama\_perf\_context\_print: eval time = 30794.18 ms / 127 runs ( 242.47 ms per token, 4.12 tokens per second) llama\_perf\_context\_print: total time = 31083.57 ms / 130 tokens

[-]

MLDataScientist@reddit

this does not look right. 4 tokens per second Text Generation (TG) for Qwen2.5 Coder 32B Q4\_K\_M? You should get around 800GB/20GB = \~40 t/s TG for that 20GB model. Here is what I get from my 8x32GB 3200Mhz ECC DDR4 with AMD EPYC 7532: ``` ./build/bin/llama-bench -m /media/ml-ai/wd_2t/models/Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf --threads 32 | model | size | params | backend | threads | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | ---------------: | ======================================= HAVE_FANCY_SIMD is NOT defined | qwen2 ?B Q4_K - Medium | 18.48 GiB | 32.76 B | CPU | 32 | pp512 | 37.23 ± 1.55 | | qwen2 ?B Q4_K - Medium | 18.48 GiB | 32.76 B | CPU | 32 | tg128 | 6.43 ± 0.03 | ```

[-]

Turbo_mafia@reddit

llm\_load\_print\_meta: [general.name](http://general.name)= Llama 3.2 3B Instruct system\_info: n\_threads = 384 (n\_threads\_batch = 384) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | LLAMAFILE = 1 | AARCH64\_REPACK = 1 | sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist generate: n\_ctx = 4096, n\_batch = 2048, n\_predict = 128, n\_keep = 1 llama\_perf\_sampler\_print: sampling time = 17.74 ms / 132 runs ( 0.13 ms per token, 7439.97 tokens per second) llama\_perf\_context\_print: load time = 3957.89 ms llama\_perf\_context\_print: prompt eval time = 63.68 ms / 4 tokens ( 15.92 ms per token, 62.82 tokens per second) llama\_perf\_context\_print: eval time = 6951.25 ms / 127 runs ( 54.73 ms per token, 18.27 tokens per second) llama\_perf\_context\_print: total time = 7061.19 ms / 131 tokens

[-]

dairyxox@reddit

I'm late to the show here, but I was testing my old system and wanted numbers to compare against to see if what I'm seeing is correct. # Dual Xeon E5-2618L v3 96GB DDR4-2400 6-channel: 81.2GB/sec Measured, 115.2GB/s Theoretical (6-channel). # The system supports 8 memory channels across the 2 nodes, but its not fully populated.

[-]

dairyxox@reddit

I finally got another couple of DDR4 modules and now am running 8-channel. *ALL Reads : 93692.0* *3:1 Reads-Writes : 90408.2* *2:1 Reads-Writes : 89774.9* *1:1 Reads-Writes : 84435.0* *Stream-triad like: 81771.6*

[-]

ElUnk0wN@reddit

Epyc 9755 12 Rdimm of Micron DDR5 64gb 5600mts https://preview.redd.it/91z2y6pf9a0f1.png?width=612&format=png&auto=webp&s=4653f1d84bf03813735b33c0e124d694769195d0

[-]

dionisioalcaraz@reddit

I'm planning to make a similar CPU only setup but with the cheaper Epyc 9115 and the supermicro H13SSL-N. Could you share a bench with a 32B or 14B model?

[-]

ElUnk0wN@reddit

Please dm me for whatever u looking for and I will run it for u!

[-]

kzoltan@reddit

Could you make the data public if you got it please?

[-]

ElUnk0wN@reddit

What data do u want me to do? Please dm me with your requests!

[-]

NBPEL@reddit

Strix Halo: 212 GB/s is 83% of its theoretical limit (256 GB/s) https://old.reddit.com/r/LocalLLaMA/comments/1kmi3ra/amd_strix_halo_ryzen_ai_max_395_gpu_llm/msbn0s7/

[-]

pixo2@reddit

CPU: Intel i7 - 12700 non K, all power limits unlocked RAM: 32GB DDR4-3200 CL16, 2 Channels 4 dims Using traffic with the following read-write ratios ALL Reads: 42089.0 3:1 Reads-Writes: 40168.3 2:1 Reads-Writes: 39299.3 1:1 Reads-Writes: 39090.0 Stream-triad like: 40996.6 Is that what you needed? I hope so.... say something if I did wrong.

[-]

un_passant@reddit

2×7R32 with 16× 64GB DDR4 @ 3200 but with only 48 cores used for the VM on Proxmox : 125.2 BG/sec

[-]

bakahk@reddit

CPU: AMD Ryzen 7 **8845HS** RAM: **96GB DDR5-5600** \#channels: **2** Intel Memory Latency Checker v3.11b: mlc --max\_bandwidth ALL Reads : **58509.98** \-------------- *HW: Beelink SER8 (8845HS) + Crucial CT2K48G56C46S5*

[-]

popecostea@reddit

Threadripper 5975WX split with 4 NUMA nodes for the CCDs. 3600MT/s 8 channel DDR4 CL18 > Measuring Peak Injection Memory Bandwidths for the system >Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) >Using only one thread from each core if Hyper-threading is enabled >Using traffic with the following read-write ratios >ALL Reads : 191076.0

[-]

LakersTriS@reddit

A bit out-dated rig but anyway... 12700k 3600C19-22-22-42 Intel(R) Memory Latency Checker - v3.11b Measuring idle latencies for random access (in ns)... Numa node Numa node 0 0 68.4 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 50915.3 3:1 Reads-Writes : 49055.1 2:1 Reads-Writes : 48421.0 1:1 Reads-Writes : 47550.9 Stream-triad like: 49003.2 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 0 50991.5 Measuring Loaded Latencies for the system Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Inject Latency Bandwidth Delay (ns) MB/sec ========================== 00000 217.71 50649.3 00002 217.31 50635.0 00008 209.68 50265.2 00015 197.73 49942.9 00050 178.58 49011.8 00100 161.49 47430.5 00200 98.59 34635.6 00300 86.02 24993.2 00400 82.27 19508.4 00500 79.19 16123.5 00700 77.89 11937.1 01000 75.04 8856.5 01300 74.46 7070.8 01700 73.80 5651.0 02500 72.52 4159.5 03500 71.52 3247.7 05000 71.57 2546.1 09000 70.32 1832.3 20000 70.20 1327.5 Measuring cache-to-cache transfer latency (in ns)... Using small pages for allocating buffers Local Socket L2->L2 HIT latency 31.5 Local Socket L2->L2 HITM latency 32.8

[-]

gmetothemoongodspeed@reddit

The Intel Xeon W-2255 has 4 memory channels not 8 in the table above. I’ve got a Xeon W-2235 with 4 memory channels, 128 ddr4 2933 ram in 4x32MB modules and get 60GB/s. When I ran the tool the cpu cores were all at 100% except for one virtual processor so I think I’m CPU bottlenecked. I get 0.93 response tokens per sec in llama3.3:70b and 7.68 tokens per sec in granite3.1:8b. No GPU, only CPU processing (headless server). Don’t know how to enable AVX512 in ollama windows build? Any tips?

[-]

L29Ah@reddit

i7-8550U 2400 MT/s 16GB R7416G2400S2S 1 channel (one DDR4 SO-DIMM) 16194.73 MiB/sec measured by `sysbench memory`

[-]

TheSilverSmith47@reddit

https://preview.redd.it/0xyjozawe7fe1.png?width=557&format=png&auto=webp&s=206816702317017305efd7e1d6bf0abcdfc6ba5f Core i7-11800H \~44 GB/sec

[-]

pyr0kid@reddit

oi u/jd_3d, is this post being maintained anymore? i see dozens of people are posting numbers but none of them are in the big comparison list.

[-]

Due-Accident1619@reddit

1. CPU: 2xIntel Xeon E5-2650 v2 RAM: 128 GB (16/16 slots with 8GB each) DDR3-1600 \# of mem. channels: 4 Measured Bandwidth: 71.9 GB/sec 2. CPU: 2xIntel Xeon Gold 6138 RAM: 384 GB (12/24 slots with 32 GB each) DDR4-2666 \# of mem. channels: 6 Measured Bandwidth: 215.9 GB/sec

[-]

altoidsjedi@reddit

* CPU: AMD Ryzen Zen 5 9600X * RAM: TeamGroup T-Create 32GB (16GB x2) set to AMD EXPO settings for 7200 MT/S * Mobo: Asus X670-P Prime * OS: Ubuntu 24.04.1 LTS  Intel(R) Memory Latency Checker - v3.11a Measuring idle latencies for random access (in ns)... Numa node Numa node 0 0 75.8 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 58224.3 3:1 Reads-Writes : 65039.6 2:1 Reads-Writes : 65401.4 1:1 Reads-Writes : 54446.4 Stream-triad like: 69844.9 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 0 58528.0 Measuring Loaded Latencies for the system Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Inject Latency Bandwidth Delay (ns) MB/sec ========================== 00000 173.42 58437.2 00002 173.93 58534.0 00008 174.60 58194.1 00015 169.40 58326.6 00050 169.58 58503.8 00100 168.71 58526.3 00200 114.55 48043.8 00300 100.49 35922.6 00400 94.33 28573.7 00500 92.21 23912.9 00700 91.01 18003.0 01000 90.01 13192.3 01300 88.43 10507.7 01700 85.41 8335.9 02500 84.91 5987.1 03500 84.66 4531.1 05000 84.71 3412.9 09000 84.83 2238.5 20000 84.51 1429.4 Measuring cache-to-cache transfer latency (in ns)... Local Socket L2->L2 HIT latency 15.8 Local Socket L2->L2 HITM latency 15.9

[-]

altoidsjedi@reddit

Update: Here is the exact same system (9600x CPU / X670-P Mobo) but with the RAM swapped out to [Crucial Pro RAM 96GB Kit (2x48GB) DDR5 5600MHz CP2K48G56C46U5](https://www.amazon.com/dp/B0C79RMMCL?ref=ppx_yo2ov_dt_b_product_details&th=1). I will next be testing (and probably settling on) TeamGroup 96GB DDR5-6400 CL32 RAM. Results will be shared from that, but in the meantime, here is the results of the DDR5-5600: Intel(R) Memory Latency Checker - v3.11a Measuring idle latencies for random access (in ns)... Numa node Numa node 0 0 97.1 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 50051.4 3:1 Reads-Writes : 51725.9 2:1 Reads-Writes : 52043.8 1:1 Reads-Writes : 46461.1 Stream-triad like: 53508.5 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 0 49579.9 Measuring Loaded Latencies for the system Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Inject Latency Bandwidth Delay (ns) MB/sec ========================== 00000 213.45 48215.9 00002 211.85 48297.9 00008 218.52 48500.3 00015 199.17 49633.5 00050 192.48 50254.3 00100 205.84 49052.0 00200 187.69 47245.6 00300 130.70 35474.6 00400 123.64 28160.3 00500 123.10 23692.0 00700 120.33 17881.7 01000 120.12 13034.6 01300 113.85 10367.8 01700 109.21 8193.2 02500 108.39 5836.6 03500 107.99 4377.8 05000 109.81 3243.2 09000 110.40 2065.5 20000 115.52 1087.7 Measuring cache-to-cache transfer latency (in ns)... Local Socket L2->L2 HIT latency 16.6 Local Socket L2->L2 HITM latency 16.1

[-]

nilesism@reddit

Dual 8259CL 2nd Gen Xeon with 12x 16GB DDR 2666: Using traffic with the following read-write ratios ALL Reads : 222073.7

[-]

lolzinventor@reddit

* CPU: 2x Xeon Platinum 8175M * RAM: 384GB DDR4 (2400 over clocked to 2667) * Motherboard EP2C621D16-4LP  ALL Reads :189321.5 3:1 Reads-Writes :175723.9 2:1 Reads-Writes :173617.4 1:1 Reads-Writes :162475.6 Stream-triad like:162950.8 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 1 080012.1 34531.9 134504.6 113786.8

[-]

slavik-f@reddit

I have DELL T7920 system, which supports 6 memory channels (2 DIMMs per channel) One Xeon Gold 5218, 8 x 16GB DDR4-2400 RAM, I got 99GB / sec: ALL Reads : 99033.6 Theoretical limit for 6 channels DDR-2400 is 115GB/s, which I think I'll have if get 4 more DIMM on my system.

[-]

Eisenstein@reddit

One more to add. Intel(R) Memory Latency Checker - v3.11a Measuring idle latencies for random access (in ns)... Numa node Numa node 0 0 95.9 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 26910.0 3:1 Reads-Writes : 27403.3 2:1 Reads-Writes : 27442.7 1:1 Reads-Writes : 27777.0 Stream-triad like: 27450.9 CPU: Name MaxClockSpeed NumberOfCores NumberOfLogicalProcessors ---- ------------- ------------- ------------------------- 12th Gen Intel(R) Core(TM) i3-12100F 3300 4 8 Dual channel, I forget why I disable XMP but I think memory bus speed isn't a huge concern on this system: Capacity Speed Manufacturer -------- ----- ------------ 8589934592 2133 PNY Technologies Inc 17179869184 2667 PNY Technologies Inc 8589934592 2133 PNY Technologies Inc 17179869184 2667 PNY Technologies Inc

[-]

Eisenstein@reddit

Some more stats for you: Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 78261.7 3:1 Reads-Writes : 71094.4 2:1 Reads-Writes : 71210.9 1:1 Reads-Writes : 68339.6 Stream-triad like: 70284.7 8x (quad channel) Total Width: 72 bits Data Width: 64 bits Size: 16 GB Type: DDR3 Type Detail: Registered (Buffered) Speed: 1333 MT/s Manufacturer: Hynix Semiconductor Part Number: HMT42GR7AFR4A-H9 Rank: 1 2x Version: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz Core Count: 10 Core Enabled: 10 Thread Count: 20 System Information Manufacturer: Dell Inc. Product Name: Precision T7610 Not bad for an old-timer.

[-]

No_Afternoon_4260@reddit

u/jd_3d core ultra 7 155H 32GB, LPDDR5-6400, 2 channels Base should be 102 GB/sec ALL Reads : 76751.7 3:1 Reads-Writes : 73796.8 2:1 Reads-Writes : 71944.4 1:1 Reads-Writes : 70181.0 Stream-triad like: 73939.8

[-]

CoqueTornado@reddit

nice numbers! anyway, do you use the igpu/npu for inferencing? how many gigas of vram does it have? thanks

[-]

No_Afternoon_4260@reddit

No igpu/npu inference, 8gb of vram with a 4060 Here are some numbers: Since the 155H is a laptop chip I'll include numbers with gpu. * core ultra 7 155H, 32GB LPDDR5-6400, nvidia 4060 8GB, nvme pcie 4.0 # 70b q3K_S GPU 16 layers Vram = 7500, ram = 4800 * -31.14 seconds, context 1113 (sampling context) * -301.52 seconds, 1.27 tokens/s, 383 tokens, context 2532 (summary) # 70b q4K_M GPU 12 layers Vram = 7800, ram = 4800 * -301.47 seconds, 0.12 tokens/s, 36 tokens, context 1114 # 70b q3K_S CPU only Vram = 0, ram = 5200 * -301.47 seconds 0.12 tokens/s, 36 tokens, context 1114 * -249.40 seconds, 0.15 tokens/s, 37 tokens, context 2704 # 8x7b q4K_M 5/33 layers GPU Vram = 7000, ram = 9000 * -138.03 seconds,, 3.71 tokens/s, 512 tokens, context 3143 * -107.35 seconds, 4.43 tokens/s, 476 tokens, context 3676 If I'm not mistaking this is nvme inference, because I have only 32gb ram, my ssd is pcie 4.0 mesured at 7gb/s read in crystalDiskMark to give you an idea. Why isn't part of it in system ram I don't know, this is llama.cpp. May be the true bottleneck is the cpu itself and the 16-22 cores of the 155h doesn't help. So llama goes to nvme. But if you are in the market for llm workload with 2k+ usd you better get some 3090s and good ddr5 system or adm epyc if you want to expend to more than 2 gpu. Check those pcie lanes, you prefer 4.0 and plenty of them only if you want to train.

[-]

CoqueTornado@reddit

how does it go in a model of 8B with EXL2 in tokens/second with that 4060ti? and with guff? one with all the layers unloaded. I would like to know the speed of that gpu card. Interesting data where the MOE has more ram loaded but it goes 4 times faster than the one with 4800 of ram loaded. Probably due to their architecture of 2 experts used in the same time

[-]

No_Afternoon_4260@reddit

A 8b q5km is about 18tk/s 8b q8 is about 9 tk/s with 27/33 layers offloaded to gpu This is all gguf, I don't have any exl2 on that laptop

[-]

CoqueTornado@reddit

cool!

[-]

CoqueTornado@reddit

yep, that is the way, or even better a 4060ti 16gb of vram route, not expensive when they get lower below 350€. I hope they reach that point someday soon. One motherboard with 2 4.0 x8 and one x4 and it would make the 48gb of vram with three of these or two of these plus one p100; something like that I am thinking of

[-]

No_Afternoon_4260@reddit

This 155h is a laptop chip so I'm talking about a 4060 laptop with 8gb of ram, if you want 16gb of vram in a laptop you need to reach for a 4090 wich will be found in 5k+ laptops. (Some "cheap" laptop with 3060ti are about 2k+ second hand) If you want to do a budget build you can look at those 3060 with 16gb they are about 250-300usd. You can use pcie bifurcator to divide a x16 port in x8x8 ports. I only found bifurcator for pcie 3.0, if you find a bifurcator for pcie 4.0 please tell me haha.

[-]

CoqueTornado@reddit

3060 with 16gb??? aren't they 12gb? is a frankestein?

[-]

No_Afternoon_4260@reddit

Ho may be 12gb yeah sorry

[-]

CoqueTornado@reddit

[-]

CoqueTornado@reddit

anyway the tokens / second of that laptop are really low for your ram... I've read somewhere it should be 1tkps not that 0.15tkps

[-]

No_Afternoon_4260@reddit

It is a 8gb vram laptop, don t expect it to be better than it is. (This is linux) the vram is like 200gb/s bandwidth, the ram is about 100gb/s. Nothing compared to the near 1tb/s of vram in a 3090 or 4090

[-]

CoqueTornado@reddit

but: 3:1 Reads-Writes : 73796.8 So, 73GBps of bandwidth in CPU for real. I know for sure the 949gb/s of these cards, you can't compare. But doing maths, 73gigabytes per second is a lot. You can move 2 of 36GB every second. So it should have 2 tokens per second if you know what I mean. As long as it is a laptop, you move 73GBps instead of the 100GBps it should. But that is the reduction. GeForce RTX 4060 Ti indeed has a memory bandwidth of 256GBps, so is x2.5 times faster. If you had 36GB of VRAM you could move at these speeds that model 70b q3K\_S faster than 0.15x2.5 if you know what I mean. I think you should have 1 token per second with that CPU inference at least moving that 33GB model q3K\_S 70b I've seen that dozens of times. They say without the 3090 I have 1.4tokens per second, with the 3090 I reach 2.5tkps So your ddr5 should go faster; please consider look for a newer way to do the inference, updating CUDA, Cudnn or pytorch or whatsoever... this is too low even for a laptop... mhmmm maybe is the processor, is that slow? 24941 points here: [https://www.cpubenchmark.net/cpu.php?id=5677](https://www.cpubenchmark.net/cpu.php?id=5677) not too slow, is over the averague; I have a laptop with 9000 of score... so... [AMD Ryzen 7 7800X3D ](https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+7+7800X3D&id=5299) [ 34,357](https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+7+7800X3D&id=5299) For comparision that one has just 10000 more points. Is not that slow your cpu. So I don't really understand that slow token per sec

[-]

CoqueTornado@reddit

maybe is windows?

[-]

CoqueTornado@reddit

interesting the bifurcator thing also there are PCI 5.0 nowadays, does it make sense with that gpu?

[-]

No_Afternoon_4260@reddit

I'm in the 3090 area, so pcie 4.0 is good to me, Keep in mind pcie 3.0 x16 = pcie 4.0 x8 = pcie 5.0 x4 in bandwidth

[-]

CoqueTornado@reddit

Q3\_K\_S is around 32gb, so having the speed of 71GBps it makes 0.15 tokens/s... unbelievable! it whould be around 2tokens/s if I am right; I don't understand anything :D

[-]

SoftwareRenderer@reddit

Dual Xeon 6126, 6 channel 192GB DDR4-2666 I'm guessing the benchmark's reported 193GB/s is combining bandwidth from both cores, since the theoretical peak is only supposed to be 128GB/s. Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 193403.4 3:1 Reads-Writes : 182445.4 2:1 Reads-Writes : 183083.9 1:1 Reads-Writes : 183494.0 Stream-triad like: 162273.0 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 1 0 97050.8 34001.9 1 34010.8 96882.1

[-]

AstronomerCareful551@reddit

Here are my machines: * Intel(R) Core(TM) i9-14900K 96 GB (2x48 GB) DDR5-6000 ALL Reads : 88439.0 3:1 Reads-Writes : 85024.7 2:1 Reads-Writes : 84382.8 1:1 Reads-Writes : 83130.2 Stream-triad like: 84298.5 * Dual Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz (HT disabled) 256 GB (16x16 GB) DDR4-2133 MHz ALL Reads : 120239.4 3:1 Reads-Writes : 113751.2 2:1 Reads-Writes : 111850.8 1:1 Reads-Writes : 100587.0 Stream-triad like: 107004.3

[-]

CoqueTornado@reddit

great numbers! in the 96GB setup how much do you achieve in 70B Llama3 q4\_K\_M model? (gpu and without) Thank you!!!!

[-]

AstronomerCareful551@reddit

llama.cpp GPU - llm\_load\_tensors: offloaded 42/81 layers to GPU llama_print_timings: load time = 2479.12 ms llama_print_timings: sample time = 69.70 ms / 159 runs ( 0.44 ms per token, 2281.07 tokens per second) llama_print_timings: prompt eval time = 13850.79 ms / 71 tokens ( 195.08 ms per token, 5.13 tokens per second) llama_print_timings: eval time = 48264.39 ms / 158 runs ( 305.47 ms per token, 3.27 tokens per second) llama_print_timings: total time = 64376.91 ms / 229 tokens CPU llama_print_timings: load time = 1834.10 ms llama_print_timings: sample time = 64.07 ms / 144 runs ( 0.44 ms per token, 2247.44 tokens per second) llama_print_timings: prompt eval time = 26920.37 ms / 71 tokens ( 379.16 ms per token, 2.64 tokens per second) llama_print_timings: eval time = 82457.16 ms / 143 runs ( 576.62 ms per token, 1.73 tokens per second) llama_print_timings: total time = 112545.14 ms / 214 tokens

[-]

CoqueTornado@reddit

3090?

[-]

AstronomerCareful551@reddit

4090

[-]

CoqueTornado@reddit

thanks!

[-]

LuxuryFishcake@reddit

i9 9900k and 32gb ddr4 3600 dual channel Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 28809.6 3:1 Reads-Writes : 27309.5 2:1 Reads-Writes : 26581.4 1:1 Reads-Writes : 26769.2 Stream-triad like: 26429.3 Measuring Maximum Memory Bandwidths for the system Will take several minutes to complete as multiple injection rates will be tried to get the best bandwidth Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 28936.77 3:1 Reads-Writes : 27381.59 2:1 Reads-Writes : 26991.74 1:1 Reads-Writes : 26724.43 Stream-triad like: 27284.67

[-]

ephem3ros@reddit

i7-12650H 16G 4800MHz + 16G 5600MHz, running at 4800MHz both, no overclocked PS D:\mlc_v3.11\Windows> .\mlc.exe Intel(R) Memory Latency Checker - v3.11 Measuring idle latencies for random access (in ns)... Numa node Numa node 0 0 94.7 Measuring Peak Injection Memory Bandwidths for the system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using traffic with the following read-write ratios ALL Reads : 65026.9 3:1 Reads-Writes : 60563.1 2:1 Reads-Writes : 59790.1 1:1 Reads-Writes : 59419.1 Stream-triad like: 60115.2 Measuring Memory Bandwidths between nodes within system Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Numa node Numa node 0 0 64882.2 Measuring Loaded Latencies for the system Using all the threads from each core if Hyper-threading is enabled Using Read-only traffic type Inject Latency Bandwidth Delay (ns) MB/sec ========================== 00000 144.87 64303.5 00002 148.34 63860.5 00008 144.59 63750.8 00015 141.10 63116.7 00050 135.75 63092.2 00100 126.49 60671.7 00200 106.92 43307.2 00300 103.83 30724.0 00400 102.82 24151.8 00500 104.13 19872.4 00700 103.35 14776.4 01000 102.57 10772.2 01300 101.85 8546.4 01700 101.35 6750.5 02500 100.40 4844.0 03500 99.62 3674.5 05000 99.46 2775.5 09000 98.98 1837.0 20000 100.19 1177.2 Measuring cache-to-cache transfer latency (in ns)... Using small pages for allocating buffers Local Socket L2->L2 HIT latency 38.2 Local Socket L2->L2 HITM latency 35.4