How does file downloading actually work (memory, disk, chunks, etc.)?
Posted by Lower_Owl_1251@reddit | Python | View on Reddit | 2 comments
I wanna understand how the whole file downloading process actually works — like file download, memory allocation, disk writing and all that stuff.
For example, I see tools like aria2c can download huge files (like 10GB) super fast while using only around 10–30MB of RAM. But when I write a Python script to download files in chunks (say 50MB chunks) and run multiple downloads (3–4 files at once), my RAM usage goes up to 700–800MB.
So I’m curious — how does the system really handle downloads?
How do RAM-based and disk-based downloads differ?
How is the data stored temporarily before writing to disk?
How can a tool like aria2c manage memory so efficiently compared to something like a Python script?
What’s the actual mechanism behind “chunk downloading” and disk writing?
If anyone has YouTube videos, Reddit threads, or good articles explaining how file download systems work internally (memory management, buffering, I/O operations, etc.), please share them! I really wanna understand this part deeply.
Top_Average3386@reddit
Would you be willing to post your script? Maybe someone can help point out what's consuming so much memory aside from the obvious overhead of launching the interpreter itself.
UysofSpades@reddit
Well first aria2c is written in a language called C++ which is blazing fast and low memory footprint because of the nature of the language. Your Python script has to load not only the entire Python interpreter into memory but also all the baggage that comes along for Python to work BEFORE you script downloads anything.