Looking for people who have configured really fast booting Linux images.
Posted by Unknown601@reddit | linux | View on Reddit | 36 comments
Hello Linux enthusiasts!
I'm looking for someone with experience in configuring an image that can boot in <2 seconds on an RK3566-based ARM board. This is, of course, paid work :)
The work:
Build a minimal Linux image (likely Yocto or Buildroot) targeting RK3566.
Optimise boot chain (u-boot, kernel, init, rootfs) for fast startup.
Strip down drivers and services to the absolute minimum needed.
Tweak
If you have relevant experience, please send me a DM.
chibiace@reddit
no systemd?
Unknown601@reddit (OP)
We are running a single application on it. So happy to look at without systemd.
jonsca@reddit
If you're running a single application, do you need an OS at all?
BraveNewCurrency@reddit
This is FAR harder than you think. (And really not the sub to discuss it.)
How do you put something on the screen? Heck, how do you configure the various display modes to enable putting something on the screen? How do you do networking? How do you handle buffering? How do you partition your firmware so you don't accidentally overwrite settings? How do you load the drivers that contain your storage? (i.e. Let's say it's on a SD Card). How do you load the executable? How do you make multiple things happen at once (say you want 'hold this button down to shut down')? It's trivial with an OS, but complicated with a single binary (How do you ensure that nothing will ever block your main loop?) How do you deal with out of memory?
It's not like you can't do these things without an OS. But each one must be thought about, decided on, prototyped, etc. Often it's much faster to start with Linux, then tweak the size/speed to what you need.
Unknown601@reddit (OP)
Yeah, Bare metal has its use case, but I think it would way too much time and complexity, especially since we are dealing with gstreamer and graghics on a touch screen. If we needed to be going down to sub 1s then it might be the way to go, but no required for this.
BraveNewCurrency@reddit
Linux can usually easily boot in a second. (I've built embedded devices that boot to serve the network in 2 seconds, and boot to custom GUI in 5 seconds.)
Things that slow it down are: BIOS/firmware/bootloader delays, and thousands of potential drivers that Linux has to probe around to find that slow down the boot. The trick is to set goals ("firmware should take no more that 200ms, linux boot should take 700ms, application should take 100ms"), then optimize until you are under the goal. You will often reach a point where it will take 100x more work to shave of another 10ms. In that case, focus on a different area instead.
https://www.e-consystems.com/articles/Product-Design/Linux-Boot-Time-Optimization-Techniques.asp
jonsca@reddit
Oh, completely agree with you. It depends 1000% on what the "one application" is. My question was more rhetorical ("can you personally get away without one in your use case?" and not a "could someone?" but I can see how my wording was ambiguous. Bare metal does eliminate the question of "How do you load the executable?" and using other things like maskable and non-maskable interrupts take care of your how do you not block the main loop and reset questions. SOCs have a lot of fancy stuff baked in.
vip17@reddit
sometimes yes, because the OS handles drivers and multithreading for you
Terewawa@reddit
How would an app run without an OS? Access the hardware directly? Is that doable?
jonsca@reddit
People do bare metal with SOCs all the time. Busing a lot of data from CPU to GPU might be a little tedious but if they share a memory mapping, then it's just a matter of rolling your own scheduling. It all really depends on what the "one app" is and how much control they have over it. There are of course saner intermediate steps like some kind of RTOS.
tonibaldwin1@reddit
Unikernel
Holiday_Floor_2646@reddit
Can't you just run the app as init? Linux can use any binary as init, as long as its statically linked.
TRKlausss@reddit
The problem is reaching the point where you run that app as init. He wants that to be fast.
Holiday_Floor_2646@reddit
if they build the kernel with -march=native and use efistub should be enough? and obviously a minimal defconfig.
5c044@reddit
rk3566 and most ARM systems don't have EFI - they have DTB device tree
archontwo@reddit
I see you already asked on /r/embedded.
The answer there is valid and I would only add that u-boot plus some optimised init, kernel image and filesystem can get boot times down to a couple of seconds.
MarzipanEven7336@reddit
I can get it down to 200ms. Been there done that.
archontwo@reddit
Yup. I did that years ago when I developed on a POS terminal. Trouble was the clients were so used to seeing a splash screen I has to add one so they knew when it restarted.
Oh well, their money was still the same so..
activedusk@reddit
What are some examples of optimised init systems. Asking this more for faster PC boot time.
archontwo@reddit
See this.
activedusk@reddit
This sent me in a mild panic
>systemd-bootchartcan be run at boot-up by addinginit=/usr/lib/systemd/systemd-bootchartto the kernel command line.I added it to the GRUB linux command line, big mistake. It got me to a text message that basically said it could not read the teletype and "you are on your own" and left me at a command line.
Luckily I remembered I could edit the command line by rebooting and holding Shift, that allowed me to go into the Grub boot list, selected advanced then the kernel version it usually boots with, pressed e and got me in an Emacs text editor and there it was, I deleted the bootchart command and pressed F10 and it booted. Followed up by editing GRUB and removing it. This is the best time I can get right now.
https://imgur.com/a/iowgL1h
archontwo@reddit
Part of the learning experience. Relish in it.
BraveNewCurrency@reddit
See also: https://lwn.net/Articles/299483/
2rad0@reddit
That will consume many hours, you might want to throw out some more info and ballpark numbers to get a bite.
codeasm@reddit
There is a defconfig you can pick, drop any modules you dont need, check what RK3566 minimally needs and just try build the kernel. https://embetrix.com/2017/05/16/embedded-linux-fast-boot-techniques/ might help, one of the first google results.
It be a fun exercise, definitly costs some time (aka money) to get it right. make anything not directly needed a module, or drop it entirely (usb stack? not needed to boot the kernel + splash welcome for example).
There are indeed some specific things that might help speed up the boot process if its known what the target look and feel is (and why the fast bootup). for example, ive setup my PC system without "bootloader", the uefi firmware has one if i need to select a different file, and its a button press away. else, just boot defaults. So does uboot and your sysv/init or systemd process chain also matter, especially when running or starting.
what about crash recovery. do we just use a ramfs thats clean every time or have persistant storage. can help in fast booting from a crash or not.
2rad0@reddit
I was thinking about the requirement "absolute minimum". To me implies no config option checked that doesn't have to be. Don't know if fun is the word I'd use, it's definitely a learning experience though. The strategy would be to find a minimal booting config, then start cutting options. VM might work for some options, but PXE boot might be needed unless you want to be swapping sdcard, usb, or however the thing boots. You need to know exactly what the target hardware needs to boot and function (not listed here), and what the program does + kernel features needed to operate correctly (also not listed) to estimate how long such a project will take.
That's what I mean we need more info, the absolute fastest option is the default initramfs but maybe the program needs swap support or other requirements so we'd have to pivot to a tmpfs. The actual logic of crash recovery is up to OP, and if they even want to be mounting a permanent storage device. Maybe you could blend that in as a background task after it's booted, or let's say half-booted, after init runs and before it forks to execve.
All I can assume for certain about the application is that if OP's program forks at all it will need a minimal init program to reap zombie processes, and to prevent a kernel panic if the program crashes.
All that being said, doing this for an embedded ARM board is probably much easier than x86, which has decades of junk drivers and features to confuse the hell out of you when tuning the .config file. SO maybe I'm overreacting.
codeasm@reddit
Your not wrong tho. not overreacting, we definitely would need more details to even begin to create a mental scope of what tasks lie before us. For whoever takes up the task, "by the hour", not "this is the amount" and in writing
zhongcha@reddit
Not me but I'm curious what it's for? Good luck!
Unknown601@reddit (OP)
It's for a thing that goes in a thing :D
ProjectSnowman@reddit
Totally NOT a ballistic missile right? 😉
pppjurac@reddit
And no automatic sentry gun too
Droppedfromjupiter@reddit
About half the population already has one of those! ;)
zhongcha@reddit
Amazing! :VV
bullshitwascalled@reddit
That's not hard you're limited by how long it takes to load the kernel, but you will have to remove all of the verification. No DM verity no hashing no signing. Compile kernel modules into the image, audit drivers based on hardware. That'll get you 90% of the way there. Then just strip system services down to the barebones. I would not recommend it.
Horror_Hippo_3438@reddit
My message is not for OP, but for those who want to work for OP.
In January, I immersed myself in learning Linux and built a minimal system with u-boot, kernel, init, rootfs for the ARM64 architecture, which started in 1.26 seconds https://2ch.su/s/src/3639805/17576770056620.png
I am absolutely sure that I could run this on RK3566.
But I am not sure that it will run in 2 seconds on RK3566 because I ran it in a QEMU VM on a more productive computer with i5-12400, DDR5, NVME Samsung EVO SSD. And I don't have RK3566 to check.
I did it just for fun. I'm not interested in money.
lennox671@reddit
It can definitely be done. At work on a single core cortex A7 I got the complete boot to about 4s, and on my Orange Pi 2w i use to mess around boots in around 3s from an sdcard without optimizing much.