Rolling out lots of bare metal linux server -looking for the best way to do it

[-]

ohmer123@reddit

I think one limit to the exercise is to think only about day 1. A better way to tackle this one is to think about the complete lifecycle of these compute resources. The PXE approach, fairly common, but how do you test changes? How do you deploy changes (package update, configuration update...)? If you go for NFS root, you might get an I/O bottleneck and availability challenges. Depending on what you do, might be ok or not. And, not all OS support this type of use case. Could you tell us more about what you need to run?

Reply

[-]

Cooleb09@reddit

How has no one mentioned satelite or foreman/katello yet?

Reply

[-]

SMLLR@reddit

Thought the same while reading through the comments. Definitely the best answer, though I’m not sure somebody asking this question would be able to get it running in a reasonable timeframe.

Reply

[-]

rainer_d@reddit

It’s not for the faint of heart. Even though it is the right answer….

Reply

[-]

mikeegg1@reddit

Ansible.

Reply

[-]

abyssea@reddit

Please don't tell me this is for MGM....

Reply

[-]

wezelboy@reddit

PXE boot to kickstart

Reply

[-]

cheetahwilly@reddit

Yeah, this is the first part for sure. I am sure redhat has some great Automation features at this point, just gotta read up.

Reply

[-]

wezelboy@reddit

RedHat owns Ansible, so there's that.

Reply

[-]

Hashrunr@reddit

You don't miss!

Reply

[-]

aracheb@reddit

Terraform

Reply

[-]

mvincent12@reddit

In order to use Terraform don't you need the libraries somewhere since they are using bare metal? We looked into this and we were going to need to install/setup a Digital Rebar instance and that quickly became way more complicated than a simple kick start server. Love it for cloud though.

Reply

[-]

aracheb@reddit

Yes, you would need to hold the libraries somewhere.

Reply

[-]

rainer_d@reddit

Terraform does not work on Bare Metal. Are we really at a point where people believe that the „servers“ they provision @cloud are bare metal?

Reply

[-]

mvincent12@reddit

This is the way!

Reply

[-]

davis-andrew@reddit

I'm not sure it's state of the art anymore, but at $dayjob we use FAI for this and it's pretty rock solid. pxe boot, mounts a base `/` filesystem over NFS then uses (off the top of my head) and then overlays a tmpfs so it's writeable (while keeping the NFS pristine). Based on some hardware identification FAI partitions the disks, mounts the target root filesystem, untars a base filesystem before finally chrooting into it and kicking off our standard config management to provision the machine into whatever we want it to be. At the end it reboots and with only a few exceptions is ready to do whatever it's job is.

Reply

[-]

sysadreq@reddit

Is there a fully automated way to do this? With minimal effort for the sysadmin who is building the server?

Reply

[-]

cheetahwilly@reddit

MGM eh

Reply

[-]

bestdriverinvancity@reddit

Help first day of a new job and I got to roll out 50 servers

Reply

[-]

tantrrick@reddit

Imagine getting that 100/hr and using Reddit to figure this out

Reply

[-]

Cooleb09@reddit

TBH I don't feel nearly as bad over that as the MSP bros billign at $150/hr and using reddit to figure out 101 shit.

Reply

[-]

GeorgeWmmmmmmmBush@reddit

Except the tech there is making $25 an hour.

Reply

[-]

bigfoot_76@reddit

Well the MSP is billing $150 and the poor sap working the tickets get $17 of that. Fuck it I’d be using Reddit too.

Reply

[-]

PoliticalDestruction@reddit

We should get a cut!

Reply

[-]

loadnurmom@reddit

Look up "cobbler" It will create a pxe server, along with dhcp. Configure your images in cobbler and you can reimage everything quickly This is how many HPC will do their clusters of hundreds. They could reimage several hundred systems in an hour or two

Reply

[-]

rainer_d@reddit

The latest upstream version is only available/packaged for SLES/Opensuse. But there’s a less recent version in EPEL. It should do the trick. You could look into Foreman, if you want complete lifecycle management of servers.

Reply

[-]

loadnurmom@reddit

It's been a while since I used cobbler so I didn't know, thank you There's also "Warewulf" which loads an image every time a system boots, but it doesn't actually install to the drive. It's a little slower to boot up, but very effective. New image created to fix bugs? Just reboot the system.

Reply

[-]

WonderousPancake@reddit

Well if you’re setting them up from scratch then you already need to setup the raid…

Reply

[-]

sryan2k1@reddit

Which can be scripted on any sane platform (Dell, HPE, etc)

Reply

[-]

Anonymous1Ninja@reddit

If they are physical clonezilla If they are virtual just duplicate the disk. If you they are virtual you can easily cline within the hypervisor.

Reply

[-]

DLRXplorer@reddit

The beauty of Ansible is you can tweak a setting down the line and update everything at once. So it's a great deployment tool but also a compliance and management tool.

Reply

[-]

Turbulent-Pea-8826@reddit

What brand servers? Dell servers allow you to image the server through the idrac and I believe you can automate the imaging through open manage. So build your first server, create a golden image. Install open manage then setup the imaging stuff. Add servers. Profit.

Reply

[-]

jeevadotnet@reddit

We use openstack ironic and Ubuntu MAAS

Reply

[-]

Zehicle@reddit

"identical" usually means via imaging which can be tricky. If you just need them nearly identical then you can net boot / PXE. Our company, RackN, made an explainer video with these options that's been really helpful: https://youtu.be/w_ZGlxihlEI?si=T9rlzbE58_8oHjwS We also make a commercial bare metal provisioner ([Digital Rebar ](https://rackn.com/rebar)) that supports PXE and imaging provisioning. API and IaC focused with a lot of features.

Reply

[-]

darknekolux@reddit

I’ve heard about https://maas.io/ never tried thought

Reply

[-]

Numerous_Ad_307@reddit

Ultimate deployment appliance still a thing?

Reply

[-]

Lammtarra95@reddit

Consider using RHEL Image Builder.

Reply

[-]

najmalesinw@reddit

Ugh, rolling out servers is such a pain. I've done it manually before and trust me, you don't wanna go down that road. Kickstart script sounds like the way to go, but Ansible could work too if you're feeling fancy. Just make sure to test everything first before unleashing those bad boys into production. And for the love of all things techy, stay away from Clonezilla. That thing's a disaster waiting to happen.

Reply

[-]

AutomaticAssist3021@reddit

As long as I needed clonezilla it worked like a charm

Reply

[-]

sanitarypth@reddit

I’ve never had a problem with Clonezilla. What’s the fear?

Reply

[-]

asachs76@reddit

Checkout tinkerbell.org for bare metal provisioning.

Reply

[-]

CyberHouseChicago@reddit

Build own clone it and just change ips once done can be scripted , can even be scripted they hypervisor depends how your doing it

Reply

[-]

zrad603@reddit

or really long DHCP lease/reservation.

Reply

[-]

CyberHouseChicago@reddit

Dhcp can work for setup also did not think of that, I don’t run dhcp where I have servers

Reply

[-]

robvas@reddit

Python scripts to do all the DRAC stuff (BIOS and PERC settings etc), kickstart for all the other stuff

Reply

[-]

maschine2014@reddit

Ansible as well i recently did this for dell servers and it worked great. [https://docs.ansible.com/ansible/latest/collections/dellemc/openmanage/index.html](https://docs.ansible.com/ansible/latest/collections/dellemc/openmanage/index.html) [https://www.ansible.com/blog/automating-a-rhel-8-installation-using-the-vmware-rest-ansible-collection](https://www.ansible.com/blog/automating-a-rhel-8-installation-using-the-vmware-rest-ansible-collection)

Reply

[-]

EZtheOG@reddit

Hard to say what your setup is, if you have to configure any bios stuff or whatnot. But a kickstart script would work or ansible is a fine idea. I think I like ansible cause it connects ssh and you can get some level of “fail” logs at the deployment console. You could use salt, or chef, or bla bla bla but what you suggested works.

Reply

Reply to Post

47 Comments