Dell PowerEdge R760XS XL Dives (RAID)
Posted by Aromatic-Coconut-122@reddit | sysadmin | View on Reddit | 27 comments
Ok, here goes.
We just received the Dell PE R769XS XL. It has 12 bays for 3.5 drives, but only 8 are occupied with Seagate Exos X18 ST16000NM002J 16TB drives.
This is for a Milestone VMS system and storage is paramount. When we ordered, the vendor messed up and ordered 8 instead of 12 drives. We didn't pay the difference, so it was kind of a no harm no fowl because I ordered the server with 12 x 12TB drives.
I went to support for the service take of this server to order parts. I guess Dell doesn't do this for enterprise servers so a call to support was made. Three technicians pretty much thought I was on something because they claimed they weren't building server with 16TB drive; only 12TB. Each looked up the configurations and choked at the fact I did have 16TB drives. The last tech was like "No problem, let's get after sales support and sales on the phone so we can see about getting the four drives you want. No one could help. They only sell 12TB drives of this particular model. Everyone hung up except the post sales support.
He gave me all of the Seagate informations and even got Seagate on the phone. Her s where things get weird and my question comes in. Seagate has the drives, just without Dells firmware. "OK" I said and asked "What does this mean to me?" Supposedly Dells firmware is "Optimized" and proprietary. Bot eh Dell tech and Seagate rep stated we could still use the vanilla version of the Seagate drives and we wouldn't experience any problems. The Dell firmware "is designed to communicate with the RAID controller more efficiently, but since you're storing video, this would not affect the performance of the array"
They both went back and forth on if we were building this array as a file server, small files would be extremely slow with the mixed drives but would be every bit reliable as the Dell labeled drives"
Now a Dell engineer enters the call. He basically confirmed the previous conversation, but reminded the support tech that we would receive no support for the RAID unless they were the Dell version of th drives. Ok, but you don't have them available, yet I can order them from Seagate, or from Amazon.
Longer story shorter, the conversation went on and on and ended up with "Well, if the array fails and it's a Dell branded drive, we'll handle the replacement via shipment. If th Seagate labeled drive fails, you'd go through Seagate"
I was ok with that. But I'm not fully convinced. The 12TB Dell drive is $1200
The 16TB is around $300 + $25 for a four back of sleds, surprising sold by Dell. I have two options here, because losing the extra storage isn't an option, I can build the array as it is, and when the four Seagate come in, build a second array and use the VMS software to use both arrays as available storage, but this limits me to RAID 5 X2 which means I'm losing 32 TB of storage
Or
Chance it and build a RAID 6 with mixed drives,
Or
Set the controller for AHCI, take the performance hit and let Windows Server handle the array via software.
My questions here are,
What do you all think is the best solutions since the Dell versions aren't available from Dell (and procurement will not let us buy them from unapproved companies that specialize in server parts)?
And the more minor question is, is it possible to flash, if available, the Dell Firmware to the Seagate drives since it's been confirmed by both Dell and Seagate that they're exactly the same drives?
The server is currently in configured and I popped an 8TB Seagate in and loaded into the RAID controller configuration and it showed the 8TB without errors, different light colors, or alert beeps.
If I can't get this sorted, I have to swap the same server that in production out with the new one just to have more storage. I obviously can't do that.
I'm about to say f it and go back to Law enforcement. Haha!
Hoosier_Farmer_@reddit
call them back and ask for the dell part number (get it off your current drive, but it's probably dell dp/n 0jwv1p)
Aromatic-Coconut-122@reddit (OP)
Ok, so... Yes Dell has "optimized" firmware. Most likely, though, from what I can see, Dells firmware merely tells the controller it's a certified drive... Which really means they blowing smoke and certifying they're overcharging like mad.
They've only put their optimized firmware on the drive to try and force people to spend thousands more than the same drive from any other source.
A Dell server drive was recognized in an HP workstation at the same time I had the Seagate and I about f'ked up...maybe. I checked the firmware on the Seagate and it wanted to update the Dell. I didn't let it get past it saying there was an update available. Dell has an update available and oddly enough, very close dates with Seagate being released prior to Dells.
Turning off the certified drives only, all drives work perfectly. I did get one defective Seagate drive, so I'm getting one cross shipped. I'll put it in as a hot spare, which this controller will spin down until it's needed.
I called Dell back, but this time specific support for this server. Dude was pretty cool. He had me request service which let him into the server. He checked iDrac and checked all settings and said he couldn't tell I had non Dell drives in and pointed out using non Dell drives ONLY affects getting support if I called for an arrays support. It does affect any warranties.
He kinda said the same thing I said, How many people call support for a failed RAID? No one, apparently. If the drive fails, they verify the server is in warranty and will ship me a new drive. Seagate, on the other hand, will require a credit card to cross ship and it can take a couple of days, so I think putting the replacement for the bad Seagate in as a hot spare makes better sense than waiting to only add 16TB. It's video, so hell, I lose a few weeks beyond our retention policy anyway.
The model numbers are the same with the exception there's an S at the end of the Dell drives. Dell:ST16000NM000JS. Seagate: ST16000NM000J
Hoosier_Farmer_@reddit
excellent follow-up - thank you so much for the information and clarification, I'll file it in my head and I'm sure anyone who later comes across your post will appreciate it too!
[fwiw - I hexedit compared the same date of release firmware seagate vs dell for this drive - dell has a 256b 'header' appended to where the actual firmware begins. the actual firmware is definitely quite different with very little correlation between the two, all the way towards the bottom where it becomes identical again when text strings start, aside from model numbers and sundry. Stripping the Dell header, the seagate utility still declines to accept the Dell as a valid flash - I stopped playing at that point, but suspect it would proceed to flash by either manipulating the model# strings at the bottom of the flash file, or debugging manipulating the compatability check in the flash program itself. I also found a 3rd party 'dolphin dfl-srp' hardware repair utility to force flash it via a hidden debug/serial that is exposed on the SATA interface. Posted as a point of curiosity, hopefully it helps someone in the future.]
MAndris90@reddit
hmm flashing the secure seagate firmware to a standard model :) if i had the tools to unbrick the drive if that fails. sed versions atleast 30% more expensive and hard to get.
Hoosier_Farmer_@reddit
guarantee that's what happened to whatever aliexpress drive you find at half cost. serial interface and label printer are dirt cheap, as is dumpster diver labor
MAndris90@reddit
unfortunatly this kind of scam selling ewaste as new product is not a capital offense
Aromatic-Coconut-122@reddit (OP)
Nice. Thanks for that info.
I did the same. I'm wondering if the top section was the verification information passed to the controller and/or is a built in md5 type verification method.
I'm just going to let it be. The Seagate drives perform the same as the Dell when connected to a standard SATA connection when in the HP workstation using Crystal Disk Mark.
Besides, I've grown tired of my company forcing me to do this stuff. Sure, we're not a billion dollar revenue company, but this is critical video. We've had proprietary video servers fail and replace them with these Dell servers and I got the previous two for $22k together. This one cost almost $30k. Spend the extra I saved on the others and buy the official drives. I hate to say that, but I've got one door out the door to go back to Law Enforcement. I miss it and basically need to go back or I'll be paying on student loans until I die.
The nice thing about getting rid of the old server is one is brand new and a few bios and firmware flashes and it's a standard Dell server with 12 x 12 TB hard drives and redundant OS NVMes. That'll be mighty nice Plex server 🤔
Hoosier_Farmer_@reddit
I sure hope your company is showing their appreciation - this kinda thinking is top 1% stuff. stay classy, friend.
Aromatic-Coconut-122@reddit (OP)
They do not. My boss does, beyond that, the company would go without security if they could. That should tell you where their mind is at. We’re the only department with edge servers. They’ve moved everything else to cloud hosted azure. We literally have tens of millions in servers going to recycling. And I mean Iron Mountain recycling not being sold to a rehab facility to eliminate the e-waste.
But, I get paid to do my job and the last line of my initial post is becoming more attractive.
But I appreciate the kind words
Aromatic-Coconut-122@reddit (OP)
Yes. I spent the day looking for both Dell drives, oddly available from countless server supply sites, but when checking them out, they're $300 with the sled, but clearly were Seagate labeled drives, so I'm sure I'll get Seagate will Dell firmware and still no support if I have a problem.
Dell sells sells with 20TB drives, but you can't buy them. Just like the second CPU kits.
MAndris90@reddit
would be intresting if you can succesfully flash a custom firmware to any of the seagate drives, as the encryption engine is firmware enabled on the "opal capable models" so you buy th echeaper drives and flash the encryption enabled version of the firmware, as the controller is the same
Aromatic-Coconut-122@reddit (OP)
That’s what I figured, but the more I go through this, how many times has any sys admin called support for a failed array? Probably never unless the RAID controller failed. If a dive fails, whether it’s the Seagate or the Dell labeled, they simply go back to their respective companies and I get a replacement, though I think Dell ships a new drive and Seagate ships a certified refurbished drive, which I don’t care. I’ve seen how they refurbish enterprise class drives, it’s literally good as new and only labeled refurbished to prevent others from getting screwed if I sold the drive as new.
I may order a spare and try to flash it, if I can find the firmware the Dells have.
MAndris90@reddit
i have in my 730xd 4 18tb dell branded drives with their firmware, bought from serverparts refurbished. the only thing i noticed with them apart from the original seagate drives with their opal firmwares. idrac immediatly found that the dell branded drives have a new firmware. but i didnt notice any other difference. ohh and the dell ones have no opal capabilities enabled, which is a shame, but the drives are too expensive to do trial and error with flashing tools to reflash them.
Aromatic-Coconut-122@reddit (OP)
The Seagate firmware recognized the Dells as Seagate and showed an update was available, clearly not the Dell firmware, but I wasn't chancing it.
I did notice iDrac detected new firmware, for 11 drives, but only 8 are Dell drives, so I'll update them with the Seagate removed.
I'm not concerned about OPAL, but for what Dell wants to charge for this, the 12TB are over $1200 from Dell. The 16TB drives aren't available. They should have every feature available. The brand new Seagates were $300 from Amazon, with $25 for the sleds (4 pack). Mine is all for video surveillance so I don't need encryption, but I can imagine for data centers it's a huge miss.
MAndris90@reddit
what i bought from serverparts is dell pn okpvdn its an st18000nm002j which is a custom part number. psid is listed on the drive but opal is not recognized, disabled in firmware.
big ass certified by dell emc enterprise class logo on the drive.
Aromatic-Coconut-122@reddit (OP)
Odd. On the 16TB drives, the only difference is the Seagate is ST000NM001J, which supports SED, while an ST000NM000J does not. Nothing in Seagates spec sheets specifies OPAL, but OPAL is a SED specification. The only thing I've found regarding OPAL refers to iDRAC and specific PERC controllers.
What a mess between drive manufacturers and their custom firmwares, not to mention no real clarification on whether the drive support OPAL or not. Yes the drives are self encrypting by default, but you'd think between Dell and Seagate somewhere it'd be documented as to which drives support which SED methods.
MAndris90@reddit
seagate has a spec sheet on the exos drives stating sata/sas versions and theor sed counterpart. but mine is not listed among them. i have the x18 18tb sed versions in the nas. thats detected wonderfully, but it has a different firmware designation too. but its a pain in the ass to get them and the hdd prices are up in the skies in the last years. wish i could get my hands on some of those 1xxtb e3s drives storagereviews guys got ith a full box.
Aromatic-Coconut-122@reddit (OP)
I went to Seagate’s EXOS18 sales page and for the 18TB the options were Standard (no SED), SED, and SED-FIP for the SAS drives and Standard or SED for SATA. Even in their datasheets, there’s no mention of any supported SED methods other than the 1307-bit encryption(? I’m bad with numbers and typing this on, well, the throne, lol)
Still, in one of the replies here, I and another user looked at both the Dell and Seagate firmware. The Dell had more data at the start of the file that wasn’t exactly understandable before the second part which aligned with the Seagate firmware. Of course modifying either file breaks its ability to flash it as I’m sure it fails the firmware integrity checks, but it makes no sense that the controller specifically was installed, by Dell, because it supported OPAL, yet the drives do not specify support anywhere and iDRAC only lets you manage the keys. Granted OPAL is SED, so is it only accessible as the default for the Dell drives and the PERC 755 controller? Another call to Dell and no one had a definite an answer and no engineer support was available. Again, I don’t really need the encryption SED or otherwise, but the drives encrypt by default. I’m going to cover this rabbit hole for now. I’m in a “I WANT to know, and MUST know” mode and the next thing I know I’ve lost 4 hours without knowing it.
MAndris90@reddit
secure firmwares start with ss0(x) if i remember. while the non sed models with sn0(x)
there are documents on seagates sites which go into detail on sed drives. if i remember right you can set the option which has finite number of retries on the password then it wipes the key from the disk "gimme the encryption key to check your files! your momma. acces denied, dont make me ask again, your pappa, date is gone :D"
i cant tell you about the controller as i have a hba330 in mine. but that should see the sed capability. but could solve few of my problems if the bios or the idrac could manage the keys and disk unlock for nvme or on the sas/sata drives on the 730xd
Aromatic-Coconut-122@reddit (OP)
I searched high and low for the drive you listed. I found nothing.
The controller might show me SED compatibility if I turn the Certified drives only option back on. The spec sheets for my drives just say SED is on by default and encrypting already, though I'd imagine if I reset the keys in iDRAC, it'd break the RAID and I'd have inaccessible data. Theres a statement on the information page about the EXOS X18 series of drive equiped with SED that essentially said, from the moment you power it on, it starts encrypting the drive, which tells be both the Dell firmware and Seagate both behave the same when plugged into a standard SATA 6 controller, right down to nearly the exact same speeds in all tests.
When I put those drives in the server ,they took longer to be initialized in the controllers firmware. They flashed like mad on activity while the server booted and continued for almost two minutes, then the server finished and I set up the array. I'm just guessing here, but if the encrypt from power on, they probably did when I formatted them in the workstation, and even blank, the server had to rectify the controller not having a key. At least I know the difference in The drives are minimal enough to not worry.
MAndris90@reddit
they encrypt on the fly without system intervention or cpu cycle. but its intresting you couldnt find any info on that drive.
https://www.ebay.com/itm/295940790612
is the ebay listing i bought them from
https://www.dell.com/support/home/hu-hu/drivers/driversdetails?driverid=rwjhj
pal9 firmware from dell's website :)
Aromatic-Coconut-122@reddit (OP)
I meant from Dell or other distributors. I skip over eBay.
Whats even weirder is the Dell site never appeared. Maybe because it's not in English or Google is filtering stuff out.
MAndris90@reddit
i dont think its region specific, but the part number exists in their website, maybe its restricted only for big players and datacenters. but spotted 26tb drives too but its only seagate with their mach2 2 head drives those would be fast. as mine can produce over 250mb/s continues on avg file sizes.
Aromatic-Coconut-122@reddit (OP)
Yeah, I don't see anything past the EXO 12TB without signing into my Dell account. I mean it's and enterprise account, but signed in, I see parts for all my servers, and going to this one, still only shows the 12TB. No options for CPU kits or larger drives, so I'm pretty sure I have to call sales to get the larger scale items.
This Milestone VMS uses a Master server, so our security and production can sign in with the AZURE Domain logins and only view cameras they are entitled to from one address. From the security standpoint, it an awesome system. I can set my backpack down in a lobby at one facility, have someone else grab it, and walk around with it at two facilities and we can select the point where I set it down and tell the VMS to locate it. I picks up everywhere it passed under a camera, exits the building, and where it's seen again at another location.
So I completely understand the cost of the drives, of Dell actually had somehow optimized the firmware to make them faster, but I don't see it. The 12TB drives are showing near identical specs as these 16TB, and there no difference between Seagate and Dell.
I wonder if I can order one of these servers without the RAID drives. I saw the invoice breakdown and the 8 Dell drives cost more than the server did. Mind blowing, and you can't help but feel kind of ripped off on the storage.
MAndris90@reddit
someone posted in the datahoarder that they got 22tb drives. 16pib and those looks like gen16 caddies
buzzsawcode@reddit
I’ve used non-Dell drives in an R450 before and never had issues - like you we couldn’t get the Dell versions to use. They will blame almost any issue you call about on those drives though - ask me how I know.
I would be curious if you can actually tell any difference performance wise with software RAID vs hardware RAID. I run a few hundred R760s and R450s with Linux that all process high amounts of streaming data on software RAID and haven’t seen a noticeable difference between hardware vs software.
Your experience with a Windows system may be wildly different though.
Aromatic-Coconut-122@reddit (OP)
I poked around in the RAID controller. There's an option to use any dive or Dell Optimized drives. Since it's new, I'll try the default Dell Optimized, than any drive and test speed. Then try software. I've used many software RAIDS and the only thing I missed out on is holding a hot spare. but won't need it for RAID 6