Need to set a static SCSI device node?

Posted by Charm-Heap@reddit | linuxadmin | View on Reddit | 8 comments

Hey all -

We've got our backup server connected to our SAS tape library. Everything works well, however, occasionally, when we have a power issue (long power outage or system crash) and the system goes down, sometimes the tape drive (inside the tape library) moves from /dev/sg3 to /dev/sg2. I have no idea why, or what the rhyme or reason is, but it doesn't seem to affect anything, and it just switches places with an unused fiber channel port on our fiber channel storage array (our volumes from this array are mounted via WWN in a multipathing configuration - so they're unaffected by any of these moves).

I need to configure this to be static, such that it comes back up in the same place every time. I think I can set it to some static name, but I haven't found anything of much use online - and what I have found (using the /lib/udev/scsi_id command) gives me errors that have blocked me.

Has anyone been able to do this yet?

[-]

dodexahedron@reddit

You could set up a udev rule to match it and give it whichever name you want, so they're stable and more obvious. You could even make it create them as /dev/tapelib/lunX/driveX if you wanted to go that far.

Or, rather than using the sgX nodes, could you use /dev/disk/by-path to get to it, perhaps? Those should be more stable nodes, so long as the tape library presents them to the host in a stable fashion.

What does lsscsi -dgi show? Anything in there is fair game for use in a udev rule (other than the /dev/s* nodes of course - I just included that in the command to make it easy to find it).

The /dev/s* nodes get populated and assigned sequential names in the order the system discovers them, so they're not stable if that order can change for any reason, such as spin-up delays, slow response from a controller or bus, boot device order changes in BIOS, driver changes, and more.

[-]

Charm-Heap@reddit (OP)

What does lsscsi -dgi show? Anything in there is fair game for use in a udev rule (other than the /dev/s* nodes of course - I just included that in the command to make it easy to find it).

I get:

# lsscsi -dgi
[0:2:0:0]    disk    DELL     PERC H710        3.13  /dev/sda [8:0]  36c81f660e1b0c50022b0781b03693da4  /dev/sg0 [21:0]
[1:0:0:0]    tape    IBM      ULT3580-HH6      J451  /dev/st0 [9:0]  -  /dev/sg1 [21:1]
[1:0:0:1]    mediumx IBM      3573-TL          F.11  /dev/sch0[86:0]  -  /dev/sg2 [21:2]
[6:0:0:0]    cd/dvd  TSSTcorp DVD-ROM TS-U333B D150  /dev/sr0 [11:0]  -  /dev/sg11[21:11]
[9:0:0:0]    disk    DELL     MD36xxf          0820  -          -  /dev/sg3 [21:3]
[9:0:0:12]   disk    DELL     MD36xxf          0820  /dev/sdb [8:16]  36f01faf000cf7ade0000022164104a0d  /dev/sg4 [21:4]
[9:0:0:13]   disk    DELL     MD36xxf          0820  /dev/sdc [8:32]  36d4ae52000809418000025236411cbfd  /dev/sg5 [21:5]
[9:0:0:31]   disk    DELL     Universal Xport  0820  -          -  /dev/sg6 [21:6]
[9:0:1:0]    disk    DELL     MD36xxf          0820  -          -  /dev/sg7 [21:7]
[9:0:1:12]   disk    DELL     MD36xxf          0820  /dev/sdd [8:48]  36f01faf000cf7ade0000022164104a0d  /dev/sg8 [21:8]
[9:0:1:13]   disk    DELL     MD36xxf          0820  /dev/sde [8:64]  36d4ae52000809418000025236411cbfd  /dev/sg9 [21:9]
[9:0:1:31]   disk    DELL     Universal Xport  0820  -          -  /dev/sg10[21:10]

The /dev/s* nodes get populated and assigned sequential names in the order the system discovers them, so they're not stable if that order can change for any reason, such as spin-up delays, slow response from a controller or bus, boot device order changes in BIOS, driver changes, and more.

That is what I was figuring, and when things are coming back following a LONG power outage, I figure that can be hit or miss. I DO specifically need a SCSI target, though - like I can't point to /dev/st0, because that's the block device node, I need to point, correctly, to the /dev/sg1 SCSI device node.

[-]

dodexahedron@reddit

EDIT: it's the fuckin' DVD-ROM drive, not for nothing lol that thing goes back and forth

Do you even still need that? Disabling or removing it might make your life easier. 😅

That device will be primarily dependent on boot order, which is a moving target if it is below USB, network, and other devices, or if entries are added/removed in the actual firmware boot order by your boot loader.

If you need the optical drive, you can make a udev rule for it, or move it to always be first, so it doesn't change.

Though putting it first will shuffle the auto-ids for the rest of the devices up a tick, which may be problematic if your boot isn't done by uuid.

Also, just as a caution, boot order for a one-off boot is potentially different, as well. If you boot from a USB drive or something, that drive is sda, and everything else is shuffled up by at least one, as if it had been configured as the first boot device in the BIOS.

[-]

dodexahedron@reddit

Man if I were at a PC right now I could help you a lot more. But you're on the right track if you're investigating udev. It is rather unforgiving of even slightly imperfect configuration, though, for sure. 😅

But also, you specifically need to access it via the generic scsi device? I'm assuming that's for some application that needs to send it commands the st driver doesn't understand? While using the /dev/st nodes may not be what you need, there should be several other ways to get to each device, since they're all just essentially links back to something in /sys and, ultimately, /proc. udev just basically automates running mknod using the major and minor device numbers of whatever matches a rule. That lsscsi command I mentioned includes those numbers in its output.

If the actual physical location of the target device doesn't change and is presented by the tape robot consistently, the device will be in a consistent place in sysfs, which you can either use directly or have a device node created for it in /dev, via udev or some other mechanism.

If the major number for the device is different in the cases when it's one name vs the other, that indicates a race condition between more than one driver wanting to claim the device, which most likely means a module needs to be blacklisted.

If the major stays consistent but the minor changes, you'll have no choice but to use some other non-changing identifier, because the minor number comes from the driver that claimed it (likely either st or sg in your case - I don't remember the specific tape major number).

[-]

Charm-Heap@reddit (OP)

But also, you specifically need to access it via the generic scsi device? I'm assuming that's for some application that needs to send it commands the st driver doesn't understand?

Correct.

So far, my udev efforts have not found success - and I need to access the tape DRIVE via standard... block commands? But the tape LIBRARY I need to access via SCSI.

I've got this:

SUBSYSTEM=="scsi", ATTRS{sas_address}=="0x5000e1111214b002", SYMLINK="tape/tl2000/library"

but it is not working as of yet. :(

[-]

mysterytoy2@reddit

I think you need delay the spin-up time when power resumes. It may be a jumper setting.

[-]

Charm-Heap@reddit (OP)

We're using a Dell PowerVault TL2000 with an LTO-6 half-height drive - there's a jumper in there somewhere?

[-]

mysterytoy2@reddit

It's possible that there is a setting in the SCSI controller. You have to connect a monitor to the server and reboot it. There probably is a key you can press when the server is booting to get into the scsi controller bios. It might be there. See if you can delay the drive boot up time.