diff options
| -rw-r--r-- | docs/persistent_naming/Testing_scsi_notes.txt | 213 | ||||
| -rw-r--r-- | docs/persistent_naming/testing_scsi_notes.txt | 203 |
2 files changed, 203 insertions, 213 deletions
diff --git a/docs/persistent_naming/Testing_scsi_notes.txt b/docs/persistent_naming/Testing_scsi_notes.txt deleted file mode 100644 index 5013c87e1d..0000000000 --- a/docs/persistent_naming/Testing_scsi_notes.txt +++ /dev/null @@ -1,213 +0,0 @@ -Using UDEV to do Persistent storage device naming -for large numbers of storage devices -3/16/2004 - -Here are some lessons we learned at OSDL recently on how to use -UDEV (version 021) to do persistent device naming for lots of storage -devices. We used what was available in udev for scsi devices. Here is -an outline of this report: - -Background information - a list of resources we needed to get -started. -Setup - what we needed to create the right enviroment (kernel, -patches, drivers) -How udev works to assign persistent storage device names - -what the documentation didn't tell us. -Performance - A sanity test we ran to compare with and without -persistent naming. - - -BACKGROUND INFORMATION -To get started, here are some references. Review the overview -articles so that the rest of the information makes sense. - -Download the latest udev stuff from: -http://www.kernel.org/pub/linux/utils/kernel/hotplug/ - -mailing list: -linux-hotplug-devel@lists.sourceforge.net - -Here is a nice overview article to get started (warning, this is from -summer 2003 so many items indicated as "todo" have been done and -configuration file name references have sometime changed): -http://www.kroah.com/linux/talks/ols_2003_udev_paper/Reprint-Kroah-Hartman-OLS2003.pdf -(also included when you download udev) - -More general info (also included in the udev package): -http://kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ -UDEV version 021 Announcement: -http://marc.theaimsgroup.com/?l=linux-hotplug-devel&m=107827264803336&w=2 - -"Managing Dynamic Naming" -http://lwn.net/Articles/28897/ - -If you are a fan of devfs, whatever you do, don't complain until you -read everything you possibly can about udev. This for example: -http://kernel.org/pub/linux/utils/kernel/hotplug/udev_vs_devfs - -You will need to create udev.rules to supply consistent names. (See -etc/udev/udev.rules in the download). This article gives you some -background about udev.rules, but avoids describing the "PROGRAM" -key which is needed for our work. Read it for background: -writing udev rules (current as of udev 018) -http://www.reactivated.net/udevrules.php - -bitkeeper tree: -bk://kernel.bkbits.net/gregkh/udev - -Libsysfs (used to get sysfs information): -http://www-124.ibm.com/linux/papers/libsysfs/libsysfs-linuxconfau2004.pdf - -UDEV works using the way hotplug events are handled by the kernel. -Several overview articles about hotplug include: -Hotplug events -http://lwn.net/Articles/52621/ -Overview of Hotplug -http://linux-hotplug.sourceforge.net/ - -Gentoo centric install info: -http://webpages.charter.net/decibelshelp/LinuxHelp_UDEVPrimer.html - -rpms built against Red Hat FC2-test1 may be available at: -http://kernel.org/pub/linux/utils/kernel/hotplug/udev-021-1.i386.rpm - -with the source rpm at: -http://kernel.org/pub/linux/utils/kernel/hotplug/udev-021-1.src.rpm - - - -SETUP - -Here is a brief checklist of what you need on your system for this to -work: - -Kernel must be a 2.6 kernel - -Must use CONFIG_HOTPLUG kernel config option, since the solution -is based on hotplug capabilities. - -To test more than 256 scsi devices you need a patch to the scsi driver -to support that many (available from IBM or SuSE). To see the patch -we used, see this link: -http://developer.osdl.org/maryedie/DCL/PSDN/lotsofdisks.patch - -Your storage device must support (via the driver) a unique identifier for -persistent device naming. (Adaptec RAID device does not, for -example.) - -Your device driver must support sysfs (new in 2.6 kernel). This is -already done for scsi devices and most if not all block devices. - -A program (scsi_id) exists in the udev download -( extras/scsi_id/scsi_id.c) for scsi devices. It can read the identifier and -is needed for persistent naming. - - -HOW UDEV WORKS TO ASSIGN PERSISTENT NAMES: - -There are three places where device information is stored that udev -uses: -(1) /sys maintained by sysfs -(2) /etc/udev/udev.rules - where you can store the identifier to NAME -mapping information. -(3) The tdb (udev-021/tdb/tdb.c), trivial data base, that is held in -memory and holds the valid system configuration. It is not saved -between one boot to the next. It is constructed at boot time and -updated with configuration changes. - -The persistent names are kept (at least this is one way to do it) in -udev.rules (uuid and NAME), one entry per device. If you want to -initially give your 1000 disk devices a default name and then make -sure those names are preserved, here is how : - -Start with no special entry in udev.rules when do you an initial boot of -your system with disks in place. Udev will assign default names (there -are ways to control what you want for default too). - -Once the names are assigned, use a script supplied for scsi devices - -udev-021/extras/scsi_id/gen_scsi_id_udev_rules.sh -to generate the lines needed for udev.rules, one per device. Each line -indicates the identifier and the NAME it was assigned. You could -optionally create this manually if you prefer other names . - -[example entries in udev.rules for scsi disks] -BUS="scsi", PROGRAM="scsi_id", RESULT="<uuid1>",NAME="<name1>" -BUS="scsi", RESULT="<uuid2>",NAME="<name2>" -... -BUS="scsi", RESULT="<uuid1000>",NAME="<name1000>" - -(The actual file we used is the file udev.rules_1000_scsi_debug in this -directory ) - -Upon reboot, for each device a hotplug event occurs. The udev.rules -file is scanned looking for the device type (BUS) in this case for "scsi". -The first entry generated by the above program references a -PROGRAM in the key field (scsi_id) which is called to probe the device -and determine the unique identifier. sysfs is used to determine the -major/minor number for the device. The result of the program -execution (the uuid) is compared with the RESULT entry in the same -udev.rules line. - --If it matches, then the NAME entered on this line is used. The uuid -and major/minor number is saved in tdb (newly recreated upon boot). -That device is created in /udev (the target directory name is -configurable) with the assigned NAME. - --If it doesn't match, the RESULT (uuid) is preserved for use on the next -udev.rules line as long as the bus type (scsi) is the same. So the result -(the uuid) is compared on the next line, and the next until a match -occurs. - --If no match occurs, the device will be assigned a default name. - --Tdb is updated with the resulting name assignment. - - -Thus if the uuid and names are enumerated, they will be found, -assigned, and are therefore permanent. - -If the device is removed from a live system, a hotplug event occurs, -and it is removed from tdb and the /udev entry disappears. - -If it is re-inserted at a new location, the udev.rules file is scanned as -above. The new major/minor number goes in tdb with the uuid , the -name in udev.rules is found again, and the /udev name re-appears. - - - -PERFORMANCE - -Now the question becomes, how much longer does it take to scan the -udev.rules table once there are 1000 entries? - -To test this, we created 1000 "scsi " devices using the scsi debug -device driver supplied in the kernel. When this device driver is loaded -you can specify how many fake scsi devices to create. There is no -real I/O involved but it does respond to some scsi commands. It -simulates the uuid by using the device number assigned when the -device is created. - -Then we auto-generated entries into udev.rules with -gen_scsi_id_udev_rules.sh. We then removed the devices and -reassigned them to simulate a reboot. The delta between assigning -defaults and assigning the names enumerated in the udev.rules file -was 7 seconds (that's for 1000 drives). - -Scripts utilized the feature (described above) that saves the "RESULT" -key after one scsi-id program call for later reference with other -udev.rules entries (so only have one PROGRAM key is the moral of -the story). If you repeated the PROGRAM key, you would -unnecessarily call the program up to 999 times! - -The script that creates udev.rules did not work for 1000 drives (the -input line is too long). We determined that a patch for this already -existed but had not yet been checked in. - - - - - - - - - diff --git a/docs/persistent_naming/testing_scsi_notes.txt b/docs/persistent_naming/testing_scsi_notes.txt new file mode 100644 index 0000000000..8d94c842b4 --- /dev/null +++ b/docs/persistent_naming/testing_scsi_notes.txt @@ -0,0 +1,203 @@ +Using UDEV to do Persistent storage device naming +for large numbers of storage devices +3/16/2004 + +Here are some lessons we learned at OSDL recently on how to use UDEV +(version 021) to do persistent device naming for lots of storage devices. +We used what was available in udev for scsi devices. Here is an outline of +this report: + +Background information + a list of resources we needed to get started. +Setup + what we needed to create the right enviroment (kernel, patches, + drivers) +How udev works to assign persistent storage device names + what the documentation didn't tell us. +Performance + A sanity test we ran to compare with and without persistent naming. + + +BACKGROUND INFORMATION +To get started, here are some references. Review the overview articles so +that the rest of the information makes sense. + +Download the latest udev stuff from: + http://www.kernel.org/pub/linux/utils/kernel/hotplug/ + +mailing list: + linux-hotplug-devel@lists.sourceforge.net + +Here is a nice overview article to get started (warning, this is from +summer 2003 so many items indicated as "todo" have been done and +configuration file name references have sometime changed): +http://www.kroah.com/linux/talks/ols_2003_udev_paper/Reprint-Kroah-Hartman-OLS2003.pdf +(also included when you download udev) + +More general info (also included in the udev package): + http://kernel.org/pub/linux/utils/kernel/hotplug/udev-FAQ +UDEV version 021 Announcement: + http://marc.theaimsgroup.com/?l=linux-hotplug-devel&m=107827264803336&w=2 + +"Managing Dynamic Naming": + http://lwn.net/Articles/28897/ + +If you are a fan of devfs, whatever you do, don't complain until you read +everything you possibly can about udev. This for example: + http://kernel.org/pub/linux/utils/kernel/hotplug/udev_vs_devfs + +You will need to create udev.rules to supply consistent names. (See +etc/udev/udev.rules in the download). This article gives you some +background about udev.rules, but avoids describing the "PROGRAM" key which +is needed for our work. Read it for background: writing udev rules +(current as of udev 018) + http://www.reactivated.net/udevrules.php + +bitkeeper tree: + bk://kernel.bkbits.net/gregkh/udev + +Libsysfs used to get sysfs information): + http://www-124.ibm.com/linux/papers/libsysfs/libsysfs-linuxconfau2004.pdf + +UDEV works using the way hotplug events are handled by the kernel. +Several overview articles about hotplug include: +Hotplug events +http://lwn.net/Articles/52621/ +Overview of Hotplug +http://linux-hotplug.sourceforge.net/ + +Gentoo centric install info: +http://webpages.charter.net/decibelshelp/LinuxHelp_UDEVPrimer.html + +rpms built against Red Hat FC2-test1 may be available at: +http://kernel.org/pub/linux/utils/kernel/hotplug/udev-021-1.i386.rpm + +with the source rpm at: +http://kernel.org/pub/linux/utils/kernel/hotplug/udev-021-1.src.rpm + + + +SETUP + +Here is a brief checklist of what you need on your system for this to +work: + +Kernel must be a 2.6 kernel + +Must use CONFIG_HOTPLUG kernel config option, since the solution is based +on hotplug capabilities. + +To test more than 256 scsi devices you need a patch to the scsi driver to +support that many (available from IBM or SuSE). To see the patch we used, +see this link: +http://developer.osdl.org/maryedie/DCL/PSDN/lotsofdisks.patch + +Your storage device must support (via the driver) a unique identifier for +persistent device naming. (Adaptec RAID device does not, for example.) + +Your device driver must support sysfs (new in 2.6 kernel). This is already +done for scsi devices and most if not all block devices. + +A program (scsi_id) exists in the udev download (extras/scsi_id/scsi_id.c) +for scsi devices. It can read the identifier and is needed for persistent +naming. + + +HOW UDEV WORKS TO ASSIGN PERSISTENT NAMES: + +There are three places where device information is stored that udev +uses: +(1) /sys maintained by sysfs +(2) /etc/udev/udev.rules - where you can store the identifier to NAME + mapping information. +(3) The tdb (udev-021/tdb/tdb.c), trivial data base, that is held in + memory and holds the valid system configuration. It is not saved + between one boot to the next. It is constructed at boot time and + updated with configuration changes. + +The persistent names are kept (at least this is one way to do it) in +udev.rules (uuid and NAME), one entry per device. If you want to initially +give your 1000 disk devices a default name and then make sure those names +are preserved, here is how : + +Start with no special entry in udev.rules when do you an initial boot of +your system with disks in place. Udev will assign default names (there +are ways to control what you want for default too). + +Once the names are assigned, use a script supplied for scsi devices - +udev-021/extras/scsi_id/gen_scsi_id_udev_rules.sh to generate the lines +needed for udev.rules, one per device. Each line indicates the identifier +and the NAME it was assigned. You could optionally create this manually if +you prefer other names . + +[example entries in udev.rules for scsi disks] +BUS="scsi", PROGRAM="scsi_id", RESULT="<uuid1>",NAME="<name1>" +BUS="scsi", RESULT="<uuid2>",NAME="<name2>" +... +BUS="scsi", RESULT="<uuid1000>",NAME="<name1000>" + +(The actual file we used is the file udev.rules_1000_scsi_debug in this +directory ) + +Upon reboot, for each device a hotplug event occurs. The udev.rules file +is scanned looking for the device type (BUS) in this case for "scsi". The +first entry generated by the above program references a PROGRAM in the key +field (scsi_id) which is called to probe the device and determine the +unique identifier. sysfs is used to determine the major/minor number for +the device. The result of the program execution (the uuid) is compared +with the RESULT entry in the same udev.rules line. + +- If it matches, then the NAME entered on this line is used. The uuid and + major/minor number is saved in tdb (newly recreated upon boot). That + device is created in /udev (the target directory name is configurable) + with the assigned NAME. + +- If it doesn't match, the RESULT (uuid) is preserved for use on the next + udev.rules line as long as the bus type (scsi) is the same. So the + result (the uuid) is compared on the next line, and the next until a + match occurs. + +- If no match occurs, the device will be assigned a default name. + +- Tdb is updated with the resulting name assignment. + + +Thus if the uuid and names are enumerated, they will be found, assigned, +and are therefore permanent. + +If the device is removed from a live system, a hotplug event occurs, and it +is removed from tdb and the /udev entry disappears. + +If it is re-inserted at a new location, the udev.rules file is scanned as +above. The new major/minor number goes in tdb with the uuid , the name in +udev.rules is found again, and the /udev name re-appears. + + + +PERFORMANCE + +Now the question becomes, how much longer does it take to scan the +udev.rules table once there are 1000 entries? + +To test this, we created 1000 "scsi " devices using the scsi debug device +driver supplied in the kernel. When this device driver is loaded you can +specify how many fake scsi devices to create. There is no real I/O +involved but it does respond to some scsi commands. It simulates the uuid +by using the device number assigned when the device is created. + +Then we auto-generated entries into udev.rules with +gen_scsi_id_udev_rules.sh. We then removed the devices and reassigned them +to simulate a reboot. The delta between assigning defaults and assigning +the names enumerated in the udev.rules file was 7 seconds (that's for 1000 +drives). + +Scripts utilized the feature (described above) that saves the "RESULT" key +after one scsi-id program call for later reference with other udev.rules +entries (so only have one PROGRAM key is the moral of the story). If you +repeated the PROGRAM key, you would unnecessarily call the program up to +999 times! + +The script that creates udev.rules did not work for 1000 drives (the input +line is too long). We determined that a patch for this already existed but +had not yet been checked in. + |