/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w27540002ac000ae2,0
8. c5t24510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w24510002ac000ae2,0
9. c5t26540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w26540002ac000ae2,0
These four devices listed above (6,7,8,9) refer to the same LUN on the 3PAR storage known as (LUN
0). This is determined by the 'd0' at the end of the device name. The problem here is that we are
looking at these different paths to the same device.
First path to LUN 0 is over the 255 interface (c3t25510002AC000AE2d0)
Second path to LUN 0 is over the 275 interface (c3t27540002AC000AE2d0)
Third path to LUN 0 is over the 245 interface (c5t24510002AC000AE2d0)
Fourth path to LUN 0 is over the 265 interface (c5t26540002AC000AE2d0)
As we see, we have 4 disk devices instead of one. We need to mount only one of these paths of the
LUN. The path that we need to choose is the path that is the 'owner' of the LUN.
For example, if the owner is the 255 interface, we should mount c3t25510002AC000AE2d0.
However, if the 255 interface has issues and we lose that path, we will lose access to the LUN.
We need a layer to abstract these 4 paths into a single device that we use. This layer would
present a new virtual/psuedo device that could be mounted. Underneath this virtual device would
be the two paths to the LUN, the owner and the non-owner. This resolves the issue of what device
to use if an interface to the LUN fails.
The multipathing layer would simply start accessing the device over the alternate path and would
not require any changes to what device the OS accesses the filesystem over.
In Solaris 10, MPXIO provides this layer of abstraction to solve this issue.
Solaris I/O multipathing gives you the ability to set up multiple redundant paths to a storage
system and gives you the benefits of load balancing and failover.
Enable MPxIO
Solaris 10 is the easier, because the mpxio capability is built-in. You just need to turn it on!
vi /kernel/drv/fp.conf
However, You can enable mulipathing for each device
First list all the FC devices
bash-3.00# ls -l /dev/fc
lrwxrwxrwx 1 root root 70 May 13 15:41 fp0 ->
././devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0:devctl
././devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0:devctl
././devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0:devctl
././devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0:devctl
or
bash-3.00# luxadm -e port
/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0:devctl NOT CONNECTED -> Port 0
/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0:devctl CONNECTED -> Port 1
/devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0:devctl CONNECTED -> Port 1
/devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0:devctl NOT CONNECTED -> Port 0
HBA is connected to port 1
We are interested in the path after '/devices' and before '/fp@0,0..'
In this case, the path we are interested in is: '/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0'.
This path is associated with port 0 of the fp driver (from the '0' in fp0).
name='fp' parent='/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0' port=1 mpxio-disable='no';
name='fp' parent='/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1' port=1 mpxio-disable='no';
name='fp' parent='/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0' port=1 mpxio-disable='no';
name='fp' parent='/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1' port=1 mpxio-disable='no';
This disables MPXIO globally and enables it for the above device.
name='fp' parent='parent name' port=port-number mpxio-disable='no/yes'
and port-number is the port number of the HBA
Step #2.
Edit /kernel/drv/scsi_vhci.conf
The scsi_vhci driver is responsible for hiding the owner and non-owner paths to the LUNs.
To enable this capability, the driver must know what type of device that the HBA is connected to.
To do this, you need to set the following in /kernel/drv/scsi_vhci.conf:
device-type-scsi-options-list='3PARdataVV', 'symmetric-option';
We knew to set the option to '3PARdata VV' from the output of format:
6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
Since 3PAR is symmetric luns which basically means active/active between all the paths the luns sees. We have to tweak the MPxIO to make it work.
In case of 3PAR /kernel/drv/scsi_vhci.conf doesn't like space between 3PARdata and VV :-)
device-type-scsi-options-list='3PARdata VV', 'symmetric-option';
make sure it is 3PARdataVV as below
device-type-scsi-options-list='3PARdataVV', 'symmetric-option';
select the lun
Vendor: 3PARdata
Revision: 0000
Product ID has 16 characters
and
then you can simply use stmsboot update and it will udpate the multipathing and vfstab
reboot -- -rv
run the stmsboot command and confirm the reboot:
another stmsboot commands:
To update = stmsboot -u
bash-3.00# stmsboot -e
WARNING: stmsboot operates on each supported multipath-capable controller
detected in a host. In your system, these controllers are
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0
If you do NOT wish to operate on these controllers, please quit stmsboot
and re-invoke with -D { fp | mpt | mpt_sas} to specify which controllers you wish
WARNING: This operation will require a reboot.
The changes will come into effect after rebooting the system.
updating /platform/sun4v/boot_archive
This will also udpate /kernel/drv/fp.conf with mpxio-disable='no';
bash-3.00# stmsboot -L
------------------------------------------------------------------
/dev/rdsk/c1t1d0 /dev/rdsk/c6t5000C500091B7D93d0
/dev/rdsk/c1t3d0 /dev/rdsk/c6t5000CCA00A33F218d0
/dev/rdsk/c1t5d0 /dev/rdsk/c6t5000CCA00A34E950d0
This is when we enable multipathing on all devices
To enable Solaris I/O Multipathing on multipath-capable Fibrechannel controller ports,
enter:
bash-3.00# stmsboot -D fp -e
Do you want to continue ? [y/n] (default: y)
The changes will come into effect after rebooting the system.
Before multipathing, you would see four copies of each disk in format. Afterwards, you'll just
see the one copy.
It assigns the next available controller ID, and makes up some horrendously long target number.
/dev/dsk/c6t600C0FF000000000086AB238B2AF0600d0s5 697942398 20825341 670137634 4% /test
At this point we see 4 devices ( 6-9 as listed above)
bash-3.00# format
AVAILABLE DISK SELECTIONS:
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
/pci@0/pci@0/pci@2/scsi@0/sd@1,0
/pci@0/pci@0/pci@2/scsi@0/sd@2,0
/pci@0/pci@0/pci@2/scsi@0/sd@3,0
/pci@0/pci@0/pci@2/scsi@0/sd@4,0
/pci@0/pci@0/pci@2/scsi@0/sd@5,0
6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w25510002ac000ae2,0
7. c3t27540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w27540002ac000ae2,0
8. c5t24510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w24510002ac000ae2,0
9. c5t26540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w26540002ac000ae2,0
bash-3.00# format
AVAILABLE DISK SELECTIONS:
/scsi_vhci/disk@g5000c500091b7d93
/scsi_vhci/disk@g5000c500091b90b3
/scsi_vhci/disk@g5000cca00a33f218
/scsi_vhci/disk@g5000cca00a34e950
/scsi_vhci/disk@g5000cca00a3468e0
/scsi_vhci/disk@g5000cca00a350350
6. c6t50002AC000220AE2d0 <3PARdata-VV-0000-2.00TB>
You will notice all the above devices changed to /scsi_vhci/XXXXX instead of /pci@0/pci@0/
Also compare your previous probe output to one below
bash-3.00# luxadm probe
Node WWN:2ff70002ac000ae2 Device Type:Disk device
Below stmsboot output shows you how all interfaces 245, 255, 265 and 275 has one pseudo name as
50002AC000220AE2
non-STMS device name STMS device name
------------------------------------------------------------------
/dev/rdsk/c1t1d0 /dev/rdsk/c6t5000C500091B7D93d0
/dev/rdsk/c1t3d0 /dev/rdsk/c6t5000CCA00A33F218d0
/dev/rdsk/c1t5d0 /dev/rdsk/c6t5000CCA00A34E950d0
/dev/rdsk/c5t26540002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c5t24510002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c3t27540002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c3t25510002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
Step #4.
using mpathadm commands
bash-3.00# mpathadm list initiator-port
Initiator Port: ,4000002a00ff
Initiator Port: 10000000c97d2c70
Initiator Port: 10000000c97bfb4d -> connected HBA
Initiator Port: 10000000c97d2c71 -> connected HBA
List all LUNs
/dev/rdsk/c6t5000C500091B90B3d0s2
Mpathadm Solaris 10 Vm
Operational Path Count: 1
Total Path Count: 1
/dev/rdsk/c6t5000CCA00A350350d0s2
Operational Path Count: 1
Total Path Count: 1
/dev/rdsk/c6t5000CCA00A3468E0d0s2
Operational Path Count: 1
Total Path Count: 1
/dev/rdsk/c6t50002AC000220AE2d0s2<--- pesudo LUN we are interested in
Operational Path Count: 4
This shows how this pesudo device using the different path underneath
bash-3.00# mpathadm show lu /dev/rdsk/c6t50002AC000220AE2d0s2
mpath-support: libmpscsi_vhci.so
Product: VV
Name Type: unknown type
Asymmetric: no
Logical Unit Group ID: NA
Auto Probing: NA
Paths:
Target Port Name: 26540002ac000ae2
Path State: OK
Target Port Name: 24510002ac000ae2
Path State: OK
Target Port Name: 27540002ac000ae2
Path State: OK
Target Port Name: 25510002ac000ae2
Path State: OK
Name: 26540002ac000ae2
Relative ID: 0
Name: 27540002ac000ae2
Relative ID: 0
Solaris 10 11/06 was released in late 2006 with a plethora of new features, and among them a new tool called mpathadm, which comes as part of the SUNWmpathadm package.
Before I delve into how this specific tool works and how it helps when managing multipathed storage in Solaris, I'll give some background on what multipathing is and how it is implemented in Solaris.
What is multipathing? Whitechapel this is exile album download.
Multipathing is a generic term relating to a certain SAN configuration where a host on that SAN (ie; a server) has multiple physical paths to a target (ie; a disk array) on said SAN. A common multipath configuration consists of a server with multiple fibre channel ports, and each port is connected to a separate fibre channel switch in the SAN. Optionally, the target device or disk array may also have multiple physical links in the same manner. In the end, though, the host 'sees' multiple instances of the same target – once through each connection to the SAN. It's up to the host to then manage this. The end result is higher reliability. If a switch were to go down or the optical cable between the host and a switch were damaged, the host's connectivity to its SAN disks would not be interrupted. IO traffic is also multiplexed over the multiple links, thus providing fail-over and higher throughput.
How multipathing works in Solaris
Multipathing has existed in Solaris since Solaris 8 via an add-on product called Sun SAN Foundation Kit. With the release of Solaris 10, all functionality of the SAN Foundation Kit was integrated into the OS and no longer requires an add-on to function.
When multipathing is enabled, the scsi_vhci driver looks for multiple instances of the same target (eg; a LUN) and upon finding one, creates a 'meta' disk device which it manages. IO to and from this device are then multiplexed over however many paths scsi_vhci sees to the LUN.
Below is an example of a host which has two fibre channel controllers (c4 and c5) connected to the sam SAN, and both see the same target LUNs, of which there are four:
<tbody><tr><td>c4</td><td>6000393000015463</td><td>0</td></tr><tr><td>fibre channel controller</td><td>WWN of target</td><td>LUN number on the target</td></tr></tbody><p>So, the aforementioned <strong>scsi_vhci</strong> driver sees that LUN <em>6000393000015463,0</em> is presented by two controllers. It will then make a new disk device using that device's WWN, LUN, and other information to construct a globally-unique ID (GUID) device name and then create device entries under <strong><em>/dev/dsk</em></strong> and <strong><em>/dev/rdsk</em></strong> to represent a multipathed view:</p><p><a href='https://site-3327722-7819-4358.mystrikingly.com/blog/what-does-it-mean-when-a-microwave-says-se'><img src='https://s4.sywcdn.net/getImage?url=https:%2F%2Fimages.homedepot-static.com%2FproductImages%2F3be8ce9e-43df-4f5a-95e3-ad9b0ec47ec7%2Fsvn%2Fstainless-steel-amana-over-the-range-microwaves-amv2307pfs-64_1000.jpg&t=Product&w=420&h=0&qlt=100&mrg=1&mh=240&mxh=330&mhmcw=0&s=031d7e0272a7eccb503cfe1afbb216f9' alt='When'></a></p><div><textarea wrap='soft' readonly='>[root@ackpthh]/>ls -l /dev/rdsk/c6t600039300001546301000000D52ACC7Ed0* lrwxrwxrwx 1 root root 69 Jul 6 2006 /dev/rdsk/c6t600039300001546301000000D52ACC7Ed0 -> ././devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:wd,raw lrwxrwxrwx 1 root root 68 Jul 6 2006 /dev/rdsk/c6t600039300001546301000000D52ACC7Ed0s0 -> ././devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:a,raw lrwxrwxrwx 1 root root 68 Jul 6 2006 /dev/rdsk/c6t600039300001546301000000D52ACC7Ed0s1 -> ././devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:b,raw ..
| [root@ackpthh]/>ls-l/dev/rdsk/c6t600039300001546301000000D52ACC7Ed0* lrwxrwxrwx1root root69Jul62006/dev/rdsk/c6t600039300001546301000000D52ACC7Ed0->../../devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:wd,raw lrwxrwxrwx1root root68Jul62006/dev/rdsk/c6t600039300001546301000000D52ACC7Ed0s0->../../devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:a,raw lrwxrwxrwx1root root68Jul62006/dev/rdsk/c6t600039300001546301000000D52ACC7Ed0s1->../../devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:b,raw |
You might notice the new controller, c6. This is a 'virtual' controller implemented by scsi_vhci since making such virtual disk devices under a real physical controller would be a broken idea since, technically, there is no such physical device. Remember, this is a meta device.
A note about those big, long multipath device names: there's actually a format to them, so don't let their length turn you off. The format is as follows, using the above device as an example:
We can now use the multipath device as we would any other kind of disk device, with format for example:
| c6 | t600039300001546301000000D52ACC7E | d0s1 |
| scsi_vhci controller | Target comprised of real target's WWN
, LUN number, and SCSI volume UID
(Page 83 data) | LUN number and then
slice number |
