EMC – Working with PowerPath

EMC PowerPath is a software designed for load balancing, path management, and device management. For each storage device assigned, it creates a virtual emc powerpath device named as emcpower### where ### is a number. Quite a powerful utility. Most of the magic is in the binary /etc/powermt.

You can all read the powerpath guide to learn internals of powerpath, and know how it does the magic. But if you want a quick overview how to use different options commands to get the things done, read on.

Display an overview of devices/hbas connectivity

# /etc/powermt display
Symmetrix logical device count=64
CLARiiON logical device count=0
Hitachi logical device count=1
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path Summary Total Dead IO/Sec Q-IOs Errors
==============================================================================
3072 pci@7c0/pci@0/pci@8/fibre-channel@0/fp@0,0 optimal 48 0 - 0 5
3073 pci@7c0/pci@0/pci@9/fibre-channel@0/fp@0,0 optimal 55 0 - 0 4

Above output shows 64 EMC and 1 HDS devices are assigned. All paths are optimal. Different number of devices have been masked across different HBAs.

Display the options set for powermt for different arrays

# /etc/powermt display options

Default storage system class: all
Show CLARiiON LUN names: true
Path Latency Monitor: Off
Path Latency Threshold: 0 Sec.
Storage
System Class Attributes
------------ ----------
Symmetrix periodic autorestore = on
status = managed
CLARiiON periodic autorestore = on
status = managed
Hitachi periodic autorestore = on
status = managed
Invista periodic autorestore = on
status = managed
HP xp periodic autorestore = on
status = managed
Ess periodic autorestore = on
status = managed
HP HSx periodic autorestore = on
status = managed

Output shows that the devices from all the arrays are managed by powerpath.

Display how many devices are assigned from each array and which arrays the server can see

# /etc/powermt display paths
Symmetrix logical device count=64
==============================================================================
----- Host Bus Adapters --------- ------ Storage System ----- - I/O Paths -
### HW Path ID Interface Total Dead
==============================================================================
3072 pci@7c0/pci@0/pci@8/lpfc@0/fp@0,0 000290101234 FA 3aA 6 0
3072 pci@7c0/pci@0/pci@8/lpfc@0/fp@0,0 000292605666 FA 5eA 23 0
3073 pci@7c0/pci@0/pci@9/lpfc@0/fp@0,0 000290101234 FA 14dB 7 0
3073 pci@7c0/pci@0/pci@9/lpfc@0/fp@0,0 000292605666 FA 12eA 23 0

Hitachi logical device count=1
==============================================================================
----- Host Bus Adapters --------- ------ Storage System ----- - I/O Paths -
### HW Path ID Interface Total Dead
==============================================================================
3072 pci@7c0/pci@0/pci@8/lpfc@0/fp@0,0 88888 1H 1 0
3073 pci@7c0/pci@0/pci@9/lpfc@0/fp@0,0 88888 2H 1 0

To check the status of ports on HBAs, IOs queued up on each port etc

# /etc/powermt display port_mode

Storage class = Symmetrix
==============================================================================
----------- Storage System --------------- -- I/O Paths -- --- Stats ---
ID Interface Wt_Q Total Dead Q-IOs Mode
==============================================================================
000290101234 FA 14dB 256 7 0 0 Enabled
000290101234 FA 3aA 256 6 0 0 Enabled
000292605666 FA 12eA 256 23 0 0 Enabled
000292605666 FA 5eA 256 23 0 0 Enabled

Storage class = Hitachi
==============================================================================
----------- Storage System --------------- -- I/O Paths -- --- Stats ---
ID Interface Wt_Q Total Dead Q-IOs Mode
==============================================================================
28972 2H 256 1 0 0 Enabled
28972 1H 256 1 0 0 Enabled

Display the status of ALL devices

# /etc/powermt display dev=all

Display the status of a single device

# /etc/powermt display dev=emcpower72a

Check the status of the HBAs

# /etc/powermt display hba_mode

Check the powerpath registration keys

# /etc/powermt check_registration

Display the powerpath version

# /etc/powermt version

To perform correction on I/O paths

(addition of new paths, detecting and deleting the dead paths etc)
# /etc/powermt check (force – to clean up without any prompt)

To change the paths from active to standby in order to remove them

# /etc/powermt set mode=standby dev=[path|device|all]

To change all the paths from active to standby through HBA 3072

# /etc/powermt set mode=standby hba=3072
(3072 is the value in first column of powermt display paths)

To remove a single path from a device

# /etc/powermt display dev=all
Pseudo name=emcpower58a
Hitachi ID=28972
Logical device ID=HITACHI R500712C2E60
state=alive; policy=ADaptive; priority=0; queued-IOs=0;
==============================================================================
--------------- Host --------------- - Stor - -- I/O Path -- -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
3073 pci@7c0/pci@0/pci@9/lpfc@0/fp@0,0 c2t50060E8005712C17d1s0 2H active alive 0 0
3072 pci@7c0/pci@0/pci@8/lpfc@0/fp@0,0 c4t50060E8005712C07d1s0 1H active alive 0 0

# /etc/powermt remove dev=c4t50060E8005712C07d1s0 -> This will remove the path to 1H (second line)
# /etc/powermt remove dev=emcpower58a -> this will remove both the paths to this device

How to configure the newly assigned devices to the server?

# /etc/powermt config
This will configure all detected logical devices, their paths in powerpath configuratioin. It doesn’t remove previously configured paths when they become dead.

How to fine tune the options for a device?

# powermt set mode (to set active/standby)
# powermt set path_latency_monitor (to turn on/off monitoring of latency on a device – time it took to complete the IO – applies to ALL the devices)
# powermt set path_latency_threshold (to set the latency threshold for IO completion. If the value is set to 1 second, and if it takes more than that, it will be recorded in log file – APPLICABLE to all the devices)
# powermt set policy (sets load balancing and failover policies)
# powermt set priority (sets the priority of the IOs to devices)
# powermt set periodic_autorestore (check the dead paths and restore them if they are online again)
# powermt set write_throttle (enable or disable the write throttling to the specified logical devices)
# powermt set write_throttle_queue (set the write throttling queue depths for a FA port connected to a device)

Powerpath load balancing and failover policies

ad (Adaptive – LB)

– IOs are assigned to paths based on an algorithm that takes into account path load and logical device priority. It is default.

bf (Basic Failover – FO)

– Failover policy which will work when the port on HBA is zoned to a port on two separate FAs on a sym)

co (Clarion Optimization – LB/FO)

– Only for Clariion – default

lb (Least blocks – LB/FO)

– LB is babased on the number of blocks in pending I/Os. I/O requests are routed to the path with the fewest queued blocks, regardless of the number of requests involved.

li (Least I/O – LB/FO)

– Load balance is based on the number of pending I/Os. I/O requests are routed to the path with the fewest queued requests, regardless of total block volume.

re (Request – FO)

– Uses one arbitrary path for all IOS, no load balancing offered. Only FO in place.

rr (Round Robin – LB/FO)

– I/O requests are assigned to each available path in rotation.

si (StreamIO – LB)

– For each possible path for an I/O to a particular volume, this policy selects the same path as was selected for the previous I/O to the volume, unless the pending I/O count since the last path change exceeds the volume.s threshold value. When the threshold is exceeded, the policy selects a new path based on the adaptive policy algorithm

so (symmetrix optimization – LB/FO)

– I/O requests are routed to paths based on an algorithm that takes into account path load and logical device priority. Load is a function of the number, size, priority, and type of I/O queued on each path. This policy is valid only for Symmetrix storage systems and is the default policy for them, on platforms with a valid PowerPath license.

How to check IO policy (Powerpath policy)?

IO Policy (or powerpath policy) can be set per device (including for all devices) on a given server. And hence there is no single value for the whole server or rather a server CAN have more than one value for IO Policy. For this reason, there is no single straightforward command to show IO policy set on a server. Instead it can be checked on per device basis by checking out of /etc/powermt display dev=all. In the example below, policy can be found in line 4 “policy=SymmOpt”.

Pseudo name=emcpower168a
Symmetrix ID=000292601234
Logical device ID=01AC
state=alive; policy=SymmOpt; priority=0; queued-IOs=0;
==============================================================================
--------------- Host ---------------   - Stor -   -- I/O Path --  -- Stats ---
###  HW Path               I/O Paths    Interf.   Mode    State   Q-IOs Errors
==============================================================================
3072 pci@7c0/pci@0/pci@8/fibre-channel@0/fp@0,0 c2t509740854011Cd8s0 FA  8eA active  alive  0  0
3074 pci@7c0/pci@0/pci@9/fibre-channel@0/fp@0,0 c4t5097408540120d8s0 FA  9eA active  alive  0  0

The policy however can be set for the whole system using /etc/powermt set policy option.

How does powerpath priority work?

powermt set priority sets the I/O priority for devices. I/O to lower-priority logical devices tends to be delivered to paths not used by higher-priority logical devices. This command is relevant only for those devices whose load-balancing and failover policy is set to Symmetrix optimization, or CLARiiON optimization, or Adaptive. Although it sets priorities for devices with other policies, it has no effect on load balancing on those devices. If the policy for a given device is anything other than Symmetrix optimization, CLARiiON optimization, or Adaptive when a priority for that device is set, the priority setting has no effect; if, however, the policy for that device is later changed to Symmetrix optimization, CLARiiON optimization, or Adaptive, the previously set priority becomes relevant when the policy change occurs. Setting a high priority for several devices improves their I/O performance at the expense of the remaining devices, while otherwise maintaining the best possible load balance across all paths. For example, to favor certain applications over others, assign a higher priority to the devices used by that application.

How to bring/remove the devices from an array under control of powerpath?

# /etc/powermt manage {dev=device| class={symm|clariion|hitachi|hpxp|hphsx|invista|ess}}
# /etc/powermt unmanage

powermt manage places a specified logical device or storage system class under PowerPath management. By default, all storage system classes are under PowerPath management. You can use powermt unmanage to exclude a storage system class from PowerPath management. A powermt manage class= operation is legal only if the class.s status is unmanaged or manage_incomplete. If the class status is unmanage_incomplete, the unmanage operation must be completed.or the host rebooted. before powermt manage succeeds. The powermt display options command displays the status of a class.
Likewise, by default, when a storage system class is under PowerPath management, all logical devices of that class are under PowerPath management. You can use powermt unmanage to exclude an individual device from PowerPath management. Use powermt manage to return that device to PowerPath control. powermt manage returns a device to PowerPath control only if its storage system class is currently managed by PowerPath.

How to delete a path to a logical device

# /etc/powermt remove [class=symm|clariion|ess|hitachi|hpxp|hphsx|invista|all] [force] hba=hba#|all | dev=path|device|all

powermt remove deletes the specified path (or paths) from PowerPath’s list of configured paths. It does not delete the logical device that the paths refer to. As long as the logical device remains visible on the system, the logical device can be reconfigured via powermt config. ! CAUTION Do not remove the last path to a logical device unless you plan to remove the logical device entirely, as data access will be interrupted. On all platforms, powermt remove does not remove a path if that native path is open.for example, if a file system is mounted on the native path device or an application has opened the native path device. Instead, powermt remove removes all specified paths that are not currently open, issues an error message, and exits. If a PowerPath device is open, powermt remove does not remove that device or the last path to that device. In addition, powermt remove operates on paths, so when powermt remove is executed with an HBA or PowerPath device argument, powermt remove iterates through all the associated paths and attempts to remove them. The HBA or PowerPath device is removed only if all the associated paths are successfully removed. To permanently remove a path from the PowerPath configuration:
1. Run powermt check or powermt remove.
2. Physically remove the path.
3. Remove the operating system objects associated with the path and/or device.
4. Run powermt save.

How to rebuild the /kernel/drv/emcp.conf file?

# /etc/powermt config

How to rebuild the powerpath database?

On Solaris:
Remove any files with a .FCS extension in /etc (if exists).
Move and rename /kernel/drv/emcp.conf file and its “.saved” versions to a different location.
Touch /kernel/drv/emcp.conf to recreate the file.
Run /etc/powercf -q and /etc/powermt config.
Verify all the devices now have Pseudo names using powermt_display_dev=all.

On other OS:
Run following commands in the same sequence.

/etc/powercf -q
/etc/powermt config
/etc/powermt check
/etc/powermt restore
/etc/powermt save

How to restore the paths

# /etc/powermt restore [class=symm|clariion|ess|hitachi|hpxp|hphsx|invista|all] [hba=hba#|all] [dev=path|device|all]

powermt restore tests and restores specified paths. It issues test I/Os and responds to the test results as follows:
* If a live path passes the test, powermt restore does nothing.
* If a dead path passes the test, powermt restore marks it alive; PowerPath can now use it. In addition, other dead paths on the same HBA and/or storage system port may be tested.
* If a live path fails the test, powermt restore marks it dead and prints a warning (every time the path fails the test). In addition, other paths that share the same HBA and port may be marked dead, and other paths that share only the HBA or only the port may be tested.
* If a dead path fails the test, powermt restore prints a warning (every time the path fails the test).

There may be a delay in accessing a recovered path. To avoid this delay, run powermt restore after a path is physically restored. In addition to testing and restoring paths, powermt restore attempts to resurrect dead volumes. A volume may be marked dead if write errors occur that could jeopardize the integrity of the data structures, and if subsequent writes could aggravate the problem.