Powerpath and VxVM work together mechanism

The below OPT 409550 could be a very good example for us to understand how Powerpath and VxVM are designed to work together from Engineering’s point of view.

 

10/30/2012 9:29:18 AM Sibel Kamer
The customer reported that unnecessary trespass on CLARiiON ALUA (Failover mode 4) device was observed whenever Solaris 10 host was booted. Then the customer changed failover mode to 1 from 4 and confirmed that trespass didn't occur. Problem was reproduced the issue in local lab

10/30/2012 10:44:07 PM Lindbergh Lin Xu
Assign to Lin for analysis
EQA Engineer changed from BLANK to Lindbergh Lin Xu
Assigned Engineer changed from BLANK to Lindbergh Xu
Lindbergh Xu is added to Notify Others List.
Status For Version 5.3.0 Changed from New to Assigned.
Responsible Engineer For Version 5.3.0 Changed from Thomas Ko to Lindbergh Xu

10/31/2012 4:18:47 AM Lindbergh Lin Xu
Checked offline messages in /var/adm/messages:

Oct 23 17:47:19 sun46 scsi: [ID 107833 kern.warning] WARNING: /pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w5006016246e01182,0 (ssd0):
Oct 23 17:47:19 sun46 drive offline
Oct 23 17:47:19 sun46 scsi: [ID 107833 kern.warning] WARNING: /pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w5006016a46e01182,0 (ssd3):
Oct 23 17:47:19 sun46 drive offline
Oct 23 17:47:19 sun46 scsi: [ID 107833 kern.warning] WARNING: /pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0/ssd@w5006016346e01182,0 (ssd2):
Oct 23 17:47:19 sun46 drive offline
Oct 23 17:47:19 sun46 scsi: [ID 107833 kern.warning] WARNING: /pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0/ssd@w5006016b46e01182,0 (ssd1):
Oct 23 17:47:19 sun46 drive offline

...

Oct 24 11:53:39 sun46 emcp: [ID 801593 kern.notice] Info: Volume 60060160A4002A00AE170E02171DE211 followed to SPA

Transfer to PowerPath sustaining team to confirm if this is related? Thanks
Assigned Engineer changed from Lindbergh Lin Xu to Praveen Satya
Responsible Engineer For Version 5.3.0 Changed from Lindbergh Lin Xu to Ira Kronitz

11/01/2012 3:40:21 PM Ira Kronitz
maybe the description had a typo? It states the failover mode was 4, but then states it was changed from 1 to 4. Was it actually, originally 1? If the failover was changed to the recommended value and the issue was resolved, this OPT should be rejected.

11/05/2012 10:58:11 AM Praveen Kumar Ni Satya
Assigned Engineer changed from Praveen Kumar Ni Satya to Prashant Kulli
Aditya Sahai is added to Notify Others List.
Praveen Satya is added to Notify Others List.

11/06/2012 1:50:31 PM Prashant Kulli
Sibel,Lindbergh,

Request you to put the basic details like Solaris 10 particular update, patch level etc in the description itself since it saves me from downloading the grabdata. Currently I am having problems with downloading the grab data. Meanwhile, Lindbergh, can you try this scenario with 5.5? There have been plenty of improvements in the multipathing extension. I am just curious that this could have been already fixed.

Thanks

11/07/2012 5:07:06 AM Suvendu Sengupta
Suvendu Sengupta is added to Notify Others List.

11/07/2012 6:05:51 AM Suvendu Sengupta
Hi Prashant,

Thanks for looking into this OPT.

For the issue description and the configuration details, please find the below text and html file apart from EMCGrab.

1.
issue_description.TXT

2.
sun46_2012-10-24-12.02.53_Configuration, Analysis Reporting Tool (CART).mht

3.
sun46_ELA_Host_Report_4bfa4cc9-5d36-495c-a978-85a182230d3d.html

In brief configuration details are --

Solaris 10 (Release Solaris 10 10/09 s10s_u8wos_08a SPARC)
Symantec VxVM 5.0 MP3 RP4
PowerPath 5.3 P 03 (build 3)
CX4-480 04.30.000.5.524


I have requested the customer to check with PP 5.5 and let us know the outcome.

Please let us know if you face any issue to download above file and check, if so I will update the text in the OPT from that file
to make it available to you.

Regards
suvendu sengupta

11/07/2012 6:48:26 AM Suvendu Sengupta
Hi Prashant,

I just received reply from the customer. They tried the same test with PP 5.5 P01 and noticed the same trespass issue.

Please check the attached mail -- "RE EMC SR# 50763628 for ITOCHU Techno-Solutions Corporation.msg" for more details.

Also attaching log for the same test customer did with PP 5.5.\

New log archive name -- "PowerPath55P01.zip"

Regards
suvendu sengupta

11/07/2012 10:58:29 AM Prashant Kulli
Hi Suvendu, can you update the problem summary correctly? Looks like there is some confusion regarding whether it is a ALUA mode in which we are seeing unnecessary trespass or it is PNR mode.

11/07/2012 11:41:41 PM Suvendu Sengupta
Hi Prashant,

Please check the earlier attached file "issue_description.TXT". Issue is in ALUA mode.

Extarct from the file --
==============================================================
Problem summary:
The customer reported that unnecessary trespass on CLARiiON ALUA (Failover mode 4) device was observed whenever Solaris 10 host was booted. Then the customer changed failover mode to 1 from 4 and confirmed that trespass didn't occur.
==============================================================

Regards
suvendu sengupta

11/08/2012 2:45:05 AM Prashant Kulli
I am bit confused here. You say that the problem is seen with failover mode 4; and then you say that it is fixed with failover mode 4. :)

Please clarify.
Status For Version 5.3.0 Changed from Assigned to Returned.

11/08/2012 3:25:16 AM Suvendu Sengupta
Hi Prashant,

I do not have idea where you had seen that issue got resolved in ALUA or Failover mode 4.

Please check the file "issue_description.TXT" --

Extract from the file --
==============================================================
Problem summary:
The customer reported that unnecessary trespass on CLARiiON ALUA (Failover mode 4) device was observed whenever Solaris 10 host was booted. Then the customer changed failover mode to 1 from 4 and confirmed that trespass didn't occur.
==============================================================

Please note -- customer changed the failover mode to "1" which was "4" (ALUA) before and issue got fixed. So we are facing issue when the
failover mode is 4 which mean ALUA.

Regards
suvendu sengupta
Status For Version 5.3.0 Changed from Returned to Assigned.

11/08/2012 4:39:49 AM Prashant Kulli
Ah, I have been reading it "from 1 to 4" instead of "to 1 from 4". My mistake.
I will try to reproduce this on Solaris 10 with 5.5 P01. I will keep you posted.

Thanks

11/08/2012 4:55:29 AM Suvendu Sengupta
Adding Sakai, Akimitsu to notify list.
Akimitsu Sakai is added to Notify Others List.

11/15/2012 4:37:06 AM Suvendu Sengupta
Hi Prashant,

Please update us on this OPT.

Regards
suvendu sengupta

11/16/2012 2:02:56 AM Prashant Kulli
Suvendu, I am unable to reproduce the issue. From the OPT description I see that it was reproduced locally. Do we have access to that setup?
Status For Version 5.3.0 Changed from Assigned to Returned.

11/16/2012 2:28:38 AM Suvendu Sengupta
Hi Prashant,

Sorry we do not have access to the set-up where Akimitsu re-produced this issue. But I sent a mail to him, awaiting his reply.

Regards
suvendu sengupta

11/16/2012 3:39:22 AM Suvendu Sengupta
Hi Prashant,

Received reply from Akimitsu, please find test set-up access details here...

=========================
Hi Suvendu,

Here is the access information. He can use them and reboot the host whatever and whenever he wants.

Solaris 10
IP: 10.32.46.46
root/password

Console: 10.32.46.45
root/password

-> start /SP/console

CX4-480 (CKM00103501202)
IP: 10.32.46.25/26
sysadmin/sysadmin


I presume that Symantec VxVM version 5.0 MP3 RP4 is an important factor to reproduce this issue.
As he may want to install RP4 in his host if it has not been installed yet, I have copied it under /usr1/software/symantec/50MP3RP4 on my host.


Regards Akimitsu,

Akimitsu Sakai
Field Support Specialist
=========================

Regards
suvendu sengupta
Status For Version 5.3.0 Changed from Returned to Assigned.

11/21/2012 6:12:51 AM Prashant Kulli
Thanks for the setup. I am currently using the setup provided above to collect debug information. I will let you know as soon as there is root cause available.

Thanks

11/27/2012 4:33:03 AM Suvendu Sengupta
Hi Prashant,

Please let us know the current status of this OPT ?

Best regards
suvendu sengupta

11/27/2012 4:57:04 AM Prashant Kulli
Suvendu, the RCA is still in progress.
Just curious, why is this U1/S1? Is customer having DU/DL situation? and also I want someone to reaffirm that this issue is seen only when VXVM is installed?

Thanks
Status For Version 5.3.0 Changed from Assigned to Returned.

11/27/2012 8:14:38 AM Suvendu Sengupta
Hi Prashant,

I received reply from Akimitsu, please find the same here..

=======================================================================
Hi Suvendu,

We can lower the severity of this issue even though the customer and Symantec have been waiting for an update for a month.
Yes, both the customer and I have confirmed that this issue was always observed with VxVM + PowerPath combination and not on simply PowerPath (no VxVM).

I have received an analysis from Symantec Engineer via the customer.
As you know, From PowerPath perspective, In ALUA mode, I/Os are not serviced by both SPs for any given LUN. The I/Os are processed by the LUN-owning SP.

However, here is what Symantec Engineer told me.
From VxVM/DMP perspective, During boot, VxVM/DMP sends an inquiry command to both SPs to distinguish which SP owns the LUN. However, PowerPath redirects this inquiry command from the non-owning SP to the LUN-owning SP. As a result, VxVM/DMP received that the LUN is being owned by both SPs and considered something wrong and then initiated a trespass command. Symantec believe that this unnecessary trespass would not occur if PowerPath doesn’t redirect the inquiry command from VxVM/DMP.


Regards Akimitsu,

Akimitsu Sakai
=======================================================================

Best regards
suvendu sengupta
Severity For Version 5.3.0 Changed from Critical/High to Medium.
Status For Version 5.3.0 Changed from Returned to Assigned.
Urgency For Version 5.3.0 Changed from Critical to High.

11/27/2012 9:05:40 AM Prashant Kulli
Thanks Suvendu. This information from Veritas folks would give a direction for our analysis.

I have below questions to Symantec:
I am just curious, would DMP still manage paths when PowerPath is there on the system? I thought DMP would act as a passthrough when PowerPath is installed on the host. So is there is a need for DMP to send inq to the native devices while PowerPath is in the picture?

Thanks
Status For Version 5.3.0 Changed from Assigned to Returned.

11/28/2012 4:02:02 AM Suvendu Sengupta
Hi Prashant,

Received reply from Akimitsu with Symantec answer, please find the same here ...

==================================================
Hi Suvendu,

Here is an answer from Symantec.
Yes, if boot process of a system with PowerPath is completed, DMP doesn't manage directly native paths. We can see this issue only during system boot. When a system is booting, VxVM/DMP starts to work earlier than PowerPath and at first determines as multi native paths configuration which DMP should manage. Then PowerPath starts to work and finishes its initialization, VxVM/DMP in turn detects PowerPath devices and switches its mode, i.e. PowerPath manages native paths and DMP acts as a passthrough. This issue can occur during this mode transition; DMP is doing device claiming via SCSI inquiry and PowerPath has just started to work.

Regards Akimitsu,

Akimitsu Sakai
==================================================

Best regards
suvendu sengupta
Status For Version 5.3.0 Changed from Returned to Assigned.

11/28/2012 7:14:31 AM Prashant Kulli
From the last two updates I feel that Symantec eng is contradicting their own statements. I feel something is really incorrect here. In the first update it was said that VxVM/DMP issues INQ on a path and PowerPath is redirecting it to a different path. That would mean - PowerPath is already there in the driver stack but still VxVM/DMP is issueing INQ? In the latest response they are saying, VxVM/DMP starts to work eariler than PowerPath. But in reality if you see /etc/system file, the vxdmp is loaded after all PowerPath drivers are loaded.
Can you ask Symantec eng if there is a way to make sure VxVM/DMP doesn't send INQ to native paths while PowerPath is managing the devices. On largers configurations this might lead unnecessary boot delays if DMP is trying to fetch the path information when PP is already managing the devices.

Thanks
Status For Version 5.3.0 Changed from Assigned to Returned.

12/03/2012 2:05:55 AM Suvendu Sengupta
Hi Prashant,

I received Symantec's answer, please find the same below.

=====================================================================================================
Hi Suvendu,

I have received the following update from Symantec.

I'm sorry that our explanation was not enough and caused confusing. Please let me explain again about the reason why VxVM/DMP sends inquiries to PP-managed LUNs during system boot.

VxVM/DMP uses two modes "boot" and "enabled" when a system is booting up. At an early stage of system boot, vxconfigd(VxVM daemon) starts to work in "boot" mode(on Solaris 10 and later, by svc:/system/vxvm/vxvm-sysboot) so that it can handle boot tasks e.g. enabling encapsulated OS disks. And then OS boot process goes on and various services are initialized, vxconfigd now works in "enabled" mode(by svc:/system/vxvm/vxvm-startup2).

On the other hand, regarding PowerPath, PowerPath kernel module also starts to work early on boot process as you pointed, but we'd like to focus powermt command here. VxVM/DMP only supports PP-unmanaged devices as OS boot disks, and it uses "powermt display unmanaged" command in the process to get the list of PP-unmanaged disks. However, in early stage of OS boot, this command is not ready yet to work(we got error 103), so VxVM/DMP treats all EMC devices as PP-unmanaged, i.e. DMP tries to manage them so that it can proceed required tasks for OS disk etc.(**1)

After OS boot process goes on, vxconfigd then switches its mode to "enabled". Since "powermt display unmanaged" command has been ready to work at this stage, VxVM/DMP can now determine if a LUN is PP-managed or not. So VxVM/DMP changes its behavior to act as a passthough about LUNs which PP manages.

This problem occurs during (**1) seen above. In this stage, VxVM/DMP sends inquiries to all LUNs to figure out the status of them, while PowerPath kernel module has been loaded and started to work. When it sends inquiries to OS disks(PP-unmanaged devices), there's no problem. But when it sends inquiries to PP-managed devices, in ALUA mode, PowerPath redirects them sent via different paths to the same owner SP and VxVM/DMP detects both SPs owns the LUN, then issues trespass.

We cannot delay the timing of starting "boot" mode until powermt is ready because VxVM/DMP has to work tasks about OS disks so that OS boot process continues. So, unfortunately, there's no easy way to work around this behavior that VxVM/DMP sends inquiries to both PP-managed and PP-unmanaged LUNs during system boot.


Regards Akimitsu,

Akimitsu Sakai
Field Support Specialist
Customer Support - Global Services - Japan
EMC Japan K.K
Phone: [phone] (Group)
Phone: [phone] (Direct)
Cell: [phone]
=====================================================================================================

Best regards
suvendu sengupta
Status For Version 5.3.0 Changed from Returned to Assigned.

12/03/2012 10:06:05 AM Suvendu Sengupta
Selva Kuttiraja is added to Notify Others List.

12/05/2012 4:11:31 AM Prashant Kulli
Thanks for providing some important information of how VxVM/DMP works during boot.
First of all, I would like to mention that it is not a correct way for VxVM/DMP to contend for the paths while there is another multipath solution, i.e. PowerPath present in the picture. It also wrong to run "powermt display unmanaged" command while PowerPath is not configured completely. VxVM/DMP has to find another way of finding out the PP unmanaged devices. They should probably create their own file to store the PP unmanaged devices and during reboot check the file instead of running "powermt display unmanaged" command.OR they MUST wait until PP configuration is done and then only proceed with "powermt display unmanaged" command. Also trespass shouldn't initited by DMP when PowerPath is loaded. Please pass this information to Vertias eng.

Regarding the INQ redirection, there is a Sun compiler bug which we are following up with Oracle, because of which INQ is getting redirected. The customer has to pursue with Symantec to workaround this so that they don't manage paths during early boot. Let us know if there is any more clarification required.
Status For Version 5.3.0 Changed from Assigned to Returned.

12/06/2012 3:50:09 AM Suvendu Sengupta
Hi Prashant,

Akimitsu received answer from Symantec, pelase find the same here...

================================================
Hi Suvendu,

I have just received the following update from Symantec.

Thanks much for your comment about VxVM/DMP boot process. Symantec will discuss based on your suggestion whether there're better and possible ways to improve our boot process. Meanwhile, please let us confirm about INQ redirection. Is it unexpected behavior, i.e. is the original design of PowerPath (without Sun compiler bug) not to redirect INQ? If so, how long can we estimate until the release of new PowerPath without Sun compiler bug? Of course, this doesn't mean that we ask you to commit, we just want to know even rough projection to look for actions Symantec can take for customers. It must be difficult to say clearly, but would you please provide forecast of the fix (a few months or over a year etc.)?

Regards Akimitsu,
================================================

Best regards
suvendu sengupta
Status For Version 5.3.0 Changed from Returned to Assigned.

12/10/2012 2:45:31 PM Prashant Kulli
We don't have timeline for a fix. We have opened the dialog with Oracle and we don't know when it will be addressed. Meanwhile we would like to know few details from Symantec folks. Is "powermt display unmanaged" run as part of a startup script? At what stage of booting, is it run? How long does VxVM/DMP has to wait for PowerPath to configure completely during boot?

As I stated earlier, customer has to push Symantec for a workaround.VxVM/DMP shouldn't initiate trespass while PowerPath is managing all the devices. What is the expectation from PowerPath by the customer currently? How often they reboot their servers?
Status For Version 5.3.0 Changed from Assigned to Returned.

12/17/2012 6:52:28 AM Prashant Kulli
Any updates from the customer on my queries in the last update?

12/18/2012 5:55:12 PM Selva Kuttiraja
Updated by Selva:
******************

Here is the answer and question from Symantec and the update about the customer status.

Answer from Symantec:
> Is "powermt display unmanaged" run as part of a startup script?
No, vxconfigd (VxVM daemon) uses array-specific liraries to manage disks. powermt is called from a library for PowerPath devices.

> At what stage of booting, is it run?
Twice, svc:/system/vxvm/vxvm-sysboot and svc:/system/vxvm/vxvm-startup2.

> How long does VxVM/DMP has to wait for PowerPath to configure completely during boot?
Unfortunately, we don't know actual time to wait, but we cannot delay the timing until powermt is ready as we explained in previous update.

Question to PowerPath Engineering from Symantec:
From PowerPath perspective, Since PowerPath Engineering has understood how VxVM/DMP works during boot, do you have any concerns other than this unnecessary trespass?


Update about the customer:
> What is the expectation from PowerPath by the customer currently?
As they want to have a permanent fix of this problem, the customer would like to EMC to help and assist Symantec when Symantec has a question and request for PowerPath design.

> How often they reboot their servers?
The customer, ITOCHU Techno-Solutions, is one of the biggest System Integrator (EMC partner) in Japan and has been selling a lot of Solaris hosts with EMC and Symantec to their customers so it depends on his customer.

12/19/2012 6:38:21 AM Prashant Kulli
We are in touch with Symantec engineering for this issue.

Can you ask the customer to download and install the latest ASL/APM for EMC arrays -version 6.0.1.002 and test if it fixes the issue?

Thanks

12/21/2012 6:44:07 PM Selva Kuttiraja
Updated Selva:
**************
Below response from customer.

I have asked Symantec if he can provide me with SF6.0.1 package.
He provided me with SF6.0.1 (6.0.100.000) package so I have just installed it on my host in local lab and tested with both PP5.3P03 and PP5.5 P1 to see if SF 6.0.1 fixes the issue.

I observed that the issue occurred with SF 6.0.1 and PP 5.5 P1. However, interestingly, I didn’t see the issue with SF 6.0.1 and PP5.3 P03.
I think that the following differences from the DMP debug logs is what Symantec was saying. It seems that it depends on timing differences of PowerPath and DMP initialization process but I am unable to find exactly why the issue was disappeared with SF 6.0.1 and PP53 P03.

Here is relevant excerpts pertaining to the time-line:
root@sun46 # grep "thread 58" vxconfigd.log.PP53
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14870 ddl_thread_claim_disk(thread 58): disk is c2t5006016A46E01182d0
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxfusionio.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxfusionio.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxpp.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxpp.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxap.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxap.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxrdac.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxrdac.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxvpath.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxvpath.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxCLARiiON.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[0] = CAB_SERIAL_NO=CKM00103501202
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[1] = LUN_SERIAL_NO=60060160A4002A00AC170E02171DE211
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[2] = REVISION=0430
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[3] = SCSI_VERSION=4
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[4] = LUN_OWNER=Y
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[5] = CUR_OWNER=Y
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[6] = ARRAY_CTLR_ID=B
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[7] = PORT_SERIAL_NO=B2
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[8] = ATYPE=CLR-ALUA
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[9] = ARRAY_VOLUME_ID=100
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[10] = PID=RAID 5
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[11] = HARDWARE_MIRROR=no
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[12] = DDL_DEVICE_ATTR=lun RAID_5
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[13] = DDL_THIN_DISK=thick
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-26801 ddl_vendor_check_attributes(thread 58): 4216830940 libvxCLARiiON.so /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-16356 ddl_vendor_claim_device(thread 58): lib = libvxCLARiiON.so dev = /dev/rdsk/c2t5006016A46E01182d0s2 returns DDL_CLAIMED
12/20 17:02:21: VxVM vxconfigd DEBUG V-5-1-24546 ddl_vendor_claim_device(thread 58): /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:02:22: VxVM vxconfigd DEBUG V-5-1-15012 dmp_make_mpnode(thread 58): devno 0x4b80000 device tag = disk_0
root@sun46 #

12/21/2012 7:03:24 PM Selva Kuttiraja
Continuous:

root@sun46 # grep "thread 58" vxconfigd.log.PP55
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14870 ddl_thread_claim_disk(thread 58): disk is c2t5006016A46E01182d0
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxfusionio.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxfusionio.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxpp.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxpp.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxap.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxap.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxrdac.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxrdac.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxvpath.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14879 ddl_vendor_claim_device(thread 58): lib = libvxvpath.so did not claim dev = /dev/rdsk/c2t5006016A46E01182d0s2 stat = UNCLAIMED
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-26802 ddl_vendor_claim_device(thread 58): devinfo 4216830940 entered for library = libvxCLARiiON.so dev = /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[0] = CAB_SERIAL_NO=CKM00103501202
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[1] = LUN_SERIAL_NO=60060160A4002A00AC170E02171DE211
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[2] = REVISION=0430
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[3] = SCSI_VERSION=4
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[4] = LUN_OWNER=N
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[5] = CUR_OWNER=N
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[6] = ARRAY_CTLR_ID=A
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[7] = PORT_SERIAL_NO=A3
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[8] = ATYPE=CLR-ALUA
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[9] = ARRAY_VOLUME_ID=100
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[10] = PID=RAID 5
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[11] = HARDWARE_MIRROR=no
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[12] = DDL_DEVICE_ATTR=lun RAID_5
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-14874 ddl_vendor_claim_device(thread 58): Returned nvlist[13] = DDL_THIN_DISK=thick
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-26801 ddl_vendor_check_attributes(thread 58): 4216830940 libvxCLARiiON.so /dev/rdsk/c2t5006016A46E01182d0s2
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-16356 ddl_vendor_claim_device(thread 58): lib = libvxCLARiiON.so dev = /dev/rdsk/c2t5006016A46E01182d0s2 returns DDL_CLAIMED
12/20 17:29:23: VxVM vxconfigd DEBUG V-5-1-24546 ddl_vendor_claim_device(thread 58): /dev/rdsk/c2t5006016A46E01182d0s2

root@sun46 # powermt display dev=emcpower3
Pseudo name=emcpower3a
CLARiiON ID=CKM00103501202 [sun46]
Logical device ID=60060160A4002A00AC170E02171DE211 [LUN 100]
state=alive; policy=CLAROpt; queued-IOs=0
Owner: default=SP B, current=SP B Array failover mode: 4
==============================================================================
--------------- Host --------------- - Stor - -- I/O Path -- -- Stats ---
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
3074 pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0/fp@0,0 c2t5006016246E01182d0s0 SP A2 active alive 0 0
3074 pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0/fp@0,0 c2t5006016A46E01182d0s0 SP B2 active alive 0 0
3072 pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0 c3t5006016346E01182d0s0 SP A3 active alive 0 0
3072 pci@0/pci@0/pci@8/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0 c3t5006016B46E01182d0s0 SP B3 active alive 0 0

root@sun46 # grep 60060160A4002A00AC170E02171DE211 /var/adm/messages.PP55
(snip)
Dec 20 17:29:23 sun46 emcp: [ID 801593 kern.notice] Info: Volume 60060160A4002A00AC170E02171DE211 followed to SPA
Dec 20 17:29:24 sun46 emcp: [ID 801593 kern.notice] Info: Assigned volume 60060160A4002A00AC170E02171DE211 to SPB

I am attaching some logs from my host so please kindly pass it to PP Engineering and he can also use my host for some testing if he wants.

Attaching the log file name "SF601.zip"

12/21/2012 7:06:15 PM Selva Kuttiraja
Updated by selva:
******************

I have received a question regarding an INQ redirection which is a Sun compiler bug from the customer.

The customer has to push Symantec for a workaround or fix, but Symantec told the customer and us that they need to change current DMP design and it is not easy. So the customer started asking EMC if we can resolve a Sun compiler bug from PowerPath side as another option.
Could you please provide us more details about a Sun compiler bug issue, so we can explain the customer that’s not the root cause. As it is now, the customer is derailed and under impression EMC has to do something.

Thanks
Selva

12/27/2012 2:46:48 AM Selva Kuttiraja
Update by selva:
*******************

Have received the below update from customer :


As I am trying to find out the workaround for the customer, I noticed that the following interesting behaviors.

With VxVM 5.0 MP3 RP4 & PP 5.3 P03:
The ALUA LUNs were trespassed during a boot. So the customer manually has to issue “powermt restore” command after any reboots. This is painful and annoying for the customer.

With VxVM 5.0 MP3 RP4 & PP 5.5 P1:
The ALUA LUNs were trespassed but they were automatically restored during a boot. So no action is required after any reboots.

With VxVM 6.0.1 & PP 5.3 P03:
The ALUA LUNs were NOT trespassed during a boot.

With VxVM 6.0.1 & PP 5.5 P1:
The ALUA LUNs were trespassed but they were automatically restored during a boot. So no action is required after any reboots.

I have checked why these trespassed LUN were automatically restored during a boot and found that the “powermt load” command is executed at startup.
Then the “powermt load” with PowerPath 5.5 P1 restores the trespassed ALUA LUNs to default SP owner but the “powermt load” with PowerPath 5.3 P03 does NOT restore trespassed LUNs.

Consequently, I think that upgrading PowerPath to 5.5 P1 can be used as a workaround for the customer.


Please advise with your inputs for the customer update.

12/28/2012 11:10:17 AM Selva Kuttiraja
Updated by Selva:
******************

The customer has accepted to upgrade PowerPath to 5.5P01 as workaround.
As the customer should follow up with Symantec for a permanent fix from DMP side, I believe that we should be able to close this SR and OPT if we can provide the customer with some more information or explanation for INQ redirection Oracle bug.

01/09/2013 6:50:50 AM Prashant Kulli
Hi Selva,

Thanks for the update. If the customer is okay to upgrade PowerPath it is good. Regarding the INQ redirection issue, we can look at it in two ways:
1. In the first place INQ shouldn't be issued by VxVM because VxVM is assuming that PowerPath is not managing any device. We are working with Symantec to make sure VxVM doesn't manage paths when PowerPath is in picture.
2. INQ redirection issue in PP code. Oracle is going to provide some recommendation or best practice which would avoid such an issue. We will include it in PP 5.5 P02 and later releases.

Please update the OPT if you close the SR.

01/10/2013 12:22:11 PM Selva Kuttiraja
Updated by Selva:
******************

Customer responsed with regards to the INQ redirection issue,
This is what the customer would like to know. Thank you.

The customer is ok to upgrade PowerPath to 5.5 as a workaround and is happy about both updates below. So we can close this SR now.

01/10/2013 9:38:35 PM Lindbergh Lin Xu
Closing OPT per updates from customer.
Chi Liu is added to Notify Others List.
Status For Version 5.3.0 Changed from Returned to Solved.

02/15/2013 6:00:55 AM Venkatram Amalanathan
QE DEA status is set to In-progress as a test case is created and send for review with QC Admin.

02/18/2013 11:55:35 AM Venkatram Amalanathan
A test cases exists for this scenario with the Test ID : 6653 and Test Name : Unnecessary trespass should not happen across host reboot.

05/21/2013 8:07:14 AM Prashant Kulli
Updating "Problem Key"

INQ redirection issue has been addressed with PP 5.5 P02 GA.