sddone: write error on virtual disk while doing NetApp failover

[former_member350489]

Some years ago we introduced NetApp Filers in our ASE environments here with great success, running ASE on SUSE Linux.

At some point when we did a failover on the NetApp Filer, we ran into serious problems, getting "sddone: write error on virtual disk" and lots of other errors like "Error : 823" and so on. We had a three-part-case open for a very long time with Sybase and NetApp, trying to solve that problem with no luck. We simply could not find the actual cause for this problem, so we took the descision to change our operating system from SUSE to RedHat. And guess what ? The problem never happened again ! According to me, this is one of the worst type of problems one can experience, since we never managed to find the actual cause...

But now we've been running ASE on RedHat for a couple of years, just doing great, and have upgraded both ASE and O/S continuously.

The other week we made a test failover on one of our NetApp Filers, with three test ASE's running against that filer.

The outcome of the tests was really bad, since two out of three ASE's went down due to this "old problem" again :

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: write error on virtual disk 27 block 762880:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: write error on virtual disk 27 block 574464:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: write error on virtual disk 27 block 382464:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: write error on virtual disk 0 block 1384:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0003:00000:00840:2014/11/26 13:47:09.68 server  Update of syslogins failed

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: write error on virtual disk 0 block 1319:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0003:00000:00324:2014/11/26 13:47:09.68 server  Update of syslogins failed

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: write error on virtual disk 0 block 1320:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0003:00000:00245:2014/11/26 13:47:09.68 server  Update of syslogins failed

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: read error on virtual disk 0 block 1457:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: read error on virtual disk 0 block 1454:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: read error on virtual disk 0 block 1454:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: read error on virtual disk 0 block 1454:

00:0015:00000:00000:2014/11/26 13:47:09.68 kernel  sddone: Input/output error

00:0003:00000:00902:2014/11/26 13:47:09.68 server  Error: 823, Severity: 24, State: 1

00:0003:00000:00104:2014/11/26 13:47:09.68 server  Error: 823, Severity: 24, State: 1

00:0003:00000:00243:2014/11/26 13:47:09.68 server  Error: 823, Severity: 24, State: 1

00:0003:00000:00009:2014/11/26 13:47:09.68 server  Error: 823, Severity: 24, State: 1

etc

etc

etc

After starting both ASE's that went down, they went straight up again without any problems whatsoever...

All three ASE's are running on excactly the same version of ASE and O/S :

Adaptive Server Enterprise/15.7.0/EBF 20953 SMP ESD#4.2 /P/x86_64/Enterprise Linux/ase157x/3262/64-bit/FBO/Fri Mar 22 08:57:02 2013

LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch

Red Hat Enterprise Linux Server release 6.5 (Santiago)

2.6.32-431.23.3.el6.x86_64

The only related document I could find was this :

ASE Encounters False 823 Errors on IBM Regatta Servers Technote: Database Management - Sybase Inc

Have anyone of you out there ever had similar problems when failing you NetApp filer ?

Please share your knowledge if you know how to solve this problem

Thanks

/Mike

[former_member188958]

Hi Mike,

Could you describe your system just a little more (are you just using mirroring, full HA, etc.) and detail just what you did to test failover?

Cheers,

-bret

[former_member350489]

Hi Bret,

Thanks for your reply, we use a setup like this

https://library.netapp.com/ecmdocs/ECMP1368845/html/GUID-31DC026F-2B78-425A-BA55-487782F9909A.html

And the failover tests was made using "takeover" and "giveback"

https://library.netapp.com/ecmdocs/ECMP1196796/html/GUID-31D6327E-48F5-461D-9C5B-9F0BDE2C7783.html

Cheers,

/Mike