UNMOUNT DATABASE command hangs for one hour

99 Views

Hello,

We encountered a strange situation today with ASE 16 SP02 PL04 on Linux. Our product issued the command "UNMOUNT DATABASE [xyz] to '/some/manifest' WITH OVERRIDE". After waiting for an hour it timed out and failed. The output to the command showed error 2261 before failing to unmount:

[2017-02-22 09:50:39,164][DEBUG][ase.target.ASEBaseTargetDatabase#unmountDatabaseException:345][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] Unmounting  xyz
[2017-02-22 10:50:40,152][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 3300 secs. Please wait.
[2017-02-22 10:50:40,153][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 3000 secs. Please wait.
[2017-02-22 10:50:40,153][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 2700 secs. Please wait.
[2017-02-22 10:50:40,154][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 2400 secs. Please wait.
[2017-02-22 10:50:40,154][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 2100 secs. Please wait.
[2017-02-22 10:50:40,154][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 1800 secs. Please wait.
[2017-02-22 10:50:40,155][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 1500 secs. Please wait.
[2017-02-22 10:50:40,155][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 1200 secs. Please wait.
[2017-02-22 10:50:40,155][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 900 secs. Please wait.
[2017-02-22 10:50:40,155][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 600 secs. Please wait.
[2017-02-22 10:50:40,156][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2261 : Dbshutdown in progress for database xyz with dbid 25310. Timeout left 300 secs. Please wait.
[2017-02-22 10:50:40,156][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:949][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ASE message 2251 : Warning: HA Subsystem Internal Error - dbshutdown failed on database xyz with dbid 25310 because the database was found to be in use after it was marked for shutdown.
[2017-02-22 10:50:40,156][DEBUG][ase.dbc.ASEJDBCAccessorImpl#executeWithProgress:916][Worker-382|JOB-252|SOURCE_STOP(ASE_VIRTUAL_SOURCE-2)][ACTION-1020] ERROR: 2251 : Warning: HA Subsystem Internal Error - dbshutdown failed on database xyz with dbid 25310 because the database was found to be in use after it was marked for shutdown.

I grabbed a stack trace of the unmount command:

1> dbcc traceon(3604)
2> go
DBCC execution completed. If DBCC printed error messages, contact a user with System Administrator (SA) role.
1> dbcc stacktrace(190)
2> go
Start dbcc stacktrace...
pc: 0x0000000001dc4908 upsleepgeneric+0x6b8()
pc: 0x0000000001dbf2a3 uppause+0x103()
pc: 0x0000000000ae0b96 dbs__check_timeout+0x47()
pc: 0x0000000000ada8ba dbs_waitfortasks+0x19e()
pc: 0x0000000000adb090 dbs__politeshutdown+0x1f()
[Handler pc: 0x0x0000000000f2e063 std_handle installed by the following function:-]
pc: 0x0000000000ad980d dbs_shutdown+0x84b()
pc: 0x00000000011f30ca umt_dbshutdown+0x199()
[Handler pc: 0x0x00000000011ee5f0 umt_handle installed by the following function:-]
pc: 0x00000000011f3fa7 unmountdb+0x124()
pc: 0x0000000001c3db23 s_execute+0x5633()
[Handler pc: 0x0x0000000001c90060 hdl_stack installed by the following function:-]
[Handler pc: 0x0x0000000001c5e6f0 s_handle installed by the following function:-]
pc: 0x0000000001c62341 sequencer+0xcb1()
pc: 0x00000000015d4c8f tdsrecv_language+0x1df()
[Handler pc: 0x0x0000000001d6eb60 ut_handle installed by the following function:-]
pc: 0x00000000015e41f9 conn_hdlr+0xe49()
dbcc stacktrace finished.
DBCC execution completed. If DBCC printed error messages, contact a user with System Administrator (SA) role.

We restarted ASE and tried it again and it succeeded.

Curious if this is a known issue? If not, what type of CSMD would be recommended (error 2261 or manual)?

Thanks,

Neal