HANA recovery fails with error Recovery failed in nameserver startup: An error occured, while recovering the persistency

[former_member248342]

  Hi Experts,

I am facing issue while HANA DB recovery. I am getting following error.

2015-08-14T11:38:04+00:00  P063577      14f2c002d18 INFO    RECOVERY from file: log_backup_4_0_3650185088_3650203328.1439296265648 in /hana_db_backup/log/

2015-08-14T11:38:04+00:00  P063577      14f2c002d18 INFO    RECOVERY from file: log_backup_4_0_3650203328_3650221440.1439297165655 in /hana_db_backup/log/

2015-08-14T11:38:04+00:00  P063577      14f2c002d18 INFO    RECOVERY from file: log_backup_4_0_3650221440_3650239552.1439298066512 in /hana_db_backup/log/

2015-08-14T11:38:04+00:00  P063577      14f2c002d18 INFO    RECOVERY from file: log_backup_4_0_3650239552_3650295232.1439298966616 in /hana_db_backup/log/

2015-08-14T11:38:04+00:00  P063577      14f2c002d18 INFO    RECOVERY from file: log_backup_4_0_3650295232_3650313600.1439299866607 in /hana_db_backup/log/

2015-08-14T11:38:04+00:00  P063577      14f2c002d18 INFO    RECOVERY from file: log_backup_4_0_3650313600_3650331776.1439300777425 in /hana_db_backup/log/

2015-08-14T11:38:04+00:00  P063577      14f2c002d18 INFO    RECOVERY state of service: nameserver, hostname:30001, volume: 1, RecoveryExecuteTopologyRecoveryInProgress

2015-08-14T11:38:05+00:00  P063577      14f2c002d18 INFO    RECOVERY state of service: nameserver, hostname:30001, volume: 1, RecoveryExecuteTopologyAndSSFSRecoveryFinished

2015-08-14T11:38:05+00:00  P063577      14f2c002d18 INFO    RECOVERY state of service: nameserver, hostname:30001, volume: 1, RecoveryPrepared

2015-08-14T11:38:05+00:00  P063577      14f2c002d18 INFO    RECOVERY state of service: indexserver, hostname:30003, volume: 3, RecoveryPrepared

2015-08-14T11:38:05+00:00  P063577      14f2c002d18 INFO    RECOVERY state of service: statisticsserver, hostname:30005, volume: 4, RecoveryPrepared

2015-08-14T11:38:05+00:00  P063577      14f2c002d18 INFO    RECOVERY state of service: xsengine, hostname:30007, volume: 2, RecoveryPrepared

2015-08-14T11:38:05+00:00  P063577      14f2c002d18 INFO    RECOVERY start of progress monitoring, volumes: 4, bytes: 0

2015-08-14T11:38:05+00:00  P063577      14f2c002d18 ERROR   RECOVERY RECOVER DATA finished with error: [110092] Recovery failed in nameserver startup: An error occured, while recovering the persistency

I will appreciate any help here.

Thanks !!

[Former Member]

HI Nitesh,

Have you done the Static server migration ?

what is the backup that you used ? is it the same version ?

Regards,

Pavan Gunda

[lucas_oliveira]

Hello,

2015-08-14T11:38:05+00:00  P063577      14f2c002d18 ERROR   RECOVERY RECOVER DATA finished with error: [110092] Recovery failed in nameserver startup: An error occured, while recovering the persistency

It's quite clear that there's something wrong going on the nameserver. Therefore, get the nameserver traces by the same timeframe you tried the restore and check what's going on. If you need further help, share the contents of that file (for the same timeframe mentioned earlier, of course).

BRs,

Lucas de Oliveira

[former_member248342]

Hi Lucas,

Following are logs from nameserver_alert_<hostname>.trc

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.245524 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.255321 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.272524 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.282711 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.297481 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.305246 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.307946 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.313681 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.326432 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.335176 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63598]{-1}[-1/-1] 2015-08-14 11:38:04.339002 e Backup           BackupDest_DestinationCheck.cpp(00181) : xxx volumeID to check: 4 servicename to check: statisticsserver

[63681]{-1}[-1/-1] 2015-08-14 11:38:05.502745 f PersistenceLayer PersistenceController.cpp(00282) : try to open existing persistence log volume, but filesystem path does not exists /hanabackup/log/mnt00001/hdb00001/

[63681]{-1}[-1/-1] 2015-08-14 11:38:05.503075 f PersistenceLayer PersistenceController.cpp(00604) : startup failed exception  1: no.70029048  (Logger/PersistenceController.cpp:284)

    try to open existing persistence log volume, but filesystem path does not exists: /hanabackup/log/mnt00001/hdb00001/

exception throw location:

1: 0x00007f1bac3c394b in PersistenceController::checkVolumes(NameServer::ServiceStartInfo const*)+0x7d7 at PersistenceController.cpp:284 (libhdblogger.so)

2: 0x00007f1bac3c601d in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, DataAccess::TablePreloadWriteCallback*, DataAccess::TablePreloadReadCallback*)+0x1099 at PersistenceController.cpp:435 (libhdblogger.so)

3: 0x00007f1baca2aa66 in NameServer::Topology::recoverPersistence()+0x222 at Topology.cpp:168 (libhdbns.so)

4: 0x00007f1bac924672 in NameServer::TREXNameServer::recoverTopology()+0x40 at TREXNameServer.cpp:1915 (libhdbns.so)

[63681]{-1}[-1/-1] 2015-08-14 11:38:05.503075 f PersistenceLayer PersistenceController.cpp(00604) :  5: 0x00007f1bac9d97c8 in NameServer::RecoverThread::run(void*&)+0x14 at TREXNameServer.cpp:327 (libhdbns.so)

6: 0x00007f1ba9a08f17 in Execution::Thread::staticMainImp(void**)+0xa93 at Thread.cpp:476 (libhdbbasis.so)

7: 0x00007f1ba9a0945d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:546 (libhdbbasis.so)

[63681]{-1}[-1/-1] 2015-08-14 11:38:05.522677 f topology         Topology.cpp(00176) : persistence startup failed with persistence startup error. exception  1: no.70029048  (Logger/PersistenceController.cpp:284)

    try to open existing persistence log volume, but filesystem path does not exists: /hanabackup/log/mnt00001/hdb00001/

exception throw location:

1: 0x00007f1bac3c394b in PersistenceController::checkVolumes(NameServer::ServiceStartInfo const*)+0x7d7 at PersistenceController.cpp:284 (libhdblogger.so)

2: 0x00007f1bac3c601d in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, DataAccess::TablePreloadWriteCallback*, DataAccess::TablePreloadReadCallback*)+0x1099 at PersistenceController.cpp:435 (libhdblogger.so)

3: 0x00007f1baca2aa66 in NameServer::Topology::recoverPersistence()+0x222 at Topology.cpp:168 (libhdbns.so)

4: 0x00007f1bac924672 in NameServer::TREXNameServer::recoverTopology()+0x40 at TREXNameServer.cpp:1915 (libhdbns.so)

5: 0x00007f1bac9d97c8 in NameServer::RecoverThread::run(void*&)+0x14 at TREXNameServer.cpp:327 (libhdbns.so)

6: 0x00007f1ba9a08f17 in Execution::Thread::staticMainImp(void**)+0xa93 at Thread.cpp:476 (libhdbbasis.so)

7: 0x00007f1ba9a0945d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:546 (libhdbbasis.so)

stopping service...

[63681]{-1}[-1/-1] 2015-08-14 11:38:05.549881 f topology         Topology.cpp(00185) : persistence recovery failed

[63681]{-1}[-1/-1] 2015-08-14 11:38:05.550453 e Backup           BackupMgr_Manager.cpp(04603) : RECOVER DATA finished with error: [110092] Recovery failed in nameserver startup: An error occured, while recovering the persistency

[63681]{-1}[-1/-1] 2015-08-14 11:38:05.551100 f NameServer       TREXNameServer.cpp(02876) : An error occured, while recovering the persistency -> stopping instance ...

[63681]{-1}[-1/-1] 2015-08-14 11:38:06.556080 e Basis            TREXNameServer.cpp(02906) : Process exited due to an error via explicit exit call with exit code 1 , no crash dump will be writtenmo-17544e70a:/usr/sap/HNA/HDB00/hostname

[lucas_oliveira]

Hi,

Well it's clear that it is trying to start the persistence on a place that either does not exist or does not have the persistence files

[...]
try to open existing persistence log volume, but filesystem path does not exists: /hanabackup/log/mnt00001/hdb00001/
[...]

You see, /hanabackup/log in my eyes would be the location for the HANA log backups and not the log persistency files.

So you need to check if you have the configuration fine for your recovery (pointing to the location of log backups) as well as the log files path (configuration basepath_logvolumes).

I hope that helps.

BRs,

Lucas de Oliveira