Skip to Content
0
Former Member
Jun 08, 2016 at 06:55 AM

Error starting database / nameserver

5756 Views

Hi gurus,

After unexpected server shutdown HANA is not starting. Getting a little more deeper in the logs I´ve found the error listet bellow, it happens in nameserver startup process. Any hint?

hana001:/usr/sap/HDB/SYS/exe/hdb> ./hdbnameserver

service startup...

accepting requests at 127.0.0.1:30201; 127.0.0.2:30201

searching for master nameserver hana001:30201 ...

assign as master nameserver. assign to volume 1 started

checking for recovery request ...

loading topology ...

opening persistence ...

assign failed with persistence startup error. exception 1: no.3020046 (DataAccess/PageAccess/impl/PageImpl.cpp:399)

Wrong savepoint version: Expected 8373 but found 8361.

exception throw location:

1: 0x00007fb55f11bbbc in PageAccess::Page::verifyHeader(PageAccess::SizeClass, DataAccess::SavepointVersion const&, bool) const+0x3d8 at PageImpl.cpp:399 (libhdbdataaccess.so)

2: 0x00007fb55ee80da7 in DataAccess::SavepointImpl::loadRestartPage(PageAccess::PageIO&, unsigned long, bool, bool)+0x693 at Page.hpp:277 (libhdbdataaccess.so)

3: 0x00007fb55ee0205f in DataAccess::PersistenceManagerImpl::prepareOpen(unsigned long, bool, bool)+0x4b at PersistenceManagerImpl.cpp:5385 (libhdbdataaccess.so)

4: 0x00007fb55ee0223c in DataAccess::PersistenceManager::open(ltt::refcounted_handle const&, bool)+0x88 at PersistenceManagerImpl.cpp:2461 ( libhdbdataaccess.so)

5: 0x00007fb56c510771 in PersistenceLayer::PersistenceSystem::initialize(NameServer::ServiceStartInfo const&, bool, PersistenceLayer::PERSISTENCE_MODE)+0x4a0 at PersistenceSystem.cpp:392 (libhdbpersistence.so)

6: 0x00007fb56c5437bc in PersistenceLayer::PersistenceFactory::initPersistence(PersistenceLayer::PERSISTENCE_MODE, ltt::releasable_handle &, DataAccess::TransactionCallback*, NameServer::ServiceStartInfo&, ltt::refcounted_handle &, bool)+0x158 at PersistenceFactory.cpp:394 ( libhdbpersistence.so)

7: 0x00007fb56c43375a in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, DataAccess::TablePreloadWriteCallback*, DataAccess::TablePreloadReadCallback*, Backup::RecoverCbc_Federation*)+0x1256 at PersistenceController.cpp:559 (libhdblogger.so)

8: 0x00007fb5751eedd5 in NameServer::Topology::initPersistence(NameServer::ServiceStartInfo&, bool, bool, TREX_ERROR::TRexError*, bool, ltt_adp::basic_string , ltt::integral_constant >, NameServer::ServiceStartInfo::RequestAction)+0x421 at Topology.cpp:285 ( libhdbns.so)

9: 0x00007fb5752fb0c4 in NameServer::TREXNameServer::loadTopology(NameServer::LoadTopologyMode, NameServer::ServiceStartInfo&, Backup::Backup_ExtendedRecoveryInformation*)+0x5b0 at TREXNameServer.cpp:18396 (libhdbns.so)

10: 0x00007fb575305c23 in NameServer::TREXNameServer::assign(NameServer::ServiceStartInfo&)+0x2790 at TREXNameServer.cpp:2123 (libhdbns.so)

11: 0x00007fb5762d4f17 in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0x93 at TREXIndexServer.cpp:992 (hdbnameserver)

12: 0x00007fb57630c129 in TRexAPI::AssignThread::run(void*)+0x35 at TREXIndexServer.cpp:543 (hdbnameserver)

13: 0x00007fb56cd03db5 in TrexThreads::PoolThread::run()+0x831 at PoolThread.cpp:389 (libhdbbasement.so)

14: 0x00007fb56cd056f0 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:165 (libhdbbasement.so)

15: 0x00007fb55471a9f0 in Execution::Thread::staticMainImp(void**)+0x700 at Thread.cpp:461 (libhdbbasis.so)

16: 0x00007fb55471bfc8 in Execution::Thread::staticMain(void*)+0x34 at ThreadMain.cpp:26 (libhdbbasis.so)

stopping service...

persistence initialization failed -> stopping instance ...

cannot send signal. (2, No such file or directory)

prepare for shutting service down...

stop ClockMonitor thread...

stop MasterTokenLockWriter /usr/sap/HDB/SYS/global//hdb/nameserver.lck thread...

setInactive(nameserver@hana001:30201)

hana001:/usr/sap/HDB/SYS/exe/hdb>

Also bellow HDB start log:

hana001:/usr/sap/HDB/HDB02> ./HDB start

StartService

Impromptu CCC initialization by 'rscpCInit'.

See SAP note 1266393.

OK

OK

Starting instance using: /usr/sap/HDB/SYS/exe/hdb/sapcontrol -prot NI_HTTP -nr 02 -function StartWait 2700 2

08.06.2016 03:44:24

Start

OK

08.06.2016 03:44:46

StartWait

FAIL: process hdbdaemon HDB Daemon not running

Final part from Daemon trace:


[17502]{-1}[-1/-1] 2016-06-08 03:44:28.545057 i Daemon TrexDaemon.cpp(03513) : stdout = /usr/sap/HDB/HDB02/hana001/trace/xsuaaserver.out [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545062 i Daemon TrexDaemon.cpp(03513) : stderr = [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545066 i Daemon TrexDaemon.cpp(03513) : maxstdfiles = 2 [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545071 i Daemon TrexDaemon.cpp(03513) : runlevel = 4 [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545075 i Daemon TrexDaemon.cpp(03513) : flags = [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545080 i Daemon TrexDaemon.cpp(03513) : window = [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545084 i Daemon TrexDaemon.cpp(03513) : isolation = 0 [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545088 i Daemon TrexDaemon.cpp(03513) : userId = 4294967295 [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545093 i Daemon TrexDaemon.cpp(03513) : groupId = 4294967295 [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545104 i Daemon Daemon.cpp(00666) : runlevel 0 completely started [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545115 i Daemon Program.cpp(00173) : line up of program group mdcdispatcher to instances <none>. [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545126 i Daemon Program.cpp(00173) : line up of program group nameserver to instances 0. [17502]{-1}[-1/-1] 2016-06-08 03:44:28.545460 i Daemon TrexDaemon.cpp(02078) : starting program at '/usr/sap/HDB/HDB02/exe/hdbnameserver' with args '' [17502]{-1}[-1/-1] 2016-06-08 03:44:28.569590 i Daemon RunningInstance.cpp(00191) : start 'hdbnameserver' as process 17518 [17502]{-1}[-1/-1] 2016-06-08 03:44:35.313937 i Daemon SignalsUNIX.cpp(00647) : signo 3 SIGQUIT from user errno 0 code 0 [17502]{-1}[-1/-1] 2016-06-08 03:44:35.314034 i Daemon SignalsUNIX.cpp(00647) : sender pid 17518 real user id 1000 executable '/hana/shared/HDB/exe/linuxx86_64/HDB_1.00.112.02.1459851171_2840326/hdbnameserver' [17502]{-1}[-1/-1] 2016-06-08 03:44:35.314167 i Daemon TrexDaemon.cpp(03558) : got shutdown event (stop) [17502]{-1}[-1/-1] 2016-06-08 03:44:35.314228 i Daemon Daemon.cpp(00752) : comment file contains: nameserver: persistence initialization failed [17502]{-1}[-1/-1] 2016-06-08 03:44:35.314836 i Daemon RunningInstance.cpp(00214) : stop process hdbnameserver with pid 17518 [17502]{-1}[-1/-1] 2016-06-08 03:44:35.315143 i Daemon TrexDaemon.cpp(02598) : stopped child with pid 17518 (17518) [17502]{-1}[-1/-1] 2016-06-08 03:44:35.401872 i Daemon TrexDaemon.cpp(03778) : process hdbnameserver with pid 17518 exited normally with status 1 [17502]{-1}[-1/-1] 2016-06-08 03:44:35.401939 i Daemon TrexDaemon.cpp(03850) : all instances in runlevel 1 stopped [17502]{-1}[-1/-1] 2016-06-08 03:44:35.401952 i Daemon Program.cpp(00173) : line up of program group dummyserver to instances <none>. [17502]{-1}[-1/-1] 2016-06-08 03:44:35.402492 i Daemon ProgramStarter.cpp(00229) : writing started programs file /usr/sap/HDB/HDB02/hana001/lock/started_programs.txt: [17502]{-1}[-1/-1] 2016-06-08 03:44:35.528876 i Daemon TrexDaemon.cpp(01616) : cleaning all [17502]{-1}[-1/-1] 2016-06-08 03:44:35.528931 i Daemon TrexDaemon.cpp(02681) : found 8 segments in use with maximum id 8 [17502]{-1}[-1/-1] 2016-06-08 03:44:35.529017 i Daemon TrexDaemon.cpp(02705) : segment 4, key 0x9f08cf81, id 3506180, sequence 107, owner 1000/1001, size 117, atime 2016-06-08 03:44:32, dtime 2016-06-08 03:44:35, ctime 2016-06-08 03:18:33, created by pid 17077, last changed by pid 17077, nattach 0 [17502]{-1}[-1/-1] 2016-06-08 03:44:35.529056 i Daemon TrexDaemon.cpp(02705) : segment 5, key 0x00000000, id 3538949, sequence 108, owner 1000/1001, size 1060911, atime 2016-06-08 03:44:35, dtime 2016-06-08 03:44:35, ctime 2016-06-08 03:18:33, created by pid 17077, last changed by pid 17077, nattach 1 (deleted) [17502]{-1}[-1/-1] 2016-06-08 03:44:35.529069 i Daemon TrexDaemon.cpp(02705) : segment 6, key 0x00000000, id 3571718, sequence 109, owner 1000/113, size 1024, atime 2016-06-08 03:44:26, dtime 2016-06-08 03:44:26, ctime 2016-06-08 03:44:25, created by pid 17494, last changed by pid 17494, nattach 1 [17502]{-1}[-1/-1] 2016-06-08 03:44:35.529157 i Daemon Main.cpp(01530) : exit hdbdaemon [17502]{-1}[-1/-1] 2016-06-08 03:44:35.529171 i Daemon Main.cpp(01541) : Success: /sapmnt/ld7272/a/HDB/jenkins_prod/workspace/FA_CO_LIN64GCC48HAPPY_rel_fa~newdb100_rel/sys/src/TrexDaemon/Main.cpp:1541 in "int main(int, char**)". Leaving the "main" [17502]{-1}[-1/-1] 2016-06-08 03:44:35.530121 i Network NetworkListener.cpp(00805) : closing listen socket 7 bound to 127.0.0.1 [17502]{-1}[-1/-1] 2016-06-08 03:44:35.530283 i Network NetworkListener.cpp(00805) : closing listen socket 8 bound to 127.0.0.2