DB2v9.7 FP1 installfixpack problem

[Former Member]

Hello all,

we've got troubles with installation of fixpack on DB2 9.7 on SuSE 11

- installfixpack done successfully;

- when I call db2start I receive

06/21/2010 15:05:06 0 0 SQL1042C An unexpected system error occurred.

SQL1032N No start database manager command was issued. SQLSTATE=57019

so, we can't start db6_update_db.sh (application is SAP)

files in /db2/db2sid/sqllib/adm are not updated and are differ from /db2/db2sid/db2_software/adm

Regards,

Konstantin

[Former Member]

No ideas?

Have anybody successfully completed installation of FP1 on DB2v9.7 on SuSE11?

We made it on SuSE10 and had no problems.

Regards,

Konstantin

[Former Member]

addition:

2010-06-22-16.44.47.785467+240 I79671E1007 LEVEL: Error

PID : 9172 TID : 140313949632256PROC : db2fm

INSTANCE: db2opq NODE : 000

FUNCTION: DB2 Common, Generic Control Facility, gcf_stop, probe:30

MESSAGE : ECF=0x9000036D=-1879047315=ECF_FM_DB2FMD_PROCESS_NOT_EXIST

There is no fault monitor daemon running

CALLED : OS, -, open

RETCODE : ECF=0x9000001A=-1879048166=ECF_FILE_DOESNT_EXIST

File doesn't exist

CALLSTCK:

[0] 0x00007F9D5FBC81BD pdOSSeLoggingCallback + 0x10D

[1] 0x00007F9D62B80064 /db2/db2opq/db2_software/lib64/libdb2osse.so.1 + 0x1C1064

[2] 0x00007F9D62B81725 ossLogRC + 0x6B

[3] 0x00007F9D5DD61D8A gcf_stop + 0x410

[4] 0x00007F9D62578051 ZN9GcfCaller4stopEP12GCFPartInfomP11GCF_RetInfo + 0x1B9

[5] 0x0000000000406911 main + 0x25E9

[6] 0x00007F9D61ABF586 __libc_start_main + 0xE6

[7] 0x0000000000404299 __gxx_personality_v0 + 0x149

[8] 0x0000000000000000 ?unknown + 0x0

[9] 0x0000000000000000 ?unknown + 0x0

2010-06-22-16.44.47.786111+240 I80679E973 LEVEL: Error

PID : 9172 TID : 140313949632256PROC : db2fm

INSTANCE: db2opq NODE : 000

FUNCTION: DB2 Common, Fault Monitor Facility, db2fm, probe:170

MESSAGE : ECF=0x90000349=-1879047351=ECF_FM_FAIL_TO_STOP_GCF_FM

Failed to stop the GCF fm module

CALLED : DB2 Common, Generic Control Facility, GcfCaller::stop

DATA #1 : signed integer, 8 bytes

0

DATA #2 : unsigned integer, 8 bytes

1

CALLSTCK:

[0] 0x00007F9D5FBC81BD pdOSSeLoggingCallback + 0x10D

[1] 0x00007F9D62B80064 /db2/db2opq/db2_software/lib64/libdb2osse.so.1 + 0x1C1064

[2] 0x00007F9D62B81779 ossLogRC + 0xBF

[3] 0x00000000004069A4 main + 0x267C

[4] 0x00007F9D61ABF586 __libc_start_main + 0xE6

[5] 0x0000000000404299 __gxx_personality_v0 + 0x149

[6] 0x0000000000000000 ?unknown + 0x0

[7] 0x0000000000000000 ?unknown + 0x0

[8] 0x0000000000000000 ?unknown + 0x0

[9] 0x0000000000000000 ?unknown + 0x0

it is an output of db2diag.log after command

./db2fm -i db2opq -D

Regards,

Konstantin

[Former Member]

Hello Paul,

as I said earlier - before calling db2iupdt I restarted server; and there is no DB2AUTOSTART option; and FMC is down

55> ./db2fmcu

Warning: db2fmcu not running as root.

FMC: down

so, there can't be any db2 process!

Regards,

Konstantin

[Former Member]

Hello,

here are logs from db2diag.log:

2010-06-21-13.37.49.341568+240 I77066E324 LEVEL: Error

PID : 6556 TID : 139801863501568PROC : db2acd 0

INSTANCE: db2opq NODE : 000

FUNCTION: DB2 UDB, common communication, sqlccipcdarihandshake, probe:4

RETCODE : ZRC=0x800F00FC=-2146500356=SQLO_INV_PARM "Invalid input parameter"

2010-06-21-14.05.39.976684+240 I77689E1007 LEVEL: Error

PID : 7849 TID : 139801901979392PROC : db2fm

INSTANCE: db2opq NODE : 000

FUNCTION: DB2 Common, Generic Control Facility, gcf_stop, probe:30

MESSAGE : ECF=0x9000036D=-1879047315=ECF_FM_DB2FMD_PROCESS_NOT_EXIST

There is no fault monitor daemon running

CALLED : OS, -, open

RETCODE : ECF=0x9000001A=-1879048166=ECF_FILE_DOESNT_EXIST

File doesn't exist

CALLSTCK:

[0] 0x00007F26275DEB4C pdOSSeLoggingCallback + 0x100

[1] 0x00007F262A4DD33C /db2/db2opq/db2_software/lib64/libdb2osse.so.1 + 0x1B233C

[2] 0x00007F262A4DE9FD ossLogRC + 0x6B

[3] 0x00007F2625BC5D0A gcf_stop + 0x410

[4] 0x00007F2629EE4FD1 ZN9GcfCaller4stopEP12GCFPartInfomP11GCF_RetInfo + 0x1B9

[5] 0x00000000004067E1 main + 0x25E9

[6] 0x00007F262942C586 __libc_start_main + 0xE6

[7] 0x0000000000404169 __gxx_personality_v0 + 0x149

[8] 0x0000000000000000 ?unknown + 0x0

[9] 0x0000000000000000 ?unknown + 0x0

2010-06-21-14.05.39.977361+240 I78697E973 LEVEL: Error

PID : 7849 TID : 139801901979392PROC : db2fm

INSTANCE: db2opq NODE : 000

FUNCTION: DB2 Common, Fault Monitor Facility, db2fm, probe:170

MESSAGE : ECF=0x90000349=-1879047351=ECF_FM_FAIL_TO_STOP_GCF_FM

Failed to stop the GCF fm module

CALLED : DB2 Common, Generic Control Facility, GcfCaller::stop

DATA #1 : signed integer, 8 bytes

0

DATA #2 : unsigned integer, 8 bytes

1

CALLSTCK:

[0] 0x00007F26275DEB4C pdOSSeLoggingCallback + 0x100

[1] 0x00007F262A4DD33C /db2/db2opq/db2_software/lib64/libdb2osse.so.1 + 0x1B233C

[2] 0x00007F262A4DEA51 ossLogRC + 0xBF

[3] 0x0000000000406874 main + 0x267C

[4] 0x00007F262942C586 __libc_start_main + 0xE6

[5] 0x0000000000404169 __gxx_personality_v0 + 0x149

[6] 0x0000000000000000 ?unknown + 0x0

[7] 0x0000000000000000 ?unknown + 0x0

[8] 0x0000000000000000 ?unknown + 0x0

[9] 0x0000000000000000 ?unknown + 0x0

To all questions about stopping db2 and other: I rebooted server and there are no processes and others in memory by db2.

Regards,

Konstantin

[paul_power]

Hello Konstantin,

Those db2diag.log entries really point to the fault monitor not being stopped which could explain the instance update issue.

If you are using a high availability clustering product such as HACMP,

MSCS, or IBM Tivoli System Automation for Multiplatforms, the fault

monitor facility must be turned off since the instance startup and shut

down is controlled by the clustering product.

To confirm that fault monitor is no longer running, please type the

following command:

ps -ef grep -i fm

An entry that shows db2fmd and DB2<sid> indicates that the fault monitor

is still running on that instance. To turn off the fault monitor, type

the following command as the instance owner:

db2fm -i db2inst1 -D

Disable the fault monitor processes.

To stop the Fault Monitor Daemon, issue the command: DB2DIR/bin/db2fm -i iname -D where DB2DIR is the location where the DB2 copy is installed and iname represents the instance owner name.

The command must be performed once for each instance in the DB2 copy. If the Fault Monitor Coordinator (FMC) is enabled, prevent your instances from auto-starting: To determine whether the FMC is enabled, issue the command: DB2DIR/bin/db2fmcu where DB2DIR is the location where the DB2 copy is installed. If the FMC is enabled, you will see output similar to the following:FMC: up: PID = 3415 .

If the FMC is disabled, the output from the db2fmcu command will be: FMC: down. If the FMC is enabled, determine whether any instances are configured to auto-start after each system restart. Issue the command: DB2DIR/instance/db2iset -i iname -all where DB2DIR is the location where the DB2 copy is installed and iname represents the instance owner name. The command must be performed once for each instance in the DB2 copy.

If the output from the db2set command includes the following, it means that the instance is configured to auto-start:DB2AUTOSTART=YES Prevent the instances from auto-starting. Issue the command: DB2DIR/instance/db2iauto -off iname where DB2DIR is the location where the DB2 copy is installed and iname represents the instance owner name. If desired, re-enable instance auto-start after you have completed the fix pack installation: DB2DIR/instance/db2iauto -on iname If there are any db2 related ipcs resources these need to be removed manualy with ipcrm command.

Regards,

Paul

[Former Member]

Yes, of course

1. ps -eaf |grep db2

root 7039 3401 0 10:21 pts/0 00:00:00 grep db2

Regards,

Konstantin

[Former Member]

Hello Konstantin,

any errors withing the db2diag.log.

DBI1122E states that there are Db2 processes running. Whaht about db2as (Administration Server)?

Deprecated with DB2 V9.7

Best regards,

Joachim

[paul_power]

Hi Konstantin,

Lets focus on seeing if something related to the db is remaining active since the database appears to be cataloged fine.

Apologies if you have done some of these already but lets be thorough to make sure we find whats blocking the instance update:

can you check if any DB2-Process is running ?

ps -ef grep db2

Maybe, there is some DB2-related servive started as system startup. Can

you check your /etc/inittab ?

If this is the problem, it should help to comment out the related line

in the /etc/inittab and reboot the box.

To stop all DB2 processes, perform the following steps:

Log on as root.

Determine which instances are associated with the DB2 copy. Issue the

command:

DB2DIR/instance/db2ilist

where DB2DIR represents the location where the DB2 copy is installed.

The default installation path for UNIX® is /opt/IBM/db2/V9.5.

Run the following commands for each instance in the DB2 copy:

su - db2

. $HOME/sqllib/db2profile

db2 force applications all

db2 terminate

db2stop

db2licd -end # run at each physical partition

exit

If you are an HACMP# user, you must use the ha_db2stop command to stop

DB2 instead of the db2stop command. If you use the db2stop command

instead of the ha_db2stop command, you will trigger a failure event.

If the DB2 Administration Server (DAS) belongs to the DB2 copy that you

are updating, stop the DAS:

su - aname

. $HOME/das/dasprofile

db2admin stop

exit

where aname represents the DAS owner name.

Note: Since there can only be one DAS on the system, this step affects

all other DB2 copies on the system.

Ensure all DB2 interprocess communications are cleaned for the instance

to be updated. As the instance owner, run the following command at each

physical partition:

$HOME/sqllib/bin/ipclean

Check for ipcs processes by running ipcs grep db2

Run slibclean to unload unused shared libraries from memory before

installation:

/usr/sbin/slibclean

Disable the fault monitor processes. To stop the Fault Monitor Daemon,

issue the command:

DB2DIR/bin/db2fm -i iname -D

where DB2DIR is the location where the DB2 copy is installed and iname

represents the instance owner name. The command must be performed once

for each instance in the DB2 copy.

If the Fault Monitor Coordinator (FMC) is enabled, prevent your

instances from auto-starting:

To determine whether the FMC is enabled, issue the command:

DB2DIR/bin/db2fmcu

where DB2DIR is the location where the DB2 copy is installed. If the FMC

is enabled, you will see output similar to the following:FMC: up: PID =

3415 . If the FMC is disabled, the output from the db2fmcu command will

be: FMC: down.

If the FMC is enabled, determine whether any instances are configured to

auto-start after each system restart. Issue the command:

DB2DIR/instance/db2iset -i iname -all

where DB2DIR is the location where the DB2 copy is installed and iname

represents the instance owner name. The command must be performed once

for each instance in the DB2 copy. If the output from the db2set command

includes the following, it means that the instance is configured to

auto-start:DB2AUTOSTART=YES

Prevent the instances from auto-starting. Issue the command:

DB2DIR/instance/db2iauto -off iname

where DB2DIR is the location where the DB2 copy is installed and iname

represents the instance owner name. If desired, re-enable instance

auto-start after you have completed the fix pack installation:

DB2DIR/instance/db2iauto -on iname

If there are any db2 related ipcs resources these need to be removed

manualy with ipcrm command.

If all the above checks out then we can be sure that the fail isnt due to something related to the instance still being active.

If you still encounter problems after running the commands above you

might want to install the image into a new path, then use db2iupdt to

update the instance to the new code level. Another option might be to

reboot the AIX server and restart the fixpak install.

Regards,

Paul

[Frank-Martin]

Hello Konstantin,

DB2 may not have been stopped cleanly. Maybe there are some leftover IPC segments. Try "ipcs | grep db2" . If those exist you can try to remove them using "db2_kill" or manuel "ipcrm" . An alternative would be to restart your DB server.

After this you can try "db2start" again.

Regards

Frank

[Former Member]

db2opq 51> db2 "list db directory"

System Database Directory

Number of entries in the directory = 1

Database 1 entry:

Database alias = OPQ

Database name = OPQ

Local database directory = /db2/OPQ

Database release level = d.00

Comment = SAP database OPQ

Directory entry type = Indirect

Catalog database partition number = 0

Alternate server hostname =

Alternate server port number =

db2opq 52> db2 "list db directory on /db2/db2opq"

SQL1031N The database directory cannot be found on the indicated file system.

SQLSTATE=58031

db2opq 53> db2 "list db directory on /db2/

list db directory on /db2/ not found

db2opq 53> db2 "list db directory on /db2/

list db directory on /db2/ not found

db2opq 53> db2 "list db directory on /db2/OPQ"

Local Database Directory on /db2/OPQ

Number of entries in the directory = 1

Database 1 entry:

Database alias = OPQ

Database name = OPQ

Database directory = SQL00001

Database release level = d.00

Comment = SAP database OPQ

Directory entry type = Home

Catalog database partition number = 0

Database partition number = 0

db2opq 54> db2 "list node directory"

SQL1027N The node directory cannot be found.

Joachim,

I rebooted server and tried to call db2iupdt -- I received the same message and log I post earlier

[Former Member]

Hello Konstantin,

do you really sure that every db2 process is stopped:

ps -eaf|grep -i db2

Please send us the output.

Best regards,

Joachim

[paul_power]

Hello Konstantin,

Please check if the database is cataloged using the following

commands: (Please issue as db2<db2sid>)

db2 "list db directory"

db2 "list db directory on <db_path>"

db2 "list node directory"

Can you uncatalog and catalog the DB again.

Then once again an instance update (db2iupdt).

Regards,

Paul

[Former Member]

Updating DB2 file sets :.......Success

Setting DB2 library path :.......Success

Executing control tasks :.......Success

Updating global registry :.......Success

Starting DB2 Fault Monitor :.......Success

Updating the db2ls link :.......Success

Updating existing DB2 instances :.......Success

and log of db2iupdt after following command from user root

/db2/db2opq/db2_software/instance/db2iupdt db2opq

cat /tmp/db2iupdt.log.29709

DBI1122E Instance db2opq cannot be updated or upgraded.

Explanation:

An attempt was made to either update or upgrade an instance. This

instance cannot be updated or upgraded because:

• One of the following commands cannot be used to either update or

upgrade this instance: db2iupdt, db2iupgrade, or db2nrupdt.

• The instance is still active.

[Former Member]

Hello Konstantin,

the instance is still active. Please stop all db2 processes and then do a db2iupdt again.

Regards,

Joachim

[Former Member]

Hello Konstantin,

in AIX we will see a file like /tmp/installFixPack.log.639188.

The instance have to be updated:

Installing or updating DB2 HA scripts for SA MP :.......Success

Setting DB2 library path :.......Success

Executing control tasks :.......Success

Updating global registry :.......Success

Starting DB2 Fault Monitor :.......Success

Updating the db2ls link :.......Success

The instance "db2bet" has been updated successfully.

Most I found that the instance isn't updated, because of a service address within the db2nodes.cfg file.

Then you can update manually: db2iupdt <db2instance>.

Perhaps this will help.

Best regards,

Joachim