on 07-19-2013 1:38 PM
Hi,
I'm having problems with one of the backup systems. The backup of the datafiles is done successfully, but an error occurs during "RMAN Catalog".
I tried to find solutions on topics already open with similar errors but none helped me. The error appears to be a problem with permissions, but I moved and do not know what else to do.
Environment:
Red Hat Enterprise Linux 5.4
Oracle 11.2.0.3.0
System ID: BID
Backup directory: /backup/BID
Below the end of the log that contains the error:
BR0203I to /backup/BID/belrcwuf/system.data1.Z ...
#FILE..... /oracle/BID/sapdata1/system_1/system.data1
#SAVED.... /backup/BID/belrcwuf/system.data1.Z #1/10 5.2850:1 117062058
BR0280I BRBACKUP time stamp: 2013-07-18 17.23.06
BR0063I 10 of 10 files processed - 37550.078 of 37550.078 MB done
BR0204I Percentage done: 100.00%, estimated end time: 17:23
BR0001I **************************************************
BR0280I BRBACKUP time stamp: 2013-07-18 17.23.06
BR0317I 'Alter tablespace SYSTEM end backup' successful
BR0280I BRBACKUP time stamp: 2013-07-18 17.23.08
BR0530I Cataloging backups of all database files...
BR0278E Command output of 'SHELL=/bin/sh /oracle/BID/112_64/bin/rman nocatalog':
Recovery Manager: Release 11.2.0.3.0 - Production on Thu Jul 18 17:23:09 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
RMAN>
RMAN> connect target *
connected to target database: BID (DBID=3506605042)
using target database control file instead of recovery catalog
RMAN> **end-of-file**
RMAN>
host command complete
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12>
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of catalog command on default channel at 07/18/2013 17:23:11
ORA-19564: error occurred writing 8192 bytes at block number 1
ORA-27091: unable to queue I/O
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
RMAN>
Recovery Manager complete.
BR0200I BR_TRACE: location BrRmanCall-44, commands for RMAN in: /oracle/BID/sapbackup/.belrcwuf.cmd
'@/oracle/BID/sapbackup/..belrcwuf..cmd^'
'host '/usr/sap/BID/SYS/exe/run/brtools -f delete /oracle/BID/sapbackup/..belrcwuf..cmd';^'
'catalog datafilecopy^'
''/oracle/BID/sapbackup/belrcwuf.spa/sr3.data1',^'
''/oracle/BID/sapbackup/belrcwuf.spa/sr3.data2',^'
''/oracle/BID/sapbackup/belrcwuf.spa/sr3.data3',^'
''/oracle/BID/sapbackup/belrcwuf.spa/sr3702.data1',^'
''/oracle/BID/sapbackup/belrcwuf.spa/sr3702.data2',^'
''/oracle/BID/sapbackup/belrcwuf.spa/sr3702.data3',^'
''/oracle/BID/sapbackup/belrcwuf.spa/sr3usr.data1',^'
''/oracle/BID/sapbackup/belrcwuf.spa/undo.data1',^'
''/oracle/BID/sapbackup/belrcwuf.spa/sysaux.data1',^'
''/oracle/BID/sapbackup/belrcwuf.spa/system.data1'^'
'level 0 tag belrcwuf;^'
'exit;^'
BR0280I BRBACKUP time stamp: 2013-07-18 17.23.11
BR0279E Return code from 'SHELL=/bin/sh /oracle/BID/112_64/bin/rman nocatalog': 1
BR0536E RMAN call for database instance BID failed
BR0280I BRBACKUP time stamp: 2013-07-18 17.23.11
BR0532E Cataloging backups of all database files failed
BR0115I Compression rate for all files 4.4742:1
BR0056I End of database backup: belrcwuf.fnd 2013-07-18 17.23.11
BR0280I BRBACKUP time stamp: 2013-07-18 17.23.11
BR0054I BRBACKUP terminated with errors
BR0280I BRBACKUP time stamp: 2013-07-18 17.23.11
BR0291I BRARCHIVE will be started with options '+belrcwuf.fnd -U -jid FLLOG20130718165621 -c force -p initBID.sap -sd'
BR0280I BRBACKUP time stamp: 2013-07-18 17.23.25
BR0292I Execution of BRARCHIVE finished with return code 0
What seems strange is to see log files in the directory /oracle/BID/sapbackup because we do not use it for backup.
Attached is the configuration file initBID.sap
Thanks
Gilson
Hi Gilson,
>> RMAN-03009: failure of catalog command on default channel at 07/18/2013 17:23:11
>> ORA-19564: error occurred writing 8192 bytes at block number 1
>> ORA-27091: unable to queue I/O
>> ORA-27041: unable to open file
Check the permission on the path "/backup/BID"
Best regards,
Orkun Gedik
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Hi again,
Try to execute the "brbackup" with "truss", in order to identify on what file has been reading/writing at the end of the backup operation.
http://www.unix.com/man-page/FreeBSD/1/truss/
It will create a huge output, but you will be able to find the cause.
Best regards,
Orkun Gedik
Hi Orkun,
Thanks the help. These tool is very powerfull (I used strace that is default on my system)
I found 2 strange lines:
("/oracle/BID/sapbackup/..belrgvmi..cmd") = -1 ENOENT (No such file or directory)
The brbackup is trying to access the backup files on incorrect folder
open("/usr/lib64/libnuma.so", O_RDONLY) = -1 ENOENT (No such file or directory)
This can to be the my root problem?
KR
Gilson
Hi Gilson,
that tool is only useful, if you know in which sequence and what files / memory, etc. should be accessed (or if you want to take a look into some internals). Unfortunately the BR*Tools are doing a lot of stuff under the hood implicitly.
> ("/oracle/BID/sapbackup/..belrgvmi..cmd") = -1 ENOENT (No such file or directory)
> The brbackup is trying to access the backup files on incorrect folder
Maybe the BR*Tools are just checking for that file at first and create it afterwards (e.g. it contains the RMAN commands, etc.). Check for a successfull fopen() and the corresponding file handle afterwards. But you still don't know how the BR*Tools generate that path.
> open("/usr/lib64/libnuma.so", O_RDONLY) = -1 ENOENT (No such file or directory)
Nothing unusual. Searching the paths for libraries is the usual way and so you will find such messages.
Run brbackup with the corresponding trace level like mentioned above to get insights into it.
Regards
Stefan
Hi Orkun,
> I believe that the problem is caused by this file or something inside of the file.
The issue is caused by RMAN and cataloging / tagging the datafile copy (the file is just the command file for RMAN). If you would run a strace on brbackup (without following the forks) you would see nothing here related to this issue. However running strace over several forks is very confusing and you won't get the puzzle parts together easily.
Regards
Stefan
Thanks Orkun Gedik!
After update the BRTools and checked permissions the backup works fine.
Kind regards
Gilson
Hi Gilson,
> The error appears to be a problem with permissions
No it is not. You perform an old-fashion OS copy based database backup ("Alter tablespace SYSTEM end backup") and brbackup tries to catalog these data file copies (in the control file) afterwards.
The problem is that it tries to catalog a data file copy called ''/oracle/BID/sapbackup/belrcwuf.spa/system.data1", but your copy was created as "/backup/BID/belrcwuf/system.data1.Z" for example. This is the root cause of the ORA-27041 and all of its following errors.
Can you please attach the corresponding backup log file, so that we can see all of the cmd line parameters as well.
Thanks.
Regards
Stefan
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
As mentioned by Stefan you use backup approach without RMAN plus compression and specified the mode FULL for backup. As mentioned in online doc about this BRBACKUP mode:
Full backup
Backs up all database files. You can perform a full backup with or without RMAN. If you do not use RMAN, then RMAN is called separately to catalog the backup as level 0. This means that you can use this backup as a reference backup for an incremental backup with RMAN.
Try to change mode from FULL to ALL or make backup with mode FULL but without COMPRESSION.
Regards,
Vladimir
Hi Gilson,
thanks for the backup log file.
Can you please run a backup with BR_TRACE 15 and upload that log file once again. It seems like SAP uses backup_root_dir only as a backup location in your scenario and generates the RMAN command file with the usual "$SAPDATA_HOME/sapbackup" path (e.g. the generated control file copy).
Your BR*Tools are also very old (7.20 PL 8 from 2010-10-06) - PL 33 is the current one, but we should get the root cause for that wrong command file generation / the issue first.
You can also try to verify the registered data file copies with that query:
shell> sqlplus / as sysdba
SQL> select NAME from V$DATAFILE_COPY;
Regards
Stefan
P.S.: I know why i don't like the BR*Tools for backup and recovery ... Native RMAN is much more straight forward without all of that unnecessary "under-the-hood" stuff :-))
Hi Gilson,
I would try to perform backup with mode ALL. In that mode BR*Tools don't use RMAN.
If it completes successfully than it is a problem with call of RMAN and most probably the permissions of user which is owner of oracle database process (ora<SID>) to directory /backup/BID.
Regards,
Vladimir
Hi Gilson,
thanks for that trace. Unfortunately not all OS commands are included, but the RMAN "catalog datafilecopy" makes currently no sense at all in your environment here.
Please crosscheck, if SAP creates symlinks in /oracle/BID/sapbackup/<sub_folder>/ to your compressed datafile copies or holds some other datafile copies there as well. If there would be no corresponding file (link) in /oracle/BID/sapbackup/<sub_folder>/ - the RMAN catalog command would fail with "ORA-27037: unable to obtain file status and Linux-x86_64 Error: 2: No such file or directory".
I was able to reproduce your ORA error in my environment as well, when i try to update the TAG in the header of a valid (read only) datafile copy.
T11DB [oracle@OEL oradata]$ ls -la users01.bkp.dbf
-r--r----- 1 oracle dba 68165632 Jul 22 15:56 users01.bkp.dbf
RMAN> catalog datafilecopy '/oracle/T11DB/oradata/users01.bkp.dbf' level 0 tag asgge;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of catalog command on default channel at 07/22/2013 16:13:33
ORA-19564: error occurred writing 8192 bytes at block number 1
ORA-27091: unable to queue I/O
ORA-27041: unable to open file
Linux-x86_64 Error: 13: Permission denied
Additional information: 3
The other thing is that you don't hold such a data file copy in your file system, because of you compress them by copy - just something like that
BR0201I Compressing /oracle/BID/sapdata2/sr3_1/sr3.data1
BR0203I to /backup/BID/belruyja/sr3.data1.Z ...
BR0248I BR_TRACE: level 4, function BrCopyBuild entry with '20 2 6228549632 /oracle/BID/sapdata2/sr3_1/sr3.data1 /backup/BID/belruyja/sr3.data1.Z'
BR0250I BR_TRACE: level 4, function BrCopyBuild exit with 'LANG=C gzip -c /oracle/BID/sapdata2/sr3_1/sr3.data1 > /backup/BID/belruyja/sr3.data1.Z'
.. and so you normally get an invalid header error by RMAN, if you would try to catalog and update the tag of that compressed datafile copy.
RMAN> catalog datafilecopy '/oracle/T11DB/oradata/users01.bkp.dbf.Z' level 0 tag dsgdsag;
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of catalog command on default channel at 07/22/2013 16:15:58
ORA-19625: error identifying file /oracle/T11DB/oradata/users01.bkp.dbf.Z
ORA-27048: skgfifi: file header information is invalid
Additional information: 9
So the question is .. what is located (and how) in /oracle/BID/sapbackup/<sub_folder>/ while running the corresponding backup, because of it is deleted successfully afterwards as well:
…
BR0248I BR_TRACE: level 2, function BrFileDelete entry with '/oracle/BID/sapbackup/belruyja.spa/sr3.data1'
BR0249I BR_TRACE: level 2, function BrFileDelete exit with 0
…
Please run the backup once again and perform the following command several times while the backup is running
shell> ls -lad /oracle/BID/sapbackup/
shell> ls -la /oracle/BID/sapbackup/*/*
Regards
Stefan
Hi Roman,
that is the strange one. In "/backup/BID/belrcwuf/" (for example) are stored the compressed (gzip) data file copies only. You get an error like "ORA-27048: skgfifi: file header information is invalid", if you would try to catalog such compressed data file copies (which is correct of course). Brbackup generates a RMAN file and tries to catalog the data file copies in "/oracle/BID/sapbackup/<sub_folder>" with an error like "Linux-x86_64 Error: 13: Permission denied", but there are usually no copies in his scenario (maybe some links?).
Regards
Stefan
Hi Stefan
I updated the brtools, now when I run the brbackup is a different error. I ran using the user ORABID and the backup finished successfully, including the RMAN Catalog. Now it seems some other permission error:
BR0278E Command output of 'SHELL=/bin/sh /oracle/BID/112_64/bin/rman nocatalog':
Recovery Manager: Release 11.2.0.3.0 - Production on Mon Jul 22 15:20:22 2013
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
RMAN>
RMAN> connect target *
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
ORA-01031: insufficient privileges
RMAN> **end-of-file**
RMAN>
host command complete
RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12>
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of catalog command at 07/22/2013 15:20:23
RMAN-06171: not connected to target database
RMAN>
Recovery Manager complete.
BR0280I BRBACKUP time stamp: 2013-07-22 15.20.23
BR0279E Return code from 'SHELL=/bin/sh /oracle/BID/112_64/bin/rman nocatalog': 1
BR0536E RMAN call for database instance BID failed
BR0200I BR_TRACE: location BrRmanCall-56, commands for RMAN in: /oracle/BID/sapbackup/.belrwddb.cmd
@/oracle/BID/sapbackup/..belrwddb..cmd
host '/usr/sap/BID/SYS/exe/run/brtools -f delete /oracle/BID/sapbackup/..belrwddb..cmd';
catalog datafilecopy
'/oracle/BID/sapbackup/belrwddb.spa/sr3.data1',
'/oracle/BID/sapbackup/belrwddb.spa/sr3.data2',
'/oracle/BID/sapbackup/belrwddb.spa/sr3.data3',
'/oracle/BID/sapbackup/belrwddb.spa/sr3usr.data1',
'/oracle/BID/sapbackup/belrwddb.spa/undo.data1',
'/oracle/BID/sapbackup/belrwddb.spa/sysaux.data1',
'/oracle/BID/sapbackup/belrwddb.spa/system.data1'
level 0 tag belrwddb;
exit;
BR0280I BRBACKUP time stamp: 2013-07-22 15.20.23
BR0532E Cataloging backups of all database files failed
BR0115I Compression rate for all files 4.4774:1
BR0056I End of database backup: belrwddb.fnd 2013-07-22 15.20.23
BR0280I BRBACKUP time stamp: 2013-07-22 15.20.23
BR0054I BRBACKUP terminated with errors
This is the output from the commands:
[root@dt121215 ~]# ls -lad /oracle/BID/sapbackup/
drwxrwxr-x 3 orabid sapsys 16384 Jul 22 14:53 /oracle/BID/sapbackup/
[root@dt121215 ~]# ls -la /oracle/BID/sapbackup/*/*
-rw-r--r-- 1 orabid sapsys 6228549632 Jul 22 14:53 /oracle/BID/sapbackup/belrwddb.spa/sr3.data1
-rw-r--r-- 1 orabid sapsys 6018834432 Jul 22 14:57 /oracle/BID/sapbackup/belrwddb.spa/sr3.data2
Kind Regards,
Gilson
Hi Stefan,
brbackup create sparse datafile copy in /oracle/BID/sapbackup/<sub_folder> (the same size as original file and copy only few starting blocks from it - it seems to be 4 blocks) to catalog it with RMAN successfully.
From log file:
BR0250I BR_TRACE: level 3, function BrCurrSizeGet exit with '6228549632'
BR0248I BR_TRACE: level 3, function BrSparseCreate entry with '/oracle/BID/sapdata2/sr3_1/sr3.data1'
BR0248I BR_TRACE: level 4, function BrNameBuild entry with '1 /oracle/BID/sapdata2/sr3_1/sr3.data1 /oracle/BID/sapbackup/belruyja.spa'
BR0250I BR_TRACE: level 4, function BrNameBuild exit with '/oracle/BID/sapbackup/belruyja.spa/sr3.data1'
BR0249I BR_TRACE: level 3, function BrSparseCreate exit with 0
You must to have write permissions to this file to change the TAG in the header (as you wrote above). This file can be safely deleted after RMAN catalog operation completed.
I don't claim 100% accuracy (I'm not developer of brtools).
Regards,
Roman
Check owners/permissions of brbackup executable.
Note 776505 - BR*Tools fail with ORA-01017 / ORA-01031 on Linux
Hi Roman,
thanks for clarification and this trace file snippet. I overlooked this and was not aware of that handling. It is absolutely crazy how SAP works around some real simple RMAN core functionality just for using that BR*Tools stuff. Just think about cross-checking these files afterwards and you have some nice inconsistency in the RMAN repository ... not quite sure why SAP adds this stuff to the RMAN (control file) repository at all, because of the compressed datafile copies need to be uncompressed and copied to the sapbackup folder anyway for using some RMAN functionality then
Thanks again.
Regards
Stefan
Hi Gilson,
please follow the note that Roman mentioned.
Oracle RMAN needs SYSDBA (and OSDBA = dba group by default) privileges by connecting this way (until / including 11gR2) and the S-bit (and ora<sid> owner) of the BR*Tools binaries is needed for that. The permission of the datafile copies in "/oracle/BID/sapbackup/belrwddb.spa/*" is correct now and you should be able to catalog these files afterwards.
Regards
Stefan
Hi all,
Problem solved!
Many thanks to everyone who helped me with this!
Summarizing the solution for those who can see the topic:
-Updated the version of BRTools
-Implemented permissions according to note 776505 - BR * Tools fail with ORA-01017 / ORA-01031 on Linux
'll Mark answers as correct and useful now. Thank you again.
Kind Regards
Gilson
BR0280I BRBACKUP time stamp: 2013-07-23 09.16.07
BR0063I 10 of 10 files processed - 37550.078 of 37550.078 MB done
BR0204I Percentage done: 100.00%, estimated end time: 9:16
BR0001I **************************************************
BR0280I BRBACKUP time stamp: 2013-07-23 09.16.07
BR0317I 'Alter tablespace SYSTEM end backup' successful
BR0280I BRBACKUP time stamp: 2013-07-23 09.16.10
BR0530I Cataloging backups of all database files...
BR0522I 10 of 10 files / save sets processed by RMAN
BR0280I BRBACKUP time stamp: 2013-07-23 09.16.15
BR0531I Backups of all database files cataloged successfully
BR0280I BRBACKUP time stamp: 2013-07-23 09.16.15
BR0340I Switching to the next online redolog file for database instance BID ...
BR0321I Switch to the next online redolog file for database instance BID succes
BR0280I BRBACKUP time stamp: 2013-07-23 09.16.17
BR0319I Control file copy created: /oracle/BID/sapbackup/cntrlBID.dbf 20496384
BR0117I ARCHIVE LOG LIST after backup for database instance BID
Hi Gilson
Please check the permissions of oracle file
SOURCS : http://d-h-n.de/blog/ora-15025-ora-27041-and-setasmgidwrap
Solution:
Change owner of binary $ORACLE_HOME/bin/oracle (of user oracle):
Before:
[grid@oel62 ~]$ ls -l /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle -rwsr-s--x 1 oracle oinstall 232399123 Jun 5 17:18 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
Run:
[grid@oel62 ~]/u01/app/grid/product/11.2.0/grid/bin/setasmgidwrap o=/u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
After:
[grid@oel62 ~]$ ls -l /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle -rwsr-s--x 1 oracle asmadmin 232399123 Jun 5 17:18 /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
Let me knof if this does not help
Best Regards
Marius
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
User | Count |
---|---|
86 | |
23 | |
11 | |
9 | |
8 | |
5 | |
5 | |
5 | |
5 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.