Adaptive Job Server - Schedules stuck at pending

[Former Member]

Hi,

Server environment:

AIX64 6.1 TL9

SAP BO BI Platform 4.1 SP8 P1 Enterprise - migrated data from 3.1 system

SAP Data Services 4.2 SP7 - migrated data from 3.2 system

Tomcat 7 - BI webapps

Oracle 12c - DWH, BO_REPO, DS_REPO, BO_AUDIT

When i schedule anything it is stuck on pending, webi report, search index schedule or monitoring probes.

During the pending state Adaprive Job Server produces a dump log every few seconds until I pause or delete scheduled instance.

The error I get in the log dump is SIGSEGV ... accessing faulting address ... Problematic frame:# C  [libTraceLog.so.4.0+0xffffffff] (i attached log)

I have attached AIX log during that period, here is one line from log:

Aug 18 13:00:33 tae1rds1 user:err|error boe_jobsd[16842822]: Subprocess (Job Server Child) could not be started. Reason: Couldn't get IJob interface or writing IAudit: Pipe exception. Reason: boe_jobcd (, 2, Could not send request to child [20775032] (timeout = [120000]ms, request time span = [5098]) .

The error is not because of email as a destination. it is the same when i schedule to BI inbox. The SAP Note 2303169 doesn't solve the problem.

I've put –type OUTPROC –forceterminate –RequestTimeout 36000000 at the end of AJS parameters with no help.

I have also stopped or paused all platform search, auditinig, monitorin and webi schedules and tried to run only one report schedule with the same result.

I have contacted the SAP support and the latest development is that I maybe have C++ redist too high. I have vc++ 13 on AIX, SAP PAM says I need 10. I thought they were backwards compatible.

Please help,

Ivan Rubil

loggingjobserver_tae1rds1.AJS_WebIJavaSchedulingService_CHILD0_dump_20775030.txt.zip

syslog_18082016_1300.txt.zip

[Former Member · Accepted solution]

Hi all,

Thank you for your help and support.

We made a test install on a older AIX server with BO 3.1 on it.

I installed BO 4.1 platform side by side with 3.1 since that server had an older AIX OS which came with XL C/C++ Runtime version xlC.aix61.rte 11.1.0.2

Now everything is working but our client is not happy since that AIX version is nearing its End of Life date. The whole migration to BO 4.1 was because the 3.1 version is close to its EOL date.

With this test we concluded that the XL C/C++ Runtime 13 libraries were the problem.

We had multiple problems that are now gone:

• APS.DSL server restarted when we tried to use UNX with Webi that displayed some CORBA error. The workaround was to set EnableJNI="No" in ../sap_bobj/enterprise_xi40/dataAccess/connectionServer/cs.cfg
• Above workaround and DSL problem meant we couldn't use universes in Dashboards
• AdaptiveJobServer wasn't working as described in this thread

All those problems were gone on a server with lower AIX/C++ versions and we used same BO 4.1 SP8.

I hope SAP fixes this because our client will hesitate to upgrade and migrate to BO 4.1 if they cannot use newer AIX versions.

I'll probably close the current incident with SAP Support but I hope they can give me some notification when the fix arrives.

Ivan Rubil

[former_member240871]

Hi Ivan,

In theory they should be backwards compatible, and its use is supported. However, we recently had a problem with the APS with xlC.aix61.rte 13.1.x and this could be related.

http://service.sap.com/sap/support/notes/2278764

If you can confirm that using a lower level of the C++ runtime works (best would be 12.x, although any version should work), please create an incident through the support portal so we can get this fixed.

Thanks,

Leslie

[Former Member]

Hi Leslie,

AIX Admin says it's not going to be possible to install a different version.

He quotes IBM here:

" The C++ Runtime Environment for AIX is not supported as a stand-alone software product.

Support for questions and problems relating to installation of the C++ Runtime Environment is available as part of AIX Software Maintenance (SWMA).

If you suspect a problem with the C++ Runtime Environment while using it with a third-party application, contact your application provider for support.

Full support (installation, defect support and general questions) for the C++ Runtime Environment is available with the software maintenance agreement for the XL C++ Compilers. A suitable test case is required for defect support; click here for details. "

The customer has provided AIX 6.1 TL9 system which I guess comes with IBM C++ Runtime Environnent Components for AIX xlC.aix61.rte 13.1.2.0

The SAP Note 2278764 you linked suggests to apply 4.1 SP8 as a solution. I already did that and even applied the Patch 1.

I may try to reinstall SAP BI platform and do it from scratch.

I already invested a lot of time to fix this and the problem with report create/refresh with UNX universes. I had to use a workaround for this, disable JNI communication on Connection Server.

I don't know if I messed up during the installations somehow or the SAP BI platform for AIX is bugged. I usually don't have much problems on Windows servers but those are for more simple deployments and customers.

This is a is a side by side installation with Oracle 12c DB and SAP Data Services on the same server so that my be a source of some complications.

Ivan Rubil

[former_member240871]

Hi Ivan,

If it's not possible to downgrade, I would open a support ticket so it can be investigated.

SP08 fixed the issue with the APS, but not necessarily the job server child processes dumping upon creation. I put it out there as it's probably a related problem.

It would be helpful for your unix admin to provide the output of KBA 1681036 for the investigation so libraries can be matched as closely as possible. The other useful thing to have is the backtrace of any coredumps which are created by following KBA 2169984

-Leslie

[Former Member]

Hi Leslie,

I ran the survery.sh before with no problem but this time i get error:

Determining OS

Collecting File System Mount Points

Collecting File System Consumption

Determining System Uptime

Determining CPU Count

Determining System Memory

./survey.sh[372]: no space

There is 20GB memory free, ulimit is unlimited, there is space in TMP, only the free page space is at 750MB

I have attached the debug run of survey.sh

I commented out a line in script which ran netstat. The script moved further but I got the error again on lsof later. Both this commands output a lot of lines

I will see with the AIX admin about the core dumps

Ivan Rubil

survey_debug.txt.zip

[Former Member]

We restarted the server and it worked but I can't upload the survey here because of file type. I'll send it to the support for the incident I opened regarding this problem.

I tried to follow the instructions regarding the core dumps but I just don't know how to read it. The instructions in the Note didn't help.

There is a 1.2GB core file in /appli2/sap41/sap_bobj/enterprise_xi40/aix_rs6000_64

Ivan Rubil

[Former Member]

The logging folder is on another location.

In the jobserver_tae1rds1.AJS_trace.000001.glf i found this error during the scheduling test (attached):

|3d276454-f758-a349-6da9-088c8391497c|2016 08 25 15:38:34:952|+0200|Error| |>>|A|X|jobserver_tae1rds1.AJS|27394136|2422|| |38|0|4|0|CMC.WebApp|tae1rds1:6422534:414.2779:1|CMS.runJobs|localhost:12648672:7968.3443:1|jobserver_tae1rds1.AJS.createJob|localhost:27394136:2422.133:1|Cqn.Ob5LjkhRtgNYcvl7RKoad9|||||||||||Scopes were closed out-of-order. TraceLog workflow data may have been corrupted.

The cms_tae1rds1.CentralManagementServer_trace.000063.glf has some errors. I have attached the problematic timeframe.

The aps_tae1rds1.APS.Core_trace.000385.glf has only "Exception occurred trying to invoke the getter on the MBean" over and over.

Ivan Rubil

cms_tae1rds1.CentralManagementServer_trace.000063_part.txt.zip

jobserver_tae1rds1.AJS_trace.000001_part.txt.zip

[former_member185603]

Based on the support suggestion and PAM requirements, since you have higher version than the supported version, it may be an issue.

If PAM says higher version than the version you have, then it is backward compatible.

[Former Member]

AIX admins will install the xlC.aix61.rte 10.1.0.3 when they can. Plan is to install it side by side with the current xlC.aix61.rte 13.1.2.0 IBM XL C++ Runtime.

I'll report back here with results.

Ivan Rubil