deployment of JAVA-Patches with SUM failed, MOPZ has downloaded malicious code

[christoph_ostrop]

some time ago, we did an upgrade of our doublestack PI-System from NW-PI-700-SPS18 to NW-PI-731-SPS05.

Now, i wanted to upgrade this PI-system from NW-PI-731-SPS05 to NW-PI-731-SPS07.

with SUM 1.0 SP10 PL4 we are now failing in the Execution-phase (Downtime):

Error in phase 5.6. execution (downtime) of the JAVA-Part

F  ********************************************************************************

F  *** ERROR => Node 'deployment' failed with exit code 67.

F  ***

F  *** Please see section 'Failures in the 'deploying offline components' phase'

F  *** in SAP Note 1316652 for additional information and trouble shooting advice.

F  ********************************************************************************

and the instance could not start up anymore !?

in sap-note 1316652 there is mentioned a separate sap-note 1550641 for this error

http://service.sap.com/sap/support/notes/1316652

in sap-note 1550641 - Update fails-Node 'deployment' failed with exit code 67.           

http://service.sap.com/sap/support/notes/1550641

Solution:

A fix cannot be deployed as a patch, because the deployer itself is broken.

The solution is to reset a back up state of the system, to download the new version of the stack from SAP Marketplace, and to rerun the update/upgrade procedure.

i downloaded the patch-stack (with Solman MOPZ) some days ago,

so MOPZ did download the malicious code of deployment components..... ?!

so what to do?

[Matt_Fraser · Accepted solution]

Hi Christoph,

"Malicious code" is a bit extreme, don't you think?  Anyway, no, there's no malicious code here, there's just a bug in one of the components.  Granted, the language in Note 1550641 is not the most clear, but basically what it is saying is that there is a known bug (with a fix available) for the offline deployment tool.  However, because the bug is in the tool itself used for deployment, you can't just patch it and move on.  Instead, you must reconfigure your stack to include the updated version of the offline deployment tool and start over.

However, I don't really think this is your problem.  That fix was incorporated into stack 3 of NetWeaver 7.30 a couple years ago, and you are well past that.  So, take a look in the log files suggested from the other Note, 1316652, and see if you can find more clues to what went wrong.  Those log files are:

• dev_jstart*
• dev_deployment*
• std_deployment.*
• jvm_deployment.*

See if you can find errors in one or more of those.  If you do, post the relevant section of the relevant log here.

Or, do as the Note suggested, and open a Customer Incident with SAP Customer Support against component BC-JAS-DPL and attach those log files to the Incident.

Regards,

Matt

[christoph_ostrop · Accepted solution]

Hi, Matt,

thanks for your reply,

> However, I don't really think this is your problem.

> That fix was incorporated into stack 3 of NetWeaver 7.30 a couple years ago,

> and you are well past that

yes, i think same,  NW-731-SP05 should be newer than the situation described in sap-note 1550641

> and open a Customer Incident with SAP Customer Support

> against component BC-JAS-DPL and attach those log files to the Incident.

yes, i did already, on 16.may2014, but until now, no answer

Chris

[Matt_Fraser · Accepted solution]

Did you find any related error messages in the log files?

[christoph_ostrop · Accepted solution]

Hi,

sorry for my late reply,

but the whole story needs a lot of time (nearly 2 month) and finally ends successfully.

first: (summary) the main-title of this thread is wrong,

it was not MOPZ or Solman download, what caused this issue.
the problem was caused by a remaining (old) sapxmltoolkit.jar in the java-bootstrap folder
details are coming ...

(all dates in yyyy-mm-dd)

=========================================

2014-05-16 at 13:04 CET: 
SUM upgrade hangs with java-server did not start (in phase 5.2. Execution = downtime)
i opend OSS-ticket with several details + logs attached and opened the connection to our system.
SAP-Ticket, prio_Medium (because it is "only" a sandbox issue),

2014-05-30 at 20:21 CET:
1."sign-of-live" from sap-support,
Asking for additional logfiles from the work-directory of the sap-system

2014-06-02 at 11:14 CET:
my Answer to the questions and adding the demanded logfiles.

2014-06-02 at 21:57 CET:
2.answer from SAP-support, asking to repeat the last SUM-step, to see, if the issue is still there

2014-06-03 at 10:22 CET:
my answer after repeating SUM-step and attaching new logfiles.

2014-06-03 at 21:18 CET:
3.answer from SAP-support, i should check if system is up and running, if hostname, port-no is correct.

2014-06-03 at 10:22 CET:
my answer: hostname + portnr are o.k. - but system still did not startup (java-server)

2014-06-04 at 19:30 CET:
4.answer from SAP-support, delivering an upload-URL, could not read attached splitted .zip-files of log-files

2014-06-05 at 07:28 CET:
i did collect all logfiles again and uploaded them to the 1.upload-URL

2014-06-05 at 16:48 CET:
5.answer from SAP-support, give hint to sap-note: 1550641 - Update fails-Node 'deployment' failed with exit code 67

2014-06-06 at 10:17 CET:
i answered that (in my opinion) the mentioned sap-note did not match the issue (only to NW-7.00) ,
because i did not have to change anything, only i should do a restore from tape and start SUM from the beginning,
i insisted in more and deeper investigations by SAP-support.

2014-06-06 at 19:00 CET:
6.answer of SAP-support: moving the ticket to the next support-level (developm.support)

2014-06-09 at 10:11 CET:
7.answer from SAP-support: asking for new submittion of logfiles of work-dir.

2014-06-10 at 08:39 CET:
asking for a new upload-URL because of large logfiles

2014-06-10 at 17:31 CET:
8.answer from SAP-support: providing an new upload-URL

2014-06-10 at 17:53 CET:
i did collect all logfiles again and uploaded them to the 2.upload-URL

2014-06-12 at 09:07 CET:
9.answer from SAP-support: asking for more details, listings of directory-content of ..../j2ee/cluster/...

2014-06-12 at 10:23 CET:
i did collect all dir-infos and attached a doc to the ticket

2014-06-19 at 10:11 CET:
10.answer from SAP-support: asking for WTS-connection to start investigate and debung on our systems

2014-06-23 at 10:44 CET:
i answered, and provided WTS-connection info

2014-06-26 at 13:12 CET:
11.answer from SAP-support: could not see a running SID-engine, no SAP-MC, no directory, no folder, where is the engine?

2014-06-27 at 08:01 CET:
i answered, and provided detailed server-info and WTS-connection info,
remembering that SID is in SUM-downtime and java-server is not running, because could not be startet,

2014-06-30 at 14:01 CET:
12.answer from SAP-support: asking for an installed java6-sdk(jdk), the java6-jre is not enough to debug.

2014-07-02 at 11:40 CET:
my answer after installing java6-sdk on the WTS-server,

2014-07-07 at 15:11 CET:
13.answer from SAP-support: after debugging found an very old sapxmltoolkit.jar in the java-bootstrap folder
moving the ticket to SUM-support - java-developm-support.

2014-07-09 at 08:36 CET:
14.answer from SAP-support: (SUM-support - java-developm-support) asking for attaching the SUM stack-xml file to the ticket.

2014-07-09 at 14:32 CET:
16.answer from SAP-support: asking to remove the sapxmltoolkit.jar from the java-bootstrap folder and restart SUM (repeat last step)

2014-07-09 at 15:00 CET:
i am happy to get the solution to this issue, 

after removing the .jar file from java-bootstrap folder, SUM restart last step,
the SUM-phase execution continues to run, java-server could start, ....

.... finally SUM ended successfully.

==================================

remarks:
formerly, we did a copy of that mentioned sapxmltoolkit.jar to the java-bootstrap folder (in March 2013) because of investigation PI-processes
after upgrading from NW-7.00 to NW-7.31, to get PI-system running after upgrade to NW-7.31
but maybe that formerly issues were caused by other things, cache or so ...
and since then (March 2013) we could start/stop that PI-system several times without any problem, only SUM-upgrade could not ......

i really understand that investigating problems in SAP-software (e.g. SUM-process, java-startup ...) is not easy and could have many causes.
but in summary this SUM-downtime was really large (55 days of downtime) , because of sandbox-system we want to investigate this time,
to get the solution to this issue.

if some SAP people want to look for speed-up options in the whole support-process, the OSS_ticket-no is:  ( 459757 / 2014 )

Thanks to SAP-support (for the helpful solution)

Christoph

[former_member219087 · Accepted solution]

Hi Christoph,

Am facing the same issue while upgrading my PI 7.1 (Dual Stack System) to 7.4 Dual.

During Execution phase - Deploy Software Components Step i have also struck with same issue.

My JAVA is not getting started and am getting the below error.

[Thr 2460] Sun Apr 24 08:39:09 2016

F  [Thr 2460] *** LOG => Process snapshot stopped (pid 5136).

F  [Thr 2460] *** LOG => Instance state is "Deploying offline components" (STOPPING @ 0, INACTIVE).

F  [Thr 2460] *** LOG => Run level 1 completed.

F  [Thr 2460] *** LOG => Instance state is "Some processes failed" (STOPPED @ 0, INACTIVE).

I  [Thr 2460] MPI: dynamic quotas disabled.

I  [Thr 2460] MPI init, created: pipes=4000 buffers=2718 reserved=815 quota=10%, buffer size=65536, total size MB=170

I  [Thr 2460] *** ERROR => FcaInit: already initialized [fcaxxcomm.c  956]

I  [Thr 2460] *** ERROR => EvtRmMgt: Evt Mgt Table not init'd [evtnt.c      255]

I  [Thr 2460] *** ERROR => ShmDelete: Invalid shared memory Key=62. [shmnt.c      753]

I  [Thr 2460] *** ERROR => ShmCleanup: ShmDelete failed for Key:62. [shmnt.c      872]

I  [Thr 2460] *** ERROR => ShmCleanup(62) failed 1 [mpixx.c      4609]

F 

F  ********************************************************************************

F  *** ERROR => Node 'deployment' failed with exit code 67.

F  ***

F  *** Please see section 'Failures in the 'deploying offline components' phase'

F  *** in SAP Note 1316652 for additional information and trouble shooting advice.

F  ********************************************************************************

F 

F  [Thr 2460] *** LOG => exiting (exitcode 22002, retcode 1).

As you have mentioned i have deleted my sapxmltoolkit.jar from my bootstrap folder and tried starting my JAVA instance from MMC but no luck.

Also tried resuming back my SUM, Since JAVA is not UP am getting the same issue.

Can you kindly brief on what have done ?

After deleting can you list out what all required restart ?

Am struggling at this phase from last 3 days.

Would be great if you can help me with the detailed actions you have performed.