Skip to Content
0
Former Member
Nov 09, 2006 at 10:08 AM

100% CPU load but no clue in thread dump, EP. Win a bottle of champagne...

134 Views

Hi,

We are suddenly facing 100% CPU load in our EP cluster (5 x 8-way Xeon multiprocessor machines). We have a serious performance problem that is burning a lot of our time and causes a lot of stress for over 3 weeks now.

We have taken tens of thread dumps from Application Nodes.

- In none of the thread dumps we see the Finalizer thread

running.

- From the garbage collector log we see that full garbage

collection runs occur very rarily (once an hour).

- We use the compacting garbage collector.

- In a lot of thread dumps we do not even see our own portal code in the stack traces of the threads!

- In Windows Task Manager we see the jlaunch.exe processes

consume all available CPU time.

- Portal users see a blank portal page when the CPU load hits 100%. When the load goes down again, things return to normal.

- There are no errors logged in any of the log files of the portal. We checked all of them. We expected a log full of errors somewhere but nothing even remotely interesting was found. Windows Event Viewer shows nothing either.

- The amount of sockets in CLOSE_WAIT status is < 10 on every machine in the cluster.

This forces me to conclude that something in the jlaunch.exe executable consumes the CPU time. This raises the following 3 questions:

- What does the mystifying jlaunch.exe do besides executing java.exe ?

- Why is the Java virtual machine launched by a custom executable like jlaunch (What is it that cannot be program med in Java) ? Can it be GZip compression ?

- If the problem is not caused by jlaunch.exe, then it must be caused by the JVM. What activity, invisible in thread dumps, is performed by the JVM that can cause the high CPU load ?

Our development- and support teams are desperate. All suggestions are welcome. The person that comes up with

the solution to our problem gets a nice bottle of champagne.

Regards,

Chris Twigt