Skip to Content

System Alert: Total Max Heap Size exceeds Physical Memory


I'm wondering about the "Total Max Heap Size exceeds Physical Memory" alert.
For example, we get the following output for one of our systems:

[TotalHeap=64.19 GB]
[TotalRAM= 64.0 GB]
APS heap: 46.0 GB
Explorer heap: 6.0 GB
Dashboard heap: 4.19 GB
CR4E heap: 8.0 GB

I didn't check the actual APS heap settings in detail, but it seems to be valid.
The value given for Explorer is OK, I checked this.
But when coming to Dashboard and CR4E it's getting a bit strange.

When checking the xmx for the Dashboard servers, I see:
DashboardsCacheServer -> Java VM Arguments: Xmx858M
DashboardsProcessingServer -> Java VM Arguments:

So why is the BIPST telling that the heap for Dashboard is 4.19 GB?

When checking CR4E it's getting even more confusing.
Since these are actually no Java processes in the first place (crproc.exe, crcache.exe; not java.exe), I'm wondering why it's called "heap" in this case.
When checking the server's properties, I can't find any max memory setting (or command line parameter xmx...).
So where/how is this "max heap" actually defined in this case?

Best Regards

Add comment
10|10000 characters needed characters exceeded

  • Get RSS Feed

5 Answers

  • Best Answer
    Feb 06, 2018 at 07:54 PM

    Hi Moritz,

    For dashboards and CR4E servers, the majority of the heavy lifting is done by the processing children for those servers which are themselves java processes. CR4E and dashboards create java children, and each child will use up to 2gb of memory (Xmx defaults to 2gb if unspecified).

    It looks like BIPST is counting each active child in the max heap measurement.

    Toby Johnston Maybe we should consider exceeds to be a warning, and exceeds by a significant margin to be an alert - It's rare for any job processing to use that much memory unlike the APSs


    Add comment
    10|10000 characters needed characters exceeded

    • Hi Leslie,

      Thanks for the explanation.
      I was aware about the java childrens spawned, but I wasn't that they each have a limit of 2 GB as default if not specified otherwise (I thought it's 1 GB) and that they are also added to the calculation (and which are actually taken into consideration).

      It would be helpful to get a more detailed calculation as output and also some more info in the description for this alert.
      But after having a look at Toby's reply, it sounds to me that this will be fixed.

      At least for our case, it would be also helpful to take the WAS (Tomcat) and a baseline for the OS (Windows) into consideration, too.

      Thanks for your support.

      Best Regards

  • Feb 06, 2018 at 05:17 PM

    Hi Moritz,

    Thanks for reporting the problem. We are investigating and will follow up again here asap.


    Add comment
    10|10000 characters needed characters exceeded

  • Feb 21, 2018 at 03:43 AM

    Hi Moritz,

    I checked in a change to this System Alert and split the one alert into two. In the upcoming patch v2.1.1 (due out in 1-2 weeks) you'll find the new alerts do not calculate the child processes at all and only focus on the java server processes.

    - Alert #1 - calculates all running/enabled parent servers for -Xmx and compares this to physical memory on the host.

    - Alert #2 - same as above except it will calculate ALL parent servers regardless of running and/or enabled status to give a potential comparison.

    The prior alert was too confusing since some of the BI parent processes are c++ but spawn java.exe child processes. For instance the CR4E Processing Server, Dashboard Proc/Cache, JobServer, etc. Although we were calculating the child process heap for many of these, the jobserver was not included.

    I have most of the code done to add a 3rd alert to calculate all potential child processes in addition to the parent procs, but I'm leaving it out of the next patch due to testing constraints. Plus i'm not 100% sure there's value in having it since the potential jobserver child procs (~15 services * 5 default child procs * 1024mb default heap) will almost always add up to more heap than most servers have in physical RAM. Once we have more feedback we'll make a decision whether to include this in patch 2.



    Add comment
    10|10000 characters needed characters exceeded

  • Feb 06, 2018 at 08:28 PM


    Thanks. Yes, Josh is looking into this one and will make changes accordingly.

    Add comment
    10|10000 characters needed characters exceeded

    • Hi Toby,

      One additional remark:
      It seems that Explorer is taken into consideration even if the respective servers are not running (stopped+disabled).
      In our setup the Explorer Servers are always active on one (the first) node only, since we had some issues to get it working on multiple nodes and it's not critical in our scenario, thus we decided to limit it to the first node.
      But it's also showing up in the calculation for the second node (on which the Explorer servers are not running),
      so this became obvious.

      Best Regards

  • Mar 05, 2018 at 04:49 PM

    Hi Moritz,

    The patch for this issue is now available for download (Version 2.1.1). Thanks for reporting all these issues!



    Add comment
    10|10000 characters needed characters exceeded