WebIDE on Hana: Builder 'hdi-builder' is not available for workspace

[draschke]

I got this error again and this time it has nothing to do with a newer patch that we installed as I described in the second part of this thread:

We get this error each time, if we try to build a modul.

There seems to be a problem with the post request (Status 500)

Console of the WebIDE

This could I find within the di-core log:

I reinstalled all DevX Apps

[draschke]

Hi Marc,

now we have an other big problem. Maybe it has something to do with "GC overhead limit exceeded"  from yesterday.

I don't know how, but we were loading data by almost 1 TB data over night.

(I got a call by our admins right now)

/HANA/usr_sap/mydummy/xs/ea_data/saphana2/executionroot

[draschke]

Folder

: /HANA/usr_sap/ mydummy/xs/ea_data/ mydummy/dropletroot/blob-store/blobs/8b3

within this data

-rw-r--r-- 1 ch2adm sapsys   880 May 18 09:52 8b31080c5a9c6624e05f37a5aabe30d144879595779a8609262b13ee8d6449e5-880.txt

-rw-r--r-- 1 ch2adm sapsys   462 Apr 27 09:39 8b31d91af2336ddba43b2c332900cc9567cd5bf9e18ec3f411cdbf161e7c5feb-462.txt

-rw-r--r-- 1 ch2adm sapsys   693 Apr 23 21:57 8b3437531b83021d39adffd0d4deaabbccd2900f7120ff5bd88005cb9d77ad0e-693.txt

-rw-r--r-- 1 ch2adm sapsys  1800 Apr 27 16:19 8b38fd0f96a10b7b5b060b58cbcdba22a87a62bc669c5f748166048a3125ceef-1800.txt

-rw-r--r-- 1 ch2adm sapsys   317 Apr 26 11:37 8b3a5edcfe2d56723f5f50f8c58f2a9860345cc84237156f496d4b3727457b3c-317.txt

-rw-r--r-- 1 ch2adm sapsys 24433 Apr 27 09:39 8b3d170aeda3cd7115eb11e66475247c71324adce313cef78225cbbdf96eadf1-24433.txt

-rw-r--r-- 1 ch2adm sapsys   572 Apr 23 21:57 8b3db01552c7ccb8c4b5ce1da07a56198191ee42895772b35e9ef3cceb7da348-572.txt

-rw-r--r-- 1 ch2adm sapsys   439 Apr 27 09:39 8b3e0492f6fe368ec15547f409effc86dd933509e0e6fd49135ab6809b1ebc83-439.txt

inside of this data:

{

  "name": "sprintf",

  "description": "JavaScript sprintf implementation",

  "version": "1.0.3",

  "main": "src/sprintf.js",

  "license": "BSD-3-Clause-Clear",

  "keywords": ["sprintf", "string", "formatting"],

  "authors": ["Alexandru Marasteanu <hello@alexei.ro> (http://alexei.ro/)"],

  "homepage": "https://github.com/alexei/sprintf.js",

  "repository": {

    "type": "git",

    "url": "git://github.com/alexei/sprintf.js.git"

  }

}

or an other sample

Cloning git repository: mydummy/git-server/2f/66/af/SYSTEM/nodejsappcontroller-buildpack-project

Cloning git repository succeeded.

[1463557869305]The run environment is: x86_64

[1463557869306]nodejs-buildpack-op compile script was started

[1463557869308]starts EmbeddedBuildpack compile script

[1463557872258]Node.js buildpack version 2.2.4

[1463557872262]Avoid using semver ranges like '*' in engines.node

[1463557872932]Downloading Node.js...

[1463557893714]Unpacking Node.js archive...

[1463557901297]Omitting npm install: node_modules directory is already present

[1463557901297]Copying SSL CA certificates...

[1463557901322]go to droplet directory

[1463557901324]extract files from xs.nodejs.appcontroller.tar located in buildpack to container droplet

[1463557901534]The run environment is x86_64

[1463557906290]nodejs-buildpack-op compile script was finished

003f7ba2-aa7f-4da8-9822-35ea6a597bbd

drwxr-xr-x 6 ch2adm sapsys 4096 May 18 12:23 app

drwxr-xr-x 2 ch2adm sapsys 4096 May 18 12:22 app-logs

-rw-r--r-- 1 ch2adm sapsys  321 May 18 12:23 app_info.dat

-rwx------ 1 ch2adm sapsys 3201 May 18 12:23 cleanup.sh

drwxr-xr-x 2 ch2adm sapsys 4096 May 18 12:23 logs

-rw-r--r-- 1 ch2adm sapsys  146 May 18 12:22 staging_info.yml

-rw-r--r-- 1 ch2adm sapsys   44 May 18 12:23 startCommand

[draschke]

We took the system down.

[marcbecker]

The /HANA/usr_sap/mydummy/xs/ea_data/saphana2/executionroot folder grew to 1TB over night? This folder contains all the data of all applications that are currently running. Is there one specifically big folder inside?

[draschke]

Yes, 1 TB over the last night.

No. There are many folders which between 200 and 600mb.

[draschke]

Could it be possible, that xsa is requesting npm packages again and again.

[marcbecker]

Hi Dirk,

most likely you have a (big) application that is repeatedly crashing. XSA continues to restart the application, since it can't know the cause why the application is crashing and if the application may actually start again some time.

In 1.0.17 the cleanup of crashed and old stopped application instances is still pretty slow. There were major optimizations introduced with SPS 12 (XSA version 1.0.24). The regularly crashing application is probably producing more "trash" than the execution agent is able to clean up.

You can find out if there are such applications by running 'xs apps' and checking if you see any application where it says something like "0/1" in the instances section, while the requested state is set to STARTED. Running 'xs app <appname>' should show you if this application has crashed recently.

Regardless of that you can stop the system, delete the complete executionroot folder and restart the system again. Your applications will all be able to start again.

[draschke]

You can find out if there are such applications by running 'xs apps' and checking if you see any application where it says something like "0/1" in the instances section, while the requested state is set to STARTED.

--yes, i saw a lot of them.

The system is until tomorrow down.

Thanks a lot!

[draschke]

Hi Marc,

"a lot of trash"

I checked this problem now on SPS 12 (XSA version 1.0.17) again and stopped the apps.

But even after we stopped this apps, there was a lot of trash generated. (and the system was restarted)

Now we'll go to 1.0.24 and check if this issue is solved and I'll give you feedback soon.

[draschke]

Tried to restart the di-builder again, but it fails:

GC overhead limit exceeded?

[Former Member]

Any chance you can try to run "HDB stop;HDB start" to restart both HANA and XS Advanced?

[draschke]

Is there a problem with the memory?

DivX Apps memo : Not

I don't know how it was before

[Former Member]

: Any idea?

[draschke]

I don't know why, but now I can build the moduls again.

But now I get the same message "GC overhead limit exceeded" when I try to start the js modul.

Until now I couldn't find somebody to restart the hana system (holidays in Germany).

[draschke]

After restarting the hana system I can start the js moduls, without this error "GC overhead limit exceeded" .

I tried at first only the services xscontroller, and so on... , but it didn't help me.

[marcbecker]

Do you see the exception stack traces for the "GC overhead limit exceeded" in any log?

They should be either in the xscontroller or in the xsexecagent log. If you find a stack trace there I might be able to narrow down the problem.

[draschke]

For this moment it works.

If the issue'll come up again, I'll search for the messages within the logs.

Thanks!

[draschke]

I'm not over it, need further help.

GC overhead limit exceeded (max heap: 128 MB)

[marcbecker]

Cyou please check the xsexecagent log file and see if there are similar error messages?

As far as I can tell the first screenshot is from the xscontroller logs and the second is from the output of xs logs <your app>, correct?

[draschke]

yes, exactly. sorry for that I missed it.

xsexecagent:

[2016-05-17 14:07:08:546]-[REST.Server]-[error]-[.]-[8646]: Unexpected Exception in Execution Agent due to internal error.

java.lang.OutOfMemoryError: GC overhead limit exceeded (max heap: 128 MB)                                                                                                                  

  at java.util.Arrays.copyOfRange(Arrays.java:3664)

  at java.lang.String.<init>(String.java:207)

  at java.lang.StringBuilder.toString(StringBuilder.java:411)

  at com.sap.xs2rt.utils.UTF8Util.asString(UTF8Util.java:68)

  at com.sap.xs2rt.utils.UTF8Util.asString(UTF8Util.java:39)

  at com.sap.xs2rt.core.impl.blobstore.FileBasedBlobStoreImpl.getBlobSetImpl(FileBasedBlobStoreImpl.java:1025)

  at com.sap.xs2rt.core.impl.blobstore.FileBasedBlobStoreImpl.getBlobSetOrNull(FileBasedBlobStoreImpl.java:980)

  at com.sap.xs2rt.core.blobstore.BlobStoreHelper.materializeBlobSet(BlobStoreHelper.java:86)

  at com.sap.xs2rt.executionagent.impl.task.DropletManager.materializeTo(DropletManager.java:585)

  at com.sap.xs2rt.executionagent.impl.task.TaskRootBasic.materializeDroplet(TaskRootBasic.java:114)

  at com.sap.xs2rt.executionagent.impl.task.DropletManager.copyDropletToExecutionDir(DropletManager.java:549)

  at com.sap.xs2rt.executionagent.impl.task.TaskControllerImpl.start(TaskControllerImpl.java:369)

  at com.sap.xs2rt.executionagent.impl.task.TaskControllerImpl.start(TaskControllerImpl.java:336)

  at com.sap.xs2rt.executionagent.impl.DropletsController.startInstance(DropletsController.java:174)

  at com.sap.xs2rt.executionagent.impl.rest.DropletsResourceHandler.lambda$putEaDropletsByGuidInstances$20(DropletsResourceHandler.java:166)

  at com.sap.xs2rt.executionagent.impl.rest.DropletsResourceHandler$$Lambda$24/2030594615.call(Unknown Source)

  at com.sap.xs2rt.executionagent.impl.rest.RequestHandlerUtils.executeRequest(RequestHandlerUtils.java:218)

  at com.sap.xs2rt.executionagent.impl.rest.RequestHandlerUtils.executeRequest(RequestHandlerUtils.java:164)

  at com.sap.xs2rt.executionagent.impl.rest.DropletsResourceHandler.putEaDropletsByGuidInstances(DropletsResourceHandler.java:164)

  at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:498)

  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)

  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:164)

  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:181)

  at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:158)

  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:101)

  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)

  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)

  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)

  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:305)

  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)

  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)

  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)

  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)

  at org.glassfish.jersey.internal.Errors.process(Errors.java:267)

  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)

  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:288)

  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1110)

  at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:381)

  at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:222)

  at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)

  at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)

  at java.lang.Thread.run(Thread.java:812)

[marcbecker]

Hi Dirk,

thank you for the detailed error message. We will look into this for sure.

For you the best workaround is to increase the memory available to the execution agent.

You can do that by editing the "xs-ea" file within the xs/bin directory

Somewhere at the end of the file you should fine a line saying something like:

JVM_OPTS="-Xmx128m ...

My suggestion is to double up the max heap to 256 mb (JVM_OPTS="-Xmx256m ...)

I hope this is a feasible workaround for you.

Best regards,

Marc

[draschke]

Hi Marc,

"You can do that by editing the "xs-ea" file within the xs/bin directory"

--> sorry, but I can't find this file xs-ea.

You don't mean the local XS_Client, or do you?

[marcbecker]

No this file is on server side: In the default installation this is at /hana/shared/<SID>/xs/bin/xs-ea

It is used to startup the xsexecagent service.

[draschke]

I didn't set this value yet, because I don't get this error at this moment.

I struggle now with an other problem, while I execute the tinyworl.js after I set the HANA Configuration.

// configure HANA

options = xsjs.extend(options, xsenv.getServices({ hana: {tag: "hana"} }));

I'm not sure, but it seems that the problems came up, after we installed Patch 17 one week ago.

(XS Advanced von Patch 13 auf Patch 17)

5/17/16 3:44:11.927 PM [API] OUT Starting new instance 'cf0cf5b6-aa24-4e67-ba0e-e5d7f2ea33c0' of application "DIRASCHK-vp6jxdvv9yzfmvye-tinyworld2-tinyjs" (port 50051).

5/17/16 3:44:12.930 PM [APP/659203-0] OUT APP Controller Listening on port: 50051

5/17/16 3:44:12.959 PM [APP/659203-0] OUT User Application Proxies to port 55009

5/17/16 3:44:12.962 PM [APP/659203-0] OUT Command To Execute: ./start_node.sh

5/17/16 3:44:12.962 PM [APP/659203-0] OUT Command to run as Child process: ./start_node.sh

5/17/16 3:44:12.962 PM [APP/659203-0] OUT Node.js/Server App Mode

5/17/16 3:44:12.969 PM [APP/659203-0] OUT Child Process ID: 79730

5/17/16 3:44:12.985 PM [APP/659203-0] OUT The app port is not yet occupied; application invocation is still in progress

5/17/16 3:44:12.985 PM [APP/659203-0] OUT Waiting for App to start, checking port: 55009 attempt: 1

5/17/16 3:44:13.987 PM [APP/659203-0] OUT The app port is not yet occupied; application invocation is still in progress

5/17/16 3:44:13.987 PM [APP/659203-0] OUT Waiting for App to start, checking port: 55009 attempt: 2

5/17/16 3:44:14.988 PM [APP/659203-0] OUT The app port is not yet occupied; application invocation is still in progress

5/17/16 3:44:15.144 PM [APP/659203-0] OUT Waiting for App to start, checking port: 55009 attempt: 3

5/17/16 3:44:15.367 PM [APP/659203-0] OUT Server listening on port 55009

5/17/16 3:44:15.367 PM [APP/659203-0] OUT stdout: Server listening on port 55009

5/17/16 3:44:15.367 PM [APP/659203-0] OUT 

5/17/16 3:44:15.369 PM [APP/659203-0] ERR events.js:85

5/17/16 3:44:15.369 PM [APP/659203-0] ERR throw er; // Unhandled 'error' event

5/17/16 3:44:15.369 PM [APP/659203-0] ERR ^

5/17/16 3:44:15.369 PM [APP/659203-0] ERR Error: listen EADDRINUSE

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at exports._errnoException (util.js:746:11)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at Server._listen2 (net.js:1158:14)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at listen (net.js:1184:10)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at Server.listen (net.js:1269:5)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at EventEmitter.app.listen (/HANA/usr_sap/mydummy/xs/ea_data/mydummy/executionroot/cf0cf5b6-aa24-4e67-ba0e-e5d7f2ea33c0/app/node_modules/express/lib/application.js:595:24)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at Object.<anonymous> (/HANA/usr_sap/mydummy/xs/ea_data/mydummy/executionroot/cf0cf5b6-aa24-4e67-ba0e-e5d7f2ea33c0/app/server.js:20:15)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at Module._compile (module.js:460:26)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at Object.Module._extensions..js (module.js:478:10)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at Module.load (module.js:355:32)

5/17/16 3:44:15.369 PM [APP/659203-0] ERR at Function.Module._load (module.js:310:12)

[eckart_liemke]

Hi Dirk,

is this a permanent issue? Or does it only occur from time to time?

Can you probably find out which process is occupying port 55009?

Best regards,

Eckart

[draschke]

I saw it more times, but didn't pay attention, which port it was in the past. Will  keep it in mind!

I'm not sure but it could be possible that it comes from the debugger.

Will see it today.

Thanks for helping!