Connect to ESP stream only once in Design Studio

[Former Member]

Hello and a happy new year!

I have time series data coming into an ESP stream and I would like to display the content of that stream in two different ways.

1. Show a time series chart of e. g. past 100 events.
2. Display most recent (i. e. latest/current) values in text components.

Now I am wondering how I can achieve this by conneting only one streaming data source to the stream in ESP. My approach up until now is, to add two streaming data sources of the same stream with different retention policies. One with a high retention policy for charting and another one with a retention policy of 1, to easily access the latest value for display in a text component.

This seems to be slowing down the application however (already at design time, it apperently makes it close to impossible to even work with Design Studio because it is getting very slow). Also, it seems reasonable that I shouldn't have to connect to the same stream twice, since the latest value is already available within the stream with the higher retention policy.

My problem now really is, that I don't now how to access the data cell with the latest time stamp, since I don't know that current time stamp, as it is constantly changing as dimension name.

Ideally I would want to work with a data binding to the latest data cell, as recommended in the users manual, quote:

Unlike analytic components, basic components do not have a data sourceproperty. If you want basic components to display data from a data source, this must be set by scripting. While data-bound components are updated automatically when data is changed, it might be significantly more difficult to synchronize basic components in the same way. In this case, you have to know all the relevant events where a script has to update the basic component's properties.
An easier way to achieve this is to bind a property of a basic component in your application to a data source, by using the property-binding function in the Properties view of this component.

Any suggestions upon how I can bind the data properly from the stream with the high retention policy or how I should script this for access of the latest data cell? Or would you rather recommend my current approach, adding the same stream twice as data streaming source with differing retention policies?

Many thanks in advance!

Daniel

[former_member228644 · Accepted solution]

Hi Daniel,

The Streaming data source is built on the SDK data source which does not support Data Binding.

Also the Streaming data source does not support the script method 'getMembers' - if this limitation is not documented we will try and do so for SP5

As for the flip flop of data values you mentioned - would it be possible for you to use a data bound component like the chart to view the data with a retention policy of 1 as well as the components showing the issue. That for your case we can narrow it down to a general issue with streaming data sources (flip flop happens in both charts and texts) or if it is an issue of using the script method getDataAsString with the datasource (happend just for texts and not chart)

Regards,

David

[Former Member · Accepted solution]

Thank you for clarifying, David. Too bad that it doesn't work that way. Too bad that my idea for a workaround produces such bugs and performance issues. Hopefully I will still be able to come up with some acceptable workaround.

Concerning the FlipFlops: They still occur after connecting that chart to STREAM_1. You could easily rebuild the test project to better investigate the issue. Otherwise, should you have access, you could find my test project attached to incident 12235 /2017.

Good luck fixing this! I am looking forward to hopefully seeing better possibilities for my use case in future releases!

[Former Member]

I want to share some update and more detail about this issue with you. I have done some more testing while I am waiting for development (or support) to get back at me with answers or even solutions.

So it appears binding a streaming data source to a basic component is not possible in the newest version Design Studio 1.6 SP4 due to a bug (or simply due to unimplemented functionality without exception handling, I am sure development will clarify what really is going on at some point). I still want to be able to continue developing my real-time dashboard now and therefore continued exploring how to script this binding. However, my approach for scripting results in that weird behaviour, which I mentioned above in my initial post without wanting to detail it, because I thought it would confuse more than it would help at that point. In order to get ahead and in hope of simple answers (since I really explored everything I can think of with no improvement), I will now go into detail about this weird behaviour and how I produce it.

The "weird behaviour" is, that variables are filled with values which should be contained in other variables. I obtain a value from my streaming data source in Design Studio using the method DS.getDataAsString in a timer component and then use TEXT.setText to write the value in a text component, and I do this for multiple text components. Then sometimes the correct value is displayed in a given component and some other times the wrong value, which should only appear in a different text component. I will continue to detail this below and it will become clearer.

For testing and showing the behaviour here, I created a minimal application running locally. It consists of a simple Java class which connects to the local ESP cluster and publishes to a stream in an ESP project. This stream can then be tapped in Design Studio using a streaming data source.

The Java class looks like the following. With a set delay it publishes three values into the stream: timestamp, 1000, 2000

(note that I pass two fixed integer values - one smaller, the other bigger)

import java.io.IOException;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;

import javax.security.auth.login.LoginException;

import com.sybase.esp.sdk.Credentials;
import com.sybase.esp.sdk.Project;
import com.sybase.esp.sdk.SDK;
import com.sybase.esp.sdk.Stream;
import com.sybase.esp.sdk.Uri;
import com.sybase.esp.sdk.Stream.Operation;
import com.sybase.esp.sdk.data.BigDatetime;
import com.sybase.esp.sdk.data.MessageWriter;
import com.sybase.esp.sdk.data.Publisher;
import com.sybase.esp.sdk.data.RelativeRowWriter;
import com.sybase.esp.sdk.exception.EntityNotFoundException;
import com.sybase.esp.sdk.exception.ServerErrorException;

/**
 * Simple class for testing real-time streaming into Design Studio 
 */
public class publisher {
    private static final SDK s_sdk = SDK.getInstance();
    private static final int WAIT_TIME_60000_MS = 60000;
    private static SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS");

    /**
     * main -> Connect to ESP stream and continuosly publish some simple rows into stream
     * @param args - none needed
     * @throws InterruptedException 
     * @throws ServerErrorException 
     * @throws IOException 
     * @throws EntityNotFoundException 
     * @throws LoginException 
     * @throws ParseException 
     */
    public static void main(String[] args) throws InterruptedException, LoginException, EntityNotFoundException, IOException, ServerErrorException, ParseException {
        /**
         * Connect to stream running on ESP studio local cluster in test project
         */
        String workspace = "default";
        String pname = "test"; // project name
        String uristring = "esp://localhost:9786/"+ workspace + "/" + pname; // 
        String streamName = "test_stream"; // name of stream in ESP project
        String user = "studio";
        String pw = "any"; // TODO: use password set in ESP studio for local cluster!!!
        System.out.println("Connecting to '" + uristring + "' with user '" + user +"'");
        
        // Get project from ESP and connect publisher
        Uri uri = new Uri.Builder(uristring).create();
        Credentials creds = new Credentials.Builder(
                Credentials.Type.USER_PASSWORD).setUser(user)
                .setPassword(pw).create();
        Project project = s_sdk.getProject(uri, creds);
        project.connect(WAIT_TIME_60000_MS);
        Publisher publisher = project.createPublisher();
        Stream stream = project.getStream(streamName);
        publisher.connect();

        /**
         * Continuously publish simple rows into stream 
         */
        long delay = 600; // TODO: set delay in ms for waiting between publishing of rows
        System.out.println("Continously publishing one new row every " + delay + " ms ...");
        while(true){
            MessageWriter writer = publisher.getMessageWriterCached(stream);
            RelativeRowWriter row = writer.getRelativeRowWriter();
            row.startRow();
            row.setOperation(Operation.INSERT);
            // insert timestamp
            row.setBigDatetime(BigDatetime.valueOf(dateFormat.format(new Date())));
            // insert an integer smaller than the latter one
            row.setInteger(1000);
            // insert an integer bigger than the previous one
            row.setInteger(2000);
            row.endRow();
            publisher.publish(writer, true);
            publisher.commit();    
            Thread.sleep(delay);
        }
    }
}

My test project in ESP studio looks like this:

And here is the stream, when I run the test project and Java client:

Now in Design Studio, I would like to display the current values of Smaller and Bigger in two text components and also in an info chart showing past 10 values. I also want to flash an alert, whenever Bigger is less than Smaller (which should obviously never be the case).

Therefore my canvas and outline looks something like the following. The chart is supposed to be displayed, once data from my test stream is coming in, the questionmarks should be replaced by 1000 or 2000 respectively, and the "!!!ALERT!!! BIGGER IS SMALLER!!!" should be turned visible if Bigger is Smaller (which again, should never be the case).

STREAM_10 means, that it has a retention policy of 10 and STREAM_1 means, it is the same stream but with a retention policy of 1. I add my streaming data source with retention policy of 10 like this (with retention policy 1 it is the same, but no time stamp dimension is needed):

In fact, STREAM_1 would not be needed at all, since the latest values I want to show are also contained in the newest element of STREAM_10 (which I definitely need, to be able to show the time chart). However, I don't find any way to access this newest element in STREAM_10 using scripting. I would like to do something like this in my timer component:

var timeMembers = STREAM_10.getMembers("Timestamp", 10);
var latestMemberKey = timeMembers.pop().internalKey;

var smaller = STREAM_10.getDataAsString("Smaller", {"Timestamp":latestMemberKey}); 
var bigger = STREAM_10.getDataAsString("Bigger", {"Timestamp":latestMemberKey});

TEXT_SMALLER_VALUE.setText(smaller);
TEXT_BIGGER_VALUE.setText(bigger);

if (Convert.stringToInt(smaller) > Convert.stringToInt(bigger)) {
    TEXT_ALERT.setVisible(true);
} else {
    TEXT_ALERT.setVisible(false);
}

Everything that might be wrong with this script already starts on Line 1. The MemberArray timeMembers is always of length 0, i. e. empty, therefore there is nothing to pop in line 2. Maybe this simply means, that my time dimension has no members. Then the question would be, how do I get the list of time stamps?

I think this is the key part of my problem, I somehow need to access the latest element, but I am not able to figure out how. Any ideas or recommendations on this?

To work around, I came up with the idea of using that additional stream of retention policy 1 for setting my text components. But this does not seem to be a good idea at all, it looks like this is something where real trouble starts.

I use this script:

var smaller = STREAM_1.getDataAsString("Smaller", {});
var bigger = STREAM_1.getDataAsString("Bigger", {});

TEXT_SMALLER_VALUE.setText(smaller);
TEXT_BIGGER_VALUE.setText(bigger);

if (Convert.stringToInt(smaller) > Convert.stringToInt(bigger)) {
    TEXT_ALERT.setVisible(true);
} else {
    TEXT_ALERT.setVisible(false);
}

This is basically working, but it results in a bug where values are being assigned to wrong variables, i. e. 1000 is sometimes contained in smaller and 2000 sometimes in bigger:

While at first 1000 and 2000 are written to the proper text component as expected and the alert remains hidden, it starts flashing randomly after a while, so that I may see 1000 or 2000 displayed twice or the whole thing flipping so Bigger is written in Smaller and the alert being shown. This is highly dependent on the delay I set in the Java client. Of course, I started by setting everything (i. e. Java client delay, streaming data source update interval, timer interval) to teh same timing of 1000 ms. The values flipped then, and they flip with any different combination. I played around with those values and I realize it is highly dependent on timing, when values flip to wrong variables. I may find delay settings for which the situation is stable for quite some time, before this weird flip flopping happens.

Side note: In my real application I have a number of values and alerts in text components, more than just two, and there things are going wild. Any measure value can appear in any text field, and there is no type security (i. e. float values or strings suddenly appear in supposed integer variables).

This behaviour and my playing around with the delays makes it appear like the issue has to do with synchronizing the stream and the timer component. Interestingly, however, when I remove STREAM_10, leaving an empty chart and the rest of the components as is, behaviour seems to be normal for quite a very long time. Leaving the dashboard open for long enough, however, will still show the flip flopping still occurs - rather seldomly however.

I also tried creating a second stream on the ESP to make sure I don't tap the same stream twice. The Java client then publishes the same values into that second stream, which I then use with a different retention policy. Asides this being another rather stupid workaround, this results in behaviour just as bad as if I tapped the same stream twice.

So clearly, I understand adding the same stream twice with different retention policies shouldn't be the way to go. Since binding basic components to a streaming data source results in an error (and still the question would remain of how to address the latest value), proper scripting with only one data source is my best hope. However, I can't figure out how to make my Timestamp available in my script as dimension in order to get its' MemberArray (or available Timestamps) to choose from.

Does anybody see what I should be doing differently? Your help is really appreciated.

Many thanks & cheers,

Daniel

[etienne_guilhaume]

Hi Daniel,

The good news is that we reproduced this issue internally. The bad news however is that it will take some time to investigate, it took about 30 minutes before the first value switch happened. We'll investigate a bit more and let you know what we find.

Cheers,

Etienne

[Former Member]

Thank you very much for your suggestions!

SP4 unfortunately does not show any different behaviour. I am still not able to bind a streaming data source to a text component without that error showing up, breaking my whole session.

Before opening any incident with SAP support I would be hoping for a reply from the community here, upon how to actually do what I am trying to achieve. All I want to do is to use in my dashboard the latest values from a streaming data source, and I want it to be as close to real time as it can get. I want to show in text components what the current reading is, and also I want to flash alerts immediately, as soon as one value is not what it is supposed to be.

I am unable to find any source addressing this topic - not in the manual, not here in the community, not in the incidents and not on google or on youtube. So I am wondering if what I want to do is actually possible with design studio's standard components. If it can't be done with standard components - e. g. because there are no implementations for this (possibly common) use case yet - I would like to know what other ways are imaginable for me to achieve my goal in the best way.

Does anybody have knowledge about this issue, any ideas or suggestions? May they be theoretical or from practical experience? I would also be thankful if you were able to point me towards documentation or tutorials which I so far was not able to find.

If indeed it should be easily possible to bind streaming data sources to any basic components the way I imagined, please let me know and also tell me how to address the latest data cell properly. Then of course I would like to get to grips with the errors I am experiencing.

Many thanks and many kind regards,

Daniel

[TammyPowlas]

Daniel - the official SAP help doesn't mention binding (or I could not locate it) - http://help.sap.com/businessobject/product_guides/AAD16/en/ds_16SP04_user_en.pdf - but that doesn't mean it is not supported. May I suggest you submit an incident to SAP Support and reference this thread?

[Former Member]

Thank you for giving me hope! 🙂 Chapter 11 is about bindings to basic components. Of course I read it thoroughly and I would be happy to be able to use bindings in that way with my streaming data source.

[TammyPowlas]

Unfortunately I am not familiar with streaming

But in another thread you indicated you were on SP3; perhaps update to SP4 - see https://launchpad.support.sap.com/#/notes/2330273/E

If the issue persists, the error you are showing above is something to open an incident with SAP Support.

[Former Member]

To be a bit more specific about my current approach and problems I am facing:

Before learning about the possibility of data binding, I was trying to update values in text components by using a script within a timer. So in that script I would getDataAsString(<measure>,{}) from my streaming data source with retention policy 1 and set this as the value of a text component. However, not only caused this approach unbearable delays, but also very weird problems with the displayed values, which I don't want to go into detail about since I am convinced now this is simply the wrong approach. I want to go with data binding (or need to know the right place for scripting, which in this case apperently is not the On Timer event of the timer component).

If I now try it with data binding of the streaming data source with retention policy 1 to the text field, I immediately get the following error:

"Analysis Application - The application does not exist. The error that occurred has been logged with ID "... [...] While processing the current request, an exception occurred which could not be handled by the application or the framework."

This error message does stick on the canvas, once it appeared and it seems to be impossible to remove it.

Stack trace looks like this, apperently some NullPointerException occurs:

=============================================================================

com.sap.ip.bi.base.exception.BIBaseRuntimeException: Unknown Error
at com.sap.ip.bi.webapplications.runtime.impl.page.Page._processRequest(Page.java:968)
at com.sap.ip.bi.webapplications.runtime.impl.page.Page.processRequest(Page.java:5358)
at com.sap.ip.bi.webapplications.runtime.impl.page.Page.processRequest(Page.java:5351)
at com.sap.ip.bi.webapplications.runtime.impl.controller.Controller.doProcessRequest(Controller.java:1228)
at com.sap.ip.bi.webapplications.runtime.impl.controller.Controller._processRequest(Controller.java:1084)
at com.sap.ip.bi.webapplications.runtime.impl.controller.Controller.processRequest(Controller.java:1050)
at com.sap.ip.bi.webapplications.runtime.impl.controller.Controller.processRequest(Controller.java:1)
at com.sap.ip.bi.server.runtime.sevice.impl.BIRuntimeServerService._handleRequest(BIRuntimeServerService.java:822)
at com.sap.ip.bi.server.runtime.sevice.impl.BIRuntimeServerService.handleRequest(BIRuntimeServerService.java:1205)
at com.sap.ip.bi.server.execution.engine.runtime.LocalBIExecutionService.executeRequest(LocalBIExecutionService.java:34)
at com.sap.ip.bi.zen.rt.client.handler.designer.LocalExecutionAdapter.executeRequest(LocalExecutionAdapter.java:39)
at com.sap.ip.bi.zen.rt.client.handler.RuntimeRequestHandler.handleRequest(RuntimeRequestHandler.java:28)
at com.sap.ip.bi.zen.rt.client.handler.designer.DesignerRuntimeRequestHandler.handleRequest(DesignerRuntimeRequestHandler.java:33)
at com.sap.ip.bi.zen.rt.client.handler.MainHandler.executeBiRequest(MainHandler.java:75)
at com.sap.ip.bi.zen.rt.client.handler.MainHandler.handleRegularRequest(MainHandler.java:59)
at com.sap.ip.bi.zen.rt.client.handler.MainHandler.handleRequest(MainHandler.java:35)
at com.sap.ip.bi.zen.rt.client.handler.PathHandler.handleRequest(PathHandler.java:54)
at com.sap.ip.bi.zen.rt.client.handler.designer.DesignerPathHandler.handleRequest(DesignerPathHandler.java:57)
at com.sap.ip.bi.zen.rt.client.handler.AbstractSupportabilityHandler.handleRequest(AbstractSupportabilityHandler.java:25)
at com.sap.ip.bi.zen.rt.client.handler.AbstractFrontServlet.handleRequest(AbstractFrontServlet.java:46)
at com.sap.ip.bi.zen.rt.client.handler.AbstractFrontServlet.handleRequest(AbstractFrontServlet.java:42)
at com.sap.ip.bi.zen.rt.client.handler.AbstractFrontServlet.doPost(AbstractFrontServlet.java:19)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.equinox.http.servlet.internal.HttpServiceRuntimeImpl$LegacyServlet.service(HttpServiceRuntimeImpl.java:1271)
at org.eclipse.equinox.http.servlet.internal.registration.EndpointRegistration.service(EndpointRegistration.java:162)
at org.eclipse.equinox.http.servlet.internal.servlet.ResponseStateHandler.processRequest(ResponseStateHandler.java:63)
at org.eclipse.equinox.http.servlet.internal.HttpServiceRuntimeImpl.doDispatch(HttpServiceRuntimeImpl.java:413)
at org.eclipse.equinox.http.servlet.internal.HttpServiceRuntimeImpl.doDispatch(HttpServiceRuntimeImpl.java:365)
at org.eclipse.equinox.http.servlet.internal.HttpServiceRuntimeImpl.doDispatch(HttpServiceRuntimeImpl.java:204)
at org.eclipse.equinox.http.servlet.internal.servlet.ProxyServlet.processAlias(ProxyServlet.java:91)
at org.eclipse.equinox.http.servlet.internal.servlet.ProxyServlet.service(ProxyServlet.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(HttpServerManager.java:356)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:812)
Caused by: java.lang.NullPointerException: while trying to invoke the method com.sap.ip.bi.webapplications.runtime.page.IDataSourceGroup.queueTask(java.util.concurrent.Callable) of a null object loaded from local variable 'group'
at com.sap.ip.bi.zen.rt.components.CellProjectionBoundProperty.prepareRendering(CellProjectionBoundProperty.java:76)
at com.sap.ip.bi.webapplications.ui.items.UiItem.prepareBoundProperties(UiItem.java:1421)
at com.sap.ip.bi.webapplications.ui.items.UiItem.prepareRendering(UiItem.java:1410)
at com.sap.ip.bi.zen.rt.components.base.ZenUiContainer.prepareRenderingOnChildren(ZenUiContainer.java:640)
at com.sap.ip.bi.zen.rt.components.base.ZenUiContainer.prepareRendering(ZenUiContainer.java:618)
at com.sap.ip.bi.webapplications.runtime.impl.page.Page.prepareRendering(Page.java:5303)
at com.sap.ip.bi.webapplications.runtime.impl.page.Page.processRendering(Page.java:5272)
at com.sap.ip.bi.webapplications.runtime.impl.page.Page.doProcessRequest(Page.java:3001)
at com.sap.ip.bi.webapplications.runtime.impl.page.Page._processRequest(Page.java:863)
... 49 more

===========================================================================

Does this give you any hint towards what I should do differently? Are there any tutorials available that I missed so far, which show how to display current values from a streaming data source properly?

Thank you very much for any help on this, I would urgently need it to get ahead.

Many greetings,

Daniel