Connecting Twitter with SAP HANA

[Former Member]

Hello,

We are planning to connect SAP HANA with Twitter for doing the analytics.

Earlier, we thought of doing this with introducing Hadoop in between, and with the help of Plume, establish the real-time streaming of tweets.

But, now we have decided to directly connect SAP HANA with Twitter.

I would like to know how can be achieve this, using any intermediate tools, and whether real-time streaming tweets is possible.

Please help me in this regard.

Best Regards,

Ram

[Former Member · Accepted solution]

There are many ways to connect HANA to Twitter.

A great way to establish real-time connectivity is to leverage SAP Event Stream Processor in conjunction with SAP HANA.  SAP Event Stream Processor (ESP) is designed to capture and analyze multiple streams of high-speed, high-volume complex event data in real time, while loading real-time data into SAP HANA.

Twitter’s streaming APIs can be leveraged from within a custom SAP Event Stream Processor (ESP) input adapter, and then you can use SAP ESP’s HANA Output Adapter to load the HANA in-memory database at high volume and low latency. ESP adapters can be developed in, Java, .NET or C/C++, and there are also a variety of third party libraries available to ease interfacing with Twitter’s Streaming APIs.

To get started with development, Twitter offers free access to a random sampling of public Tweets in real-time up to a certain rate limit they define.  The Tweets supplied in this free level are those public Tweets that match filters that you define.  Higher levels of access (access on behalf of users, access to the full “firehose” of public Tweets) are available for a fee.

Mike

[Former Member · Accepted solution]

Just saw the ESP Roadmap:

This suggests ESP is moving to HANA,  to be know as 'Hana streaming'. 

This might be the logical tool to use when it's available????

In the meantime Python might be the easiest for a PoC, as Jose below mentioned.

Going forward I wonder how SAP's partnership with Hortonworks may turn out.

Hortonworks has an open source tool called 'Storm' which also could be used.

http://hortonworks.com/hadoop/storm/

My guess is that for small amounts of streaming data then 'Hana streaming' may be the perfect fit.

As the data grows then ESP standalone  or open source Hadoop Storm/Flume may need to be considered.