HANA Spark Controller 2.0 SP02 PL00 fails to start

[former_member462195]

Dear community,

I want to set up SAP HANA Spark Controller 2.0 SP02 PL00 using Cloudera 5.9, Spark 1.6, JDK 8.0 and Vora 1.4 Patch 2. According to product availability matrix this combination should be compatible.

For the purpose of HANA Spark Controller Installation I am following installation guide Hana Spark Controller 2.0 Installation Guide_SP02_PL00.pdf.

When I set sap.hana.hadoop.datastore=vora, initially it is started but after 1/2 mins, it goes down throwing the below error:

17/08/03 23:10:04 ERROR CommandRouterCloudera: Fatal Error
java.lang.NoClassDefFoundError: org/apache/spark/sql/SapSQLContext
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.apache.spark.sql.hana.HanaSQLContext.<init>(HanaSQLContext.scala:57)
        at com.sap.hana.spark.network.CommandRouter.initialize(CommandRouter.scala:420)
        at com.sap.hana.spark.network.CommandRouter.preStart(CommandRouter.scala:212)
        at akka.actor.ActorCell.create(ActorCell.scala:562)
        at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:425)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262)
        at akka.dispatch.Mailbox.run(Mailbox.scala:218)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SapSQLContext
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 24 more
17/08/03 23:10:04 ERROR Server: Fatal Error: Actor System Terminated
17/08/03 23:10:06 INFO CommandRouter$$anon$1: Invoking stop() from shutdown hook
17/08/03 23:10:06 INFO CommandRouter$$anon$1: Successfully stopped SparkContext

1.jpeg

when I set sap.hana.hadoop.datastore=hive the log of /var/log/hanaes/hana_controller.log

2.jpeg

when I set sap.hana.hadoop.datastore=vora the log of /var/log/hanaes/hana_controller.log

Is it some kind of Java Exception Error?

Can you please advice, how I can resolve this error?

Thanks a lot!

[JasonHinsperger]

Aside from the Hana docs. for setting up the Spark Controller, there is some additional setup from the Vora side you should review in the Vora install/admin guide section 2.9:

https://help.sap.com/http.svc/rc/f09ec811fe634f588647c342cac84c38/1.4/en-US/SAP_Vora_Installation_Ad...

Also, if you are using the Vora new relational engine or the disk engine with 1.4, you might investigate using SDA directly from Hana to Vora with a voraodbc connection. This is documented in the install/admin guide for Vora 1.4. Then you would not need the Spark Controller at all.

--Jason