SAP Vora table auto refresh

[former_member331402]

Hi Community,

We have 2 queries regarding SAP Vora:

1. We came across a scenario where we need to refresh Vora tables with original data source files (present in HDFS) automatically. is there a way this can be achieved?

Can we create and schedule a job in Vora that would integrate with original data source files(located in HDFS) and update and(or) append the required changes into existing Vora tables?

OR

Is it possible to update Vora tables with recent data from HDFS while executing them from Vora?

2. Relational Engine tables created on top of HDFS files gets memory allocated in Vora, is there a way to find out if the memory allocated is on disk or in Memory without restarting Vora instance.

3. Need to know the directory path in Vora that can be accessed via command line (CLI) to know location of tables stored in Vora (across different engines)

[JasonHinsperger]

Ravi,

1. You can always append new data files from HDFS to an existing table using SQL. However, Vora does not automatically pick-up changes to existing files it has already loaded. If you update/append HDFS files with new data, you must either drop and recreate the table in Vora, or restart the Vora in-memory engine.

2. I am not sure I understand you here. Generally, the in-memory relational engine always uses RAM to store table data, and the disk engine always uses disk. The disk engine uses some RAM for processing, but this is limited by the disk engine deployment parameters. You might consider playing around with the Grafana UI and looking at the statistics that are measured/monitored there to see if there is something you can use (see section 3.3 of the Install and Admin. guide).

3. Again, I am not exactly sure what you are looking for, but the location of data files used by Vora is defined in the deployment parameters for the various services (see section 2.5 of the Install and Admin. guide).

--Jason

[former_member331402]

Hi Jason,

Thanks for your reply.

PFB the explanation for question 2 and 3:

2. If Vora relational in-memory engine is rebooted, what happens, if sql query is executed on relational table. Will Vora connect with HDFS location and gets data in-memory or will give error.

3. We are looking for directory location where Vora stores table metadata information

Like /var/log/vora/vora-relational for Relational In memory and /var/log/vora/vora-disk for disk engine. These directory locations may not be correct. We need to check size of Vora tables stored in disk and in-memory

Thanks,

Ravi

[JasonHinsperger]

If Vora is restarted, the engines will reload any hdfs files you have previously added to the table when you query them.

Table metadata is kept by the Vora Catalog service and durability is provided by the Distributed Log (DLog) service. The location of your distributed log files is defined in your DLog service definition parameters, as I mentioned in my original answer.

--Jason