Steps to Create an EIQ Index with Parquet Files  1

Installing MapR.. 1

Creating the Index. 3

Creating the Virtual Data Source. 6

Configuring EIQ Server to Update Parquet Files. 9

 

Steps to Create an EIQ Index with Parquet Files

Installing MapR

MapR uses Apache Drill to access the parquet files. Apache Drill is included in MapR installations, so the drill explorer and other Apache Drill functions can be used once MapR is installed.

·         Install MapR on the local machine or another machine on the network.

·         Install the MapR Drill ODBC Driver to the local machine.

·         Create a MapR ODBC Connection to the installation location.

·         Configure the driver as such:

o    Enter a Data Source Name

o    Select “Direct to Drillbit” as the connection type.

o    Enter the machine IP for the drillbit, if on the same machine use ‘localhost’.

o    ADD the tag ‘GetMetadataWithQueries=True’ to the end of the advanced properties options.

Creating the Index

Once MapR and the MapR ODBC Driver are installed, SDF can connect to the parquet files on the MapR machine.

·         Open EIQ RTI Tool and connect to the MapR machine using the MapR driver configured in the steps above.

·         Make sure the index path is correct and give the EIQ Index a name.

·         Change the EIQ RTI Tool to RTI Mode

·         Add the desired parquet files to the index.

Each table under the ‘parquet (dir) folder are the csv parquet files.

·         Once the desired tables and columns are added, build an empty Index using the build button in the toolbar.

·         Add entries in XML File which has path of the parquet files input/outputs and selected columns.

 

Creating the Virtual Data Source

After building the index and adding entries to the XML file containing the map to the parquet files, a virtual data source (VDS) needs to be configured. Configuring the VDS for the MapR/Parquet scenario is similar to configuring any new VDS.

·         Open the EIQ Server Configuration Tool.

·         Register the DSN as an EIQ Server Data Source using the Data Sources tab.

·         Configure the VDS.

·         Map the columns to a Standard Data View.

 

Configuring EIQ Server to Update Parquet Files

Once the Virtual Data Source has been created and mapped, the parquet files need to be updated from the source. This is done by configuring EIQ RTIS through the EIQ Server Update Configuration tool to look for the parquet files and update the index.

·         Open the EIQ Update Configuration Tool.

·         Right click ‘Tasks’ and create a new task. Make sure to select the MAPR VDS created earlier.

·         Right click the task and create a new task item using the MULTI – SEGMENT INGESTION option.

·         Click Next.

·         Select PARQUET as the Stream Type.

·         Select the Parquet radio box and enter the XML File path.

·         Click Next.

The next window lists the available parquet folders/files.

·         Select the necessary tables and add them using the ‘Add ->’ button. using ADD ALL option select all files

·         Then, click ‘OK’.

The task item will now appear under the created task.

·         Right-click the Task Item and click ‘Start’ to begin the update process.