Introduction.. 1

SDF Configuration Concepts. 1

Virtual Data Source Interface. 1

Schema Views. 2

Query Execution. 2

SDF Configurations. 3

Configuration Flexibility. 4

SDF Schema Interfaces. 5

SDF Configurations in Deployment 5

 

Introduction

This document describes the SmartData Fabric^® concepts, its various configurations and its deployment options. SmartData Fabric^® , aka SDF, was previously referred to as EIQ Products or EIQ Server. The two terms are used interchangeably in parts of this documentation. This document covers the following topics:

 

• SDF Configuration Concepts
• SDF Configurations
• SDF Configurations in Deployment

 

SDF Configuration Concepts

SDF can be configured in multiple ways to provide different functionalities. Understanding the following concepts is important for configuring SDF.

 

Admins use the EIQ Server Configuration Tool to configure and manage SDF configurations. SDF server components run as services and they must be running before clients can connect. PGAdmin/PSQL query client tools can connect to SDF adapter and federation configurations for executing queries and returning results.

Virtual Data Source Interface

SDF provides standard data access for clients through a Virtual Data Source (VDS) interface. Clients connect to SDF query servers similarly to how they connect to standard data source servers through ODBC, JDBC or native connection drivers. They use a VDS like any other standard relational data source.

 

A Virtual Data Source (VDS) consists of the following:

1.    a name for client applications to connect to (similar to data source name)

2.    a schema view definition (a business data view or data source native schema),

3.    data source(s) registered with the SDF query server

4.    depending on the configuration, a pairing of EIQ Indexes with corresponding registered data source

 

A VDS provides access to a single data source (EIQ Indexed Adapter, EIQ Hybrid Adapter, and EIQ Conventional Adapter) or multiple data sources (EIQ Federation Server).

Schema Views

An SDF VDS can provide two types of schema views:

1.    a virtual schema view based on a standard data model, or

2.    native data source schema

 

SDF’s business data view mapping feature allows admins/data engineers to map data source specific columns to a business data view for Virtual Data Sources. This hides data source specific schema names. A VDS without a business data view exposes the original data source schema to clients.

 

Virtual Data Sources hide details of configured data sources behind a business data view using a mapping. A business data view contains one or more tables composed of column names from a business data dictionary based on a standard data model that are mapped to data source specific columns containing the corresponding data.

Query Execution

SDF configurations execute queries in one of three ways:

1.    at EIQ Indexes corresponding to the data source being accessed

2.    by passing queries to the data source being accessed

3.    by passing queries to other SDF configurations

 

In a typical edge-ware configuration, as an EIQ Indexed Adapter or EIQ Hybrid Adapter (see SDF Configurations below ), SDF executes queries on EIQ Indexes corresponding to data sources and access data sources only for retrieving query results. EIQ Indexes, when cleansed, standardized, and enhanced, enable additional query processing capabilities otherwise not available at data sources.

Admins register data sources with SDF to provide it with necessary information for connecting to data sources, such as, log-on credentials, driver type, and data source schema name. In typical configurations, data sources and their corresponding EIQ Indexes are paired so that SDF configuration knows which index to use for query execution for a data source. Depending on the configuration, some SDF configurations may not have EIQ Indexes. They pass queries to other SDF Configurations or directly to data sources for execution.

When a client connects to an EIQ Indexed Adapter Virtual Data Source and submits a query, the Indexed Adapter translates the query from a business data view to data source-specific schema. It executes the translated query on EIQ Indexes corresponding to the data source. The query on indexes DOES NOT return the data requested by the client. It returns only ROWID values for rows in the data source satisfying query requirements. Then the EIQ Indexed Adapter uses the ROWID values to retrieve raw data from the rows in the data source.

EIQ Federation passes the client query to other configurations (EIQ Indexed Adapter, EIQ Hybrid Adapter, EIQ Conventional Adapter, OR yet another EIQ Federation Server) which each in turn process the query and return the results to the federation . The federation server combines the results from every EIQ Product server and returns results to clients.

 

SDF Configurations

Based on factors, such as the number of data sources being accessed, schema view type, and the component executing the queries, SDF components can be deployed in the following configurations:

 

1. EIQ Federation Server:  Configured as a sub-middleware federation server for accessing multiple data sources in a federated fashion, it uses a business data view or data source native schema. An EIQ Federation Server, typically, does not contain local indexes and passes on queries for execution to other EIQ Product servers. It facilitates Link Index configuration, building and link query processing across multiple data sources through EIQ Adapters.
2. EIQ Indexed Adapter:  Configured as an adapter for accessing a single data source, it uses standardized and cleansed EIQ Indexes for enhanced query processing and a business data view for common interface with other SDF instances.
3. EIQ Hybrid Adapter:  Configured as an adapter for accessing a single data source, it uses standardized and cleansed EIQ Indexes for enhanced query processing and a business data view for common interface with other SDF configuration instances, as well as translates and passes queries to the data source for execution and presents results according to the business data view. An EIQ Hybrid Adapter indexes, cleanses, and standardizes the data that needs it, while directly accessing the data that does not simultaneously.
4. EIQ Conventional Adapter:  Configured as a conventional adapter for accessing a single data source, it uses no EIQ Indexes but has a business data view. It translates and passes queries to the data source for execution and presents results according to the business data view.

 

Note: EIQ Indexed Adapter, EIQ Hybrid Adapter, and EIQ Conventional Adapter are collectively referred to as EIQ Edgeware or EIQ Adapters. These three configurations are used in place of conventional adapters in federated data systems. EIQ Indexed Adapter uses EIQ Indexes for query execution. EIQ Conventional Adapter is similar to conventional adapters by other vendors and passes queries directly to the data source for execution. EIQ Hybrid Adapter uses both EIQ Indexes and a direct query passthrough simultaneously.

 

The following table summarizes the major features of each SDF configuration.

 

 

      Table 2: EIQ Product Server configurations and connection options for EIQ Server components

 

Feature	EIQ Federation Server™	EIQ Indexed Adapter™	EIQ Hybrid Adapter™	EIQ Conventional Adapter™
Typical Use	Accessing multiple data sources through EIQ Adapters	Accessing a single data source	Accessing a single data source	Accessing a single data source
Has EIQ Indexes for Query Processing?	No	Yes	Yes	No – queries are passed through and executed at the data source
Schema	Business Data View or data source Native schema	Business Data View	Business Data View and data source Native schema	Business Data View or data source Native schema
Standardization/Cleansing Data Transforms at EIQ Indexes?	N/A	Yes – as per business data model	Yes - as per business data view	No
Applies Data Transforms to Results	Not currently (likely in future)	Yes	Yes	Yes
Supports Link Indexes	Yes	Yes	Yes	No
Query Support	Federation SQL Syntax	Business Data View SQL Syntax	Business Data View SQL Syntax	Data source SQL Dialect

 

Configuration Flexibility

SDF configurations can be flexible. Apart from the ones mentioned above, users can have special case configurations within a single instance of an SDF configuration:

1.    A special configuration can be a hybrid of EIQ Indexed Adapter and EIQ Federation Server configurations - EIQ Federation Server with local 'Data Source - EIQ Index' pairs for one or more Virtual Data Sources (VDSs).

2. EIQ Indexed Adapter instances are generally dedicated to a single data source. However, a special EIQ Indexed Adapter configuration may have more than one 'Data Source - EIQ Index' pair in a single Virtual Data Source (VDS) for accessing data from multiple disparate data sources. In this case, all the EIQ Indexes reside locally with the server where queries are executed. Queries formed based on virtual schema submitted to EIQ Indexes after translation to data source specific schemas.

For an EIQ Indexed Adapter, the queries can be formed on business data views. To enable this, the server needs additional configuration for column mapping and PK-FK relationships on tables across data sources.

A single instance of an EIQ Indexed Adapter or EIQ Hybrid Adapter can be configured to have these two data sources under a single VDS by registering both 'Data Source - EIQ Index' pairs. Clients can connect to the VDS and execute queries on tables across data sources. An example query enabled by this configuration could be: 'Find the total sales grouped by the sales person's salary grade.

SDF Configurations in Deployment

Below are some example deployments of various SDF configurations in relation to data sources, applications, and third party federation servers. Note that EIQ Federation Servers present a single view of multiple data sources and EIQ Edgeware servers (EIQ Indexed Adapter, EIQ Conventional Adapter, and EIQ Hybrid Adapter) provide access to a single data source each. 

Figure 5: An example SDF deployment in a shared-nothing configuration.

 

 

  

 

Figure 6: An example SDF deployment along with third party adapters.