Link Indexes

Link Indexes are a special JOIN index. Link Indexes maintain associations between records with matching values based on pre-configured criteria (PK-FK relations and/or Entity attributes) in order to speed up query response time. More importantly, EIQ Link Indexes hold links between records not only within a data source but across data sources. This finds non-obvious relationships across disparate data sources. Also, real-time updates to EIQ Indexes and Link Indexes provide social network analysis and other analytics applications the unprecedented ability to dynamically monitor for changes in relationships across disparate data sources in near-real-time.

The basic premise behind link indexing is that all structures (hierarchical, relational, or network) can be represented as combinations of pairs or links. For example, primary key (PK) and foreign key (FK) relationships between records in relational databases are links between records in multiple tables.

In EIQ Link Indexes, links between structured and/or unstructured data sources are maintained by storing pairs of Universal Record Numbers (URNs). An URN is an integer value that can uniquely identify any record in any data source that is registered with any network-accessible EIQ SuperAdapter in the system.

Link Indexes - Configuration and Build

A sample EIQ Server deployment configuration for building and querying Link Indexes (based on sample data sources from the tutorials) is described below. The EIQ indexes (both content indexes and link indexes) are built on four sample data sources - FBI, POLICE, INS, and DOC. DOC contains entity data extracted from the unstructured documents and is stored as a data source.

See EIQ Server Configuration Tool for configuration details.

An EIQ SuperAdapter is dedicated for each of the data sources. Admins define the entities that form the basis for links across the system data sources at an EIQ Federation Server. Admins instruct EIQ Federation Server to initiate the link index build process. It communicates with the connected EIQ SuperAdapters to build their respective link indexes. All the EIQ SuperAdapters have their corresponding EIQ Update tasks configured and running which monitor the data source for changes and update the content indexes and link indexes in real-time.

EIQ Federation Servers provide admins with capabilities to define entities and initiate a Link Index build process. Apart from PK-FK links, users may create links between records based on matching values for preconfigured entity attributes. Users define entities by specifying the attributes that form the entities in terms of standard data model columns mapped to individual data source specific columns (as per SuperSchema mapping). In a simplified case, two URNs will have a link pair when ALL the attributes match. For example, if a PERSON entity comprises of attributes first name (NAME_FIRST column), last name (NAME_LAST column) and date of birth (DOB column), a link will be created between the records in the mapped data source tables that have matching values for ALL these 3 attributes.

After the initial Link Indexes are built, EIQ RTIS configured at each of the EIQ SuperAdapters will maintain their respective Link Indexes in real-time along with EIQ Content Indexes. EIQ Federation Server and EIQ SuperAdapters are altogether responsible for updating the content indexes and local link indexes and broadcasting the updated link index information (URNS) throughout to the rest of the network. The remaining EIQ SuperAdapters store any resulting links locally and send back additional links to the originating EIQ SuperAdapter.

Link Analysis Queries

Link Index queries are handled by the EIQ Federation Server. It acknowledges all EIQ SuperAdapters in the system that will be included for Link analysis.

Link Index queries based on entities come in two types:

A to B - network showing links between any given two entities A and B
Degrees of Separation (DOS) - expands the network of a particular entity up to given 'n' degrees.

A simplified scenario that uses a single machine with one instance of EIQ Server configured to handle multiple data sources is illustrated in the EIQ Product Server Tutorial. In the example scenario, four data sources are registered under one EIQ SuperAdapter, and the same EIQ Product Server also takes the role of the EIQ Federation Server. The tutorial shows the steps to setup, build, configure, and query link indexes.

Notes on using Link Indexes

A unique Data source ID should be assigned for each data source/EIQ SuperAdapter (ODBC/Documents) that is visible by a EIQ Federation Server. Currently, a maximum of 8 data sources/EIQ SuperAdapters is supported.
Link Indexes can be used to accelerate internal and external joins, enable 'n' degrees of separation queries, and perform and visualize link analysis. Link maps are read from Link Indexes and combined using very fast Boolean arithmetic to reveal direct/obvious and indirect/non-obvious relationships among data. With third-party visualization tools, revelation of these relationships or networks can prove invaluable to analysts.
Link Indexes operate in conjunction with content indexes to enable a powerful set of tools to zero in on critical entities and their associated relationships. If there are probabilities associated with the relationships, these can be used as part of the analyses, e.g. probabilistic link analysis.
Networks can be manually contracted and expanded entities and links can be removed and retained. Subscriptions/alerts can be added to these networks and analysts notified when changed and new data within n degrees of separation impacts networks, i.e. living networks.
The living network capability is the end-result of the near-real-time update processes established as part of the normal content indexes. From the moment a network is instantiated, an audit record is retained and updated with any changes that are made to provide a record and enable a reconstruction or refresh if needed.
Several of these capabilities are in an early form and are being developed, but they can be quickly implemented as production-ready when needed.