EIQ ADAPTERS™ IN DETAIL

External Index and Query (EIQ) adapters™ are unique federated data access adapters that externally INDEX and execute queries against these external Indexes for all indexable data and information. They can perform structured database queries and unstructured text searches WITHOUT submitting queries to the data source, copying and storing data elsewhere, and changing data schemas.

EIQ Adapters™ are NOT conventional federated adapters, NOT data warehouses, NOT Enterprise Search only, and NOT Extract, Transform and Load (ETL) tools.

EIQ Adapters™ are NOT conventional technology.

WhamTech and its predecessors developed unique high performance index and query processing technologies over the last 25 years as part of their own relational database management system (RDBMS), called Thunderbolt.  Thunderbolt was so fast that WhamTech used it to develop unstructured text search technologies for its search engine products for Web, enterprise and specialized search, which, by definition, operate externally for files, documents, including Web pages, and e-mail.  WhamTech’s EIQ Products™ bring the power of its advanced index and structured query processing technology, and unstructured text search, to databases and data sources other than its own. (Please visit our products page for the range of EIQ Products™ offered by WhamTech.)

EIQ Adapters™ enable radically improved and/or changed processes – not just better technology.

WhamTech’s EIQ Adapters™ are not conventional. They take a very different approach to the major problems facing almost all large organizations: How to integrate, share and interoperate data and information in near real-time without (a) creating a huge additional super-infrastructure, e.g., massive data warehousing, (b) overloading existing systems, e.g., conventional adapters in federated systems, and (c) losing the ability to execute structured database queries, e.g., enterprise search?

EIQ Adapters™ offer the advantages of the above three approaches without the disadvantages associated with each of them. EIQ Adapters™ not only enable data and information (hereinafter collectively referred to as data) virtualization, federation, integration and interoperability within and between organizations, but can also “turbo charge” existing data source access and add significant features such as:

• Complex query processing
• Heuristic data mining
• Pre-aggregated and pre-calculated indexes for direct queries
• Monitoring key performance indicators (KPIs) and performance
• Querying across and between databases, files, documents and e-mail
• Link mapping and link analysis (connecting seemingly unconnected data and information)

EIQ Adapters™ include a highly flexible middleware/sub-middleware product, called EIQ Federation Server™, that resides between applications/middleware and other EIQ Adapters™. EIQ Federation Server™ can access other EIQ Federation Servers™ to allow indefinite performance, scalability, load balancing, redundancy and failover. EIQ Adapters™ can apply business rules within to enable the best combination of data to be queried and passed back to applications. This provides true organization-wide real-time Data Virtualization, federation, integration and interoperability. EIQ Adapters™ are accessed by applications, including middleware, through standard ODBC, JDBC, OLE-DB, Web Services and a native driver. Standard Query Language (SQL) or WhamTech’s native language, TQL, are used to submit queries.  The following diagram, EID1, illustrates where EIQ Adapters™ reside in the application/data source architecture:

EID1: EIQ Adapters™ reside between applications/middleware and data sources.

There are some similarities to conventional approaches, but the differences become clear as the process is described in more detail:

• Discover data sources using ODBC calls, if available, local DBA and/or other means.
• Profile data using raw indexes, which consist of trees and lists (see Technologies for more information). The trees are histograms of domain values and are used as data profiles. The lists do not need to be retained, however, a possible near-future option may allow these lists to be used to build the initial clean production indexes, without having to re-read the data source.
• Develop data quality transforms from the data profiles generated by raw indexes, which cleanse, transform and standardize data read for building and maintaining indexes. The data profiles allow a “before” and “after” iteration process.
• Read source data using one or more of twelve ways.
• Clean, transform and standardize data used for indexes.
• Build and maintain indexes using the same index schemas as data sources – no major Schema transforms.
• Discard data.
• Map Standard Data Models to EIQ Indexes™. More than one standard data model can be mapped.
• Applications/middleware access EIQ Products™ like a database through standard drivers and/or Web services.
• Present a single virtual indexed view of all data sources to applications based on standard data model(s).
• Execute SQL queries for structured database queries and unstructured text search almost entirely in EIQ Indexes™.
• Generate a list of pointers, URLs, RDFs, etc., to raw results data in data sources.
• Use pointers to retrieve raw results data from data sources through user-level access with appropriate authentication and security.
• Clean, transform and standardize raw results data.
• Consolidate, or otherwise post-process, results data from multiple data sources.
• Present results to applications/middleware in any format.

The above process is illustrated in some detail in the following diagram EID2:

EID2: EIQ adapter process in detail, in this case, an EIQ SuperAdapter™

  EIQ ADAPTER TECHNOLOGY KEY POINTS (OTHERS ARE LISTED UNDER 28-POINT COMPARISON)

1. Virtual, private CLOUD-like access through multiple distributed universal and uniform INDEXES and query engines for structured queries and text search on ALL data sources, from mainframes to Web documents to email – seen as one consistent data source conforming to one or more standard data models.
2. Data warehouse query and results quality, and performance, due to:

• • Cleansed, transformed and standardized indexes (and result sets)
  • Multiple indexes available, including fuzzy, pre-aggregated and pre-calculated, join and Link Indexes™
  • Complete control over query processing
  • Combined structured queries and text search

3. Obtain results when data sources are unavailable through index inversion.
4. Archive options available through (i) storing changed data in a separate database, (ii) date-time stamping indexes and/or (iii) embedding changed data in indexes.
5. Leave source data in place in original format and index schemas as per original data source – only INDEXES mapped to standard data models.
6. Continue to use legacy applications and data, but allow for:

• • Modern application access to legacy data
  • Legacy application access to modern databases
  • Use as a bridge/transition/migration tool from legacy to modern systems

7. Scale through non-tiered and/or multi-tiered, independently maintained indexes with single-point for an application and/or multi-point access for middleware, including Web services and XML.
8. High performance queries:

• • Typically, 10 – 100 times faster than other data-based systems
  • Use the three “Bs” of computing – binary index trees, bitmap result-set representations and Boolean arithmetic operations on both physical and virtual bitmaps (see TECHNOLOGIES for more information)

9. Near real-time updates immediately available to queries and monitoring for alerts/notifications – watch-lists, event-triggers, subscriptions and thresholds.
10. Create almost any and all indexes, such as pre-aggregated, pre-calculated, composite, fuzzy, advanced text, Link Indexes™ and Embedded Value Indexes™, which allow data and metadata to be stored in indexes.
11. Using a combination of indexes and the appropriate standard data model terms, create VIRTUAL data warehouses/data marts for multi-dimensional queries.
12. Link Indexes™ add-on option to enable link mapping, allows for more advanced queries such as complex joins, degrees of separation, physical, logical and ontology model discovery, merge, combination, validation and presentation, and link analysis. The combination of normal content indexes and Link Indexes™ externally manages obvious/direct links and reveals non-obvious/indirect links among disparate data and information across multiple disparate data sources.
13. Other add-on options including open source or commercial categorization software, WhamTech’s eDiscovery tool, Teracase™, Intelligent Search’s fuzzy match, open source GATE entity extraction and WhamTech’s advanced text search.
14. Significant benefits compared to conventional adapters in federated systems, data warehouse or enterprise search – combines the best aspects of these approaches and overcomes the worst

For more information on WhamTech products, click here.