Gateway Series – Inside the DemystGateway (Part I)

The DemystGateway is a cloud-based, Data Gateway designed to safely accelerate integration time, provide deep insights into the quality and performance of the data, and help fulfill the full potential of these sources.

This blog series dives deep into the features of the DemystGateway and unpacks each of these areas in great detail, bringing you real world examples and use cases.

First we’d like to walk you through the entire platform to show you the breadth of capabilities the gateway will bring to your organization.

1. Data Acceleration

If you’ve integrated five different data sources in the past you’ve undoubtedly experienced five different ways to represent an address, five different ways to show a record wasn’t found, and five different ways to serialize a profile. You’ve probably parsed XML and JSON, integrated REST and SOAP APIs and authenticated with API keys, OAuth tokens and more. You’ve differentiated between and handled error codes like 400, 401, 404, 500, 502 and many, many more. The DemystGateway standardizes and streamlines all external API connections to give you a single common interface to interact with all the data needed to power your business.

Smart Schema

Data scientists generally spend most of their time cleaning and organizing data. Whether it’s the format of fields like Phone Numbers, States and Dates, or treatment of missing values, or how a nested document object is turned into a flattened CSV, each data provider has its own idiosyncrasies and semantics that need to be understood.

Smart Schema

The Demyst Smart Schema ingests each integrated data provider’s bespoke representation of data and applies a consistent, standardized shaping to the data. This schema conforms all of these varied data elements to immediately actionable attributes your teams can start creating value from today.

Single Integration Point
Single Integration Point

Writing software that concurrently connects to dozens of data providers is difficult enough, but ensuring that each of those requests meets your strict SLAs and appropriately handles timeouts and errors, is much more complex. This complexity begins to compound when you add the myriad approaches to both authorization (Basic/Digest HTTP Auth, Client Certificates, Tokens, Oauth) and serialization (JSON, XML, CSV).

By leveraging the DemystGateway your teams can shift focus from the minutiae of integration, to spending time and resources on core business needs and differentiated development efforts. A single integration to access the vast sea of data ready to enrich your applications.

Reliable Test Data

Whether it’s a large US bureau or a small alternative data provider, we’ve seen time and time again that providers don’t always appreciate the need for test facilities as much as our customers do.

Innovation in financial applications cannot come at the cost of safety and security to users, which is why the DemystGateway provides a comprehensive set of testing capabilities. Our APIs are designed to work alongside your Continuous Integration, System Integration and Acceptance Testing phases to provide the validation your applications need.

Standardized Error Handling

Standardized Error Handling

You’d be hard pressed to find a real-time data provider that isn’t serving their data over HTTP these days. But once you peel back that common layer you will find anything but standards. Unfortunately providers seem to find new and unique ways to represent malformed/missing parameters, empty results, fatal errors and timeouts. This variation all needs to be accounted for, but regularly we’ve seen corners cut and models affected by these all too frequent edge cases.

The DemystGateway brings these edge cases front and center with easy to use, standardized API semantics and rich reporting capabilities.

2. Data Insights

While every integration should involve the tracking, monitoring and close analysis of a number of different data points, in practice we find that as schedules slip, operational and reporting capabilities are de-prioritized. These capability gaps mean critical integrations aren’t managed properly nor leveraged to their full potential.

By centralizing all of your integrations through one secure, highly scalable gateway you unlock tremendous insights and operational capability – both via high level top-down dashboards and granular record by record reporting.

Response Time Monitoring and Alerting

Decision logic is only as good as the data it relies on. Ensuring your data providers are returning results under their SLA is a critical part of a production data integration. At DemystData, we’ve seen it all and understand how hard it can be to communicate and remediate these one-time or ongoing issues with providers.

Response Time Monitoring and Alerting

The DemystGateway gives you the ability to track and alert on SLA violations across unlimited providers from a single console. And because all of this meta information is stored in our reporting systems you can be confident that you can call up these critical data points at anytime … like, say for your yearly contract renegotiation.

Data Privacy

Each time data moves outside of your compute infrastructure, it should be tracked. Whether it’s an Email Address or an SSN, your organization has trusted a provider to receive and handle your customer’s personal information with the utmost care, but all too often organizations are unable to answer basic questions about where data has been sent and how it has been handled.

The reporting capabilities of the DemystGateway are able to give you insights into what kind of data is being sent outside your infrastructure, where it’s going and what permitted purpose it was sent under. This allows you to pull up required information in case of an audit, or better assess and manage risks around data privacy and potential exposure.

Attribute Distributions

The power of real-time data lies in its ability to help organizations make better decisions by ensuring the most up to date information is being used. In today’s world data changes too quickly to rely on static files that update quarterly or annually. However, with this shift to real-time data comes the responsibility to track and understand how that data changes over time. Data consumers need to be confident the stability and performance of data points and scores will be stable over time and need to be alerted when that changes so rules and models can be re-tuned accordingly.

The DemystGateway gives unrivaled insights into the stability and distribution of attributes your applications rely on. These insights enable your teams to make better informed decisions about how to leverage the vast quantities of data accessible through our platform.

3. Data Value Expansion

So far we’ve shown you how the DemystGateway can accelerate your integrations ensuring that your data is sound and production ready, but the gateway not only makes data access easier but also makes data more powerful once you have it.

A/B Testing

Testing and implementing a new decision rule can be extremely time consuming. Between the technical requirements to set-up and track test groups and the later process of pulling and analyzing the data, what should be a simple comparison can end up taking months.

A/B Testing

When your production integrations go through the DemystGateway you make automatic use of Data Channels. Channels are input data demultiplexers that allow you to send your real-time production data to unlimited parallel provider sets. The functionality allows you to seamlessly test alternative sources and strategies without disrupting production systems. If or when you decide to switch to an alternate configuration the DemystGateway means you can do so with limited overhead costs.

Passive Analysis

Analyzing a new data provider can be extremely time consuming. Between the technical hurdles and legal requirements, gaining approval to gather tens of thousands of customer records to be enriched for analysis can take months of work for a single provider. By the time you’ve successfully tested a new provider the conditions under which it was tested are frequently no longer valid, meaning the results of the test are now meaningless.


An additional use of Data Channels is to passively collect and test, in real-time, additional data sources and attributes. The DemystGatewayallows organizations to move from a periodic, sequential, one-off testing of sources and strategies to ongoing rolling testing that can be adjusted or implemented at any time. Additionally, because all of our providers have an accompanying Smart Schema new data can be easily swapped in without interrupting downstream processing.

Data Functions

Continuously available rich, real-time data is useless if the algorithms that leverage them are years old. Companies have invested incredible amounts of money on data science teams to push innovation, but time and time again we’ve found our customer’s’ own internal processes get in the way of that goal.

Data Functions are DemystData’s answer to this problem.

Data Functions

Data Functions are horizontally scalable, independent functions that can be embedded in DataChannels. Customers are given total control of development and deployment and with rich A/B testing capability, customers can be sure that developed functions are able to be continuously monitored and tested. Data Functions are written in Python and can be used to execute a decision tree, ruleset, model, or other custom code. Data Functions can also be used to create waterfall sequences of provider queries or automatic failover in the case a single provider is down or has missing data.

Please check back in as we dive head first into each of these topics, and if you have any questions or would like to schedule a demo, let us know.



Share on facebook
Share on twitter
Share on linkedin

More to explore

The Demyst Approach to “Agile Data”

Analogous to the contrast between waterfall and agile methodologies in software development, Agile Data is focused on achieving a minimum viable releasable improvement. Ignore the scrum masters for a moment; what allows organizations to follow

Read More »

Just Click Here

In today’s marketing landscape, the function of lead generation has expanded beyond the traditional mandate of procuring contact details and now requires that companies have an in-depth understanding of their customers’ firmographic data. Just Click

Read More »

Digital Verification APIs

Uber, Amazon, Netflix and every other breakout success of our age set the new standard in consumer engagement – leverage technology and data to enable 1 click to buy. They weren’t different processes, retailers previously

Read More »
Close Menu