The DemystGateway is a cloud-based, Data Gateway designed to safely accelerate integration time, provide deep insights into the quality and performance of the data, and help fulfill the full potential of these sources.
This blog series dives deep into the features of the DemystGateway and unpacks each of these areas in great detail, bringing you real world examples and use cases.
First we’d like to walk you through the entire platform to show you the breadth of capabilities the gateway will bring to your organization.
1. Data Acceleration
If you’ve integrated five different data sources in the past you’ve undoubtedly experienced five different ways to represent an address, five different ways to show a record wasn’t found, and five different ways to serialize a profile. You’ve probably parsed XML and JSON, integrated REST and SOAP APIs and authenticated with API keys, OAuth tokens and more. You’ve differentiated between and handled error codes like 400, 401, 404, 500, 502 and many, many more. The DemystGateway standardizes and streamlines all external API connections to give you a single common interface to interact with all the data needed to power your business.
Data scientists generally spend most of their time cleaning and organizing data. Whether it’s the format of fields like Phone Numbers, States and Dates, or treatment of missing values, or how a nested document object is turned into a flattened CSV, each data provider has its own idiosyncrasies and semantics that need to be understood.
The Demyst Smart Schema ingests each integrated data provider’s bespoke representation of data and applies a consistent, standardized shaping to the data. This schema conforms all of these varied data elements to immediately actionable attributes your teams can start creating value from today.
Single Integration Point
Writing software that concurrently connects to dozens of data providers is difficult enough, but ensuring that each of those requests meets your strict SLAs and appropriately handles timeouts and errors, is much more complex. This complexity begins to compound when you add the myriad approaches to both authorization (Basic/Digest HTTP Auth, Client Certificates, Tokens, Oauth) and serialization (JSON, XML, CSV).
By leveraging the DemystGateway your teams can shift focus from the minutiae of integration, to spending time and resources on core business needs and differentiated development efforts. A single integration to access the vast sea of data ready to enrich your applications.
Reliable Test Data
Whether it’s a large US bureau or a small alternative data provider, we’ve seen time and time again that providers don’t always appreciate the need for test facilities as much as our customers do.
Innovation in financial applications cannot come at the cost of safety and security to users, which is why the DemystGateway provides a comprehensive set of testing capabilities. Our APIs are designed to work alongside your Continuous Integration, System Integration and Acceptance Testing phases to provide the validation your applications need.
Standardized Error Handling
You’d be hard pressed to find a real-time data provider that isn’t serving their data over HTTP these days. But once you peel back that common layer you will find anything but standards. Unfortunately providers seem to find new and unique ways to represent malformed/missing parameters, empty results, fatal errors and timeouts. This variation all needs to be accounted for, but regularly we’ve seen corners cut and models affected by these all too frequent edge cases.
The DemystGateway brings these edge cases front and center with easy to use, standardized API semantics and rich reporting capabilities.
2. Data Insights
While every integration should involve the tracking, monitoring and close analysis of a number of different data points, in practice we find that as schedules slip, operational and reporting capabilities are de-prioritized. These capability gaps mean critical integrations aren’t managed properly nor leveraged to their full potential.
By centralizing all of your integrations through one secure, highly scalable gateway you unlock tremendous insights and operational capability – both via high level top-down dashboards and granular record by record reporting.
Response Time Monitoring and Alerting
Decision logic is only as good as the data it relies on. Ensuring your data providers are returning results under their SLA is a critical part of a production data integration. At DemystData, we’ve seen it all and understand how hard it can be to communicate and remediate these one-time or ongoing issues with providers.
The DemystGateway gives you the ability to track and alert on SLA violations across unlimited providers from a single console. And because all of this meta information is stored in our reporting systems you can be confident that you can call up these critical data points at anytime … like, say for your yearly contract renegotiation.
Each time data moves outside of your compute infrastructure, it should be tracked. Whether it’s an Email Address or an SSN, your organization has trusted a provider to receive and handle your customer’s personal information with the utmost care, but all too often organizations are unable to answer basic questions about where data has been sent and how it has been handled.
The reporting capabilities of the DemystGateway are able to give you insights into what kind of data is being sent outside your infrastructure, where it’s going and what permitted purpose it was sent under. This allows you to pull up required information in case of an audit, or better assess and manage risks around data privacy and potential exposure.
The power of real-time data lies in its ability to help organizations make better decisions by ensuring the most up to date information is being used. In today’s world data changes too quickly to rely on static files that update quarterly or annually. However, with this shift to real-time data comes the responsibility to track and understand how that data changes over time. Data consumers need to be confident the stability and performance of data points and scores will be stable over time and need to be alerted when that changes so rules and models can be re-tuned accordingly.
The DemystGateway gives unrivaled insights into the stability and distribution of attributes your applications rely on. These insights enable your teams to make better informed decisions about how to leverage the vast quantities of data accessible through our platform.
3. Data Value Expansion
So far we’ve shown you how the DemystGateway can accelerate your integrations ensuring that your data is sound and production ready, but the gateway not only makes data access easier but also makes data more powerful once you have it.
Testing and implementing a new decision rule can be extremely time consuming. Between the technical requirements to set-up and track test groups and the later process of pulling and analyzing the data, what should be a simple comparison can end up taking months.
When your production integrations go through the DemystGateway you make automatic use of Data Channels. Channels are input data demultiplexers that allow you to send your real-time production data to unlimited parallel provider sets. The functionality allows you to seamlessly test alternative sources and strategies without disrupting production systems. If or when you decide to switch to an alternate configuration the DemystGateway means you can do so with limited overhead costs.
Analyzing a new data provider can be extremely time consuming. Between the technical hurdles and legal requirements, gaining approval to gather tens of thousands of customer records to be enriched for analysis can take months of work for a single provider. By the time you’ve successfully tested a new provider the conditions under which it was tested are frequently no longer valid, meaning the results of the test are now meaningless.
An additional use of Data Channels is to passively collect and test, in real-time, additional data sources and attributes. The DemystGatewayallows organizations to move from a periodic, sequential, one-off testing of sources and strategies to ongoing rolling testing that can be adjusted or implemented at any time. Additionally, because all of our providers have an accompanying Smart Schema new data can be easily swapped in without interrupting downstream processing.
Continuously available rich, real-time data is useless if the algorithms that leverage them are years old. Companies have invested incredible amounts of money on data science teams to push innovation, but time and time again we’ve found our customer’s’ own internal processes get in the way of that goal.
Data Functions are DemystData’s answer to this problem.
Data Functions are horizontally scalable, independent functions that can be embedded in DataChannels. Customers are given total control of development and deployment and with rich A/B testing capability, customers can be sure that developed functions are able to be continuously monitored and tested. Data Functions are written in Python and can be used to execute a decision tree, ruleset, model, or other custom code. Data Functions can also be used to create waterfall sequences of provider queries or automatic failover in the case a single provider is down or has missing data.
Please check back in as we dive head first into each of these topics, and if you have any questions or would like to schedule a demo, let us know.