Optimizing Technology for Data Processing and Integration

DemystData’s engineering team recently began a substantial redesign of our technical infrastructure that will enable us to release faster, service more clients with unique needs (e.g., custom waterfall attributes, algorithmic attributes), and process more transactions (we add less than one second of overhead across all of our different vendors, with transaction times generally ranging from 0–3 seconds).

We predicated redesigning our infrastructure on achieving three key objectives:

Immutable Infrastructure: “On the fly” changes reduce stability and slow down deployments. With our design, live production infrastructure cannot be changed on the fly by engineers, but must always be committed, tested, and deployed following strict production change protocols.

Deployment Testing: As we modify our product, DemystData employs a “Blue/Green” deployment strategy in which a modified environment (blue) is built and deployed along side the live environment (green). Once the “blue” environment is determined to be stable, the “green” environment is decommissioned.

Scalable Infrastructure: Processing data transactions requires a production environment that may be scaled up and down based on resources and cost-efficiency. Simply put, certain clients require tens of thousands of transactions per week while others only require a few. Technology should be designed to serve a range of client needs.

As a side note, while building a solution to achieve these objectives, our engineering team is making vital contributions to the Linux Container Ecosystem, particularly a recipe for running Apache Marathon to deliver resource-based scheduling on CoreOS Linux (click here to see our technical thought leadership on this topic, shared through github).

The ultimate goal of technology platforms that serve financial services firms should be to make the process easy, fast, and secure, regardless of whether that technology is for asset allocation, data management, data integration (what we do at DemystData), or any one of the hundreds of other use-cases pertaining to financial services firms.

Here at DemystData, we’ve focused on these principles while building a nimble platform that can be optimized for any size client, from bulge bracket banks to alternative lenders.

Next week, we’ll be returning to more traditional “medium data” and “finance” topics, but highlighting the technology behind our products is crucial to understanding how we can truly transform the lending space through data.

Happy Friday!



Share on facebook
Share on twitter
Share on linkedin

More to explore

The Demyst Approach to “Agile Data”

Analogous to the contrast between waterfall and agile methodologies in software development, Agile Data is focused on achieving a minimum viable releasable improvement. Ignore the scrum masters for a moment; what allows organizations to follow

Read More »

Just Click Here

In today’s marketing landscape, the function of lead generation has expanded beyond the traditional mandate of procuring contact details and now requires that companies have an in-depth understanding of their customers’ firmographic data. Just Click

Read More »

Digital Verification APIs

Uber, Amazon, Netflix and every other breakout success of our age set the new standard in consumer engagement – leverage technology and data to enable 1 click to buy. They weren’t different processes, retailers previously

Read More »
Close Menu