Good engineers understand the importance of resilience. It’s the reason why Netflix built software that attacks its own network more than 1,000 times a week. Problems can come from anywhere — caused by internal systems or external networks — and clients don’t care why a service was disrupted, they care about having it back in working order. 

The Demyst platform was built to prioritize resilience. When we encountered a problem with a major data service earlier this week, the platform was ready to handle it. 

4 Steps to Fix a Data Problem

1. Inform the affected client(s). Three Demyst clients use APIs from a leading global information services vendor to pull data on Small and Medium Businesses (SMBs). When the integration team identified "Account Lock" errors for APIs used by all 3 clients, those clients were rapidly notified about the production issue.
   
   
2. Rule out suspicious activity. The APIs showed regular activity and no signs of a data breach. Passwords were set to expire on a staggered schedule, and there was no reason for all the clients to experience the same issue on the same day.
   
   
3. Confirm that your own systems are working. This was the first time that this type of error had been encountered on the Demyst platform. After performing internal checks, the root cause of the account lock was diagnosed as a bug in the data vendor’s system.
   
   
4. Work with the data vendor to fix the issue. Demyst contacted the vendor’s helpdesk to unlock the accounts and resume production. Detailed information had been collected about the problem, and because the issue had been experienced by multiple clients, it was easily reported for resolution.  

Fix the Problem, not the Blame

All vendors make mistakes, and the next production issue could come from anywhere. In this case, our valued partnership with the data provider allowed us to quickly resolve the problem together. 

The Demyst platform was designed with the resiliency necessary to manage these types of incidents. We were able to recognize the issue ahead of our clients and respond immediately.