Overview

The Demyst platform provides accelerated deployment of external data solutions for the world’s leading banks, insurers and fintechs. Demyst’s platform eliminates the friction associated with data procurement, testing, and deployment to unlock innovation and drive business growth. Personalized solutions for every business use case are made possible with access to 1,000’s of data providers worldwide via a single tool. Getting started is easy, access your free trial at www.demyst.com/sign-up.

Users can easily access data by simply creating a Data API from our data dictionaries found in the Dataset Catalog or Recipe Catalog. From there, you can insert the required inputs into the Data API and run your request to get an output consisting of a set of attributes.

Data selection: refine

The refine option allows the user to define custom attributes, which will be evaluated and returned in the response.

Example

The left-hand side of the JSON object is always an alphanumeric attribute key.

The right-hand side can be either of the following.

  • A JSON string — a path to an attribute.

    The custom attribute will contain the value at this path.

  • A JSON object from connector to a function

    Example

    { "domain_from_email": "$isMatch" }

    Returns

    true if the connector function matches

  • A JSON object from connector to a function

    Example

    { "seon_fraud_v2.data.fraud_score": { "$gte": 0.5 }}

    Returns

    true if the connector function matches

Example

          
            {
              "refine": {
                  "bic":       "experian_business_search_api.results[0].bic", // select attribute at path
                  "matched":   { "domain_from_email": "$isMatch" },                   // function on connector, returns true / false           
                  "is_fraud":  { "seon_fraud_v2.data.fraud_score": { "$gte": 0.5 }},   // function on an attribute, returns true / false
                  "sic_code":  { "$firstOf": [ "equifax_austin_tetra_details.primary_sic.sic_code", // function on list of attributes 
                                  "hosted_infogroup_business_places.results[0].primary_sic_code_id" ] }
                  }
            }
          
        

Connector preconditions: when

Connections with a when expression are only executed if the expression evaluates.

Syntax

An arbitrary expression on a connector or attribute level, identical with those options in refine

  • A JSON object from connector to a function

    Example

    { "domain_from_email": "$isMatch" }

    Returns

    true if the connector function matches

  • A JSON object from an attribute to a function

    Example

    { "seon_fraud_v2.data.fraud_score": { "$gte": 0.5 }}

    Returns

    true if the connector function matches

Example

            
              {
                "providers": {
                  "domain_from_email": {
                    "version": "$latest"
                  },
                  "seon_fraud_v2": {
                    "version": "$latest",
                    "when": {
                      "domain_from_email": "$failed"
                    }
                  }
                }
              }
            
          

seon_fraud_v2 will only be called after the domain_from_email connector failed.

Entity Resolution: match_key_conditions

The match_key_conditions option is available for hosted connectors (Connector IDs beginning with 'hosted') to specify the matching criteria from the inputs. This is helpful when the accuracy of your inputs are uncertain. For example, there may be a business that has changed locations but you only know their city and state. You can filter your matches to only return results with matching city and state.

Example

Use a key-value pair within the config section to implement match_key_conditions. Below is an example in which we are confident in the business_name, city, and state, while the post_code and street could be inaccurate since the business office may have moved recently.

          
            {
              "providers": {
                "hosted_experian_cpdb": {
                  "version": "$latest"
                }
              },
              "inputs": {
                "business_name": "DemystData",
                "street": "28 W 25th St",
                "city": "New York",
                "state": "NY"
                "post_code": "10010"
              },
              "config": {
                "match_key_conditions": ["business_name, city, state"]
              }
            }
          
        

The value associated with the match_key_conditions key is a JSON array. This array can include the string of comma-separated attributes as seen in the above example, or their equivalent 2-character encoding. Below are a set of possible values that an array can contain.

            
              2-character encoding, comma-separated string of attributes

              "00", "ssn"
              "01", "first_name, last_name, dob, whole_address"
              "02", "first_name, last_name, dob, post_code"
              "03", "first_name, last_name, whole_address"
              "04", "first_name, last_name, post_code"
              "05", "first_name, last_name, city, state"
              "06", "last_name, post_code, dob"
              "07", "whole_address, dob"
              "08", "first_name, email_address"
              "09", "first_name, phone"
              "10", "last_name, email_address"
              "11", "business_name, whole_address"
              "12", "business_name, post_code"
              "13", "business_name, city, state"
              "14", "phone"
              "15", "ein"
              "16", "domain"

              (match_key_conditions: ["04", "07"]) AND (match_key_conditions: ["first_name, last_name, post_code", "whole_address, dob"]) are the same!
            
          

Input waterfalls

For search products we often want to pass search results to another provider, we can solve this by specifying which attributes map to which input parameter. You could also pass a mix of attributes - from another provider output and an attribute from the initial inputs section

Syntax

  • A JSON object

  • The left-hand side will always be an attribute name

  • The right-hand side is a path to an attribute from an arbitrary connector or the attribute from inputs section

Example

          
            {
              "providers": {
                "tloxp_business_search": { "version": 2 },
                "tloxp_business_report": {
                  "version": "$latest",
                  "inputs": {
                    "business_id": "tloxp_business_search.business.id" // will be merged with other inputs given in the request
                    "country": "inputs.country"
                  }
                }
              },
                "inputs" : {
                    "country" : "US"
              }
            }
          
        

Wildcard Queries

Add a wildcard to a list/array structure to return the first filled attribute. This removes the dependency to find and specify location/index of the item in the list/array

Syntax

  • The symbol * included in the indexing operator

  • indexing operator must be within the right side in a path to an attribute for a connector

Example

          
            "refine": {
              "sic" : "hosted_infogroup_business_places.results[*].primary_sic_code_id"
            }
          
        

Function Library

Functions on connectors

$succeeded

Returns true if the connector succeeded

Example

          
            {"domain_from_email": "$succeeded"}
          
        
$failed

Returns true if the connector failed

Example

          
            {"domain_from_email": "$failed"}
          
        
$isMatch

Returns true if the connector succeeded, and has an is_hit attribute containing the true value

Example

          
            {"domain_from_email": "$isMatch"}
          
        
$noMatch

Returns true if the connector failed, or has succeeded but doesn’t have an is_hit attribute which contained the true value

Example

            
              {"domain_from_email": "$noMatch"}
            
          

Functions on attributes

$or

Example

          
            {"$or": [
                {"experian_business_facts.employee_size": {"$lte": 50}},
                {"experian_business_facts.employee_size": {"$lte": 100}}
              ]
            }
          
        
$and

Example

          
            {"$and": [
                {"experian_business_facts.employee_size": {"$gte": 20}},
                {"experian_business_facts.employee_size": {"$lte": 50}}
              ]
            }
          
        
$empty

Returns true if the attribute was empty.

Example

          
            { "experian_business_facts.ticker_symbol": "$empty" }
          
        
$nonEmpty

Returns true if the attribute was not empty.

Example

            
              { "experian_business_facts.ticker_symbol": "$nonEmpty" }
            
          
$eq json

Returns true if the attribute was not empty and equal to the passed in JSON node.

Example

          
            { "domain_from_email.host": { "$eq": "demystdata.com" }}
          
        
$in [json]

Returns true if the attribute was not empty and was equal to any of the given arguments.

Example

            
              { "hazardhub_risks_and_enhanced_property2.risks.wildfire.score": { "$in": ["A", "B", "C"] }}
            
          
$notIn

Example

          
            { "hazardhub_risks_and_enhanced_property2.risks.wildfire.score": {"$notIn": ["D", "E", "F"] }}
          
        
$gt json-number

Returns true if the attribute was not empty and > to the passed in JSON number.

Example

          
            { "experian_business_facts.employee_size": { "$gt": 75 }}
          
        
$gte json-number

Returns true if the attribute was not empty and ≥ to the passed in JSON number.

Example

          
            { "experian_business_facts.employee_size": { "$gte": 75 }}
          
        
$lt json-number

Returns true if the attribute was not empty and to the passed in JSON number.

Example

          
            { "experian_business_facts.employee_size": { "$lt": 20 }}
          
        

$lte json-number Returns true if the attribute was not empty and ≤ to the passed in JSON number.

Example

            
              { "experian_business_facts.employee_size": { "$lte": 20 }}
            
          
$between [json-number, json-number]

Returns true if the attribute was not empty and in the defined inclusive range.

Example

          
            { "experian_business_facts.employee_size": { "$between": [20, 75] }}
          
        

Arithmetic functions

$add

Returns the sum of the attribute and the passed in JSON number.

Example

          
            {"experian_business_facts.employee_size": { "$add": 10 }}
          
        
$subtract

Returns the difference of the attribute and the passed in JSON number.

Example

          
            {"experian_business_facts.employee_size": { "$subtract": 5 }}
          
        
$multiply

Returns the product of the attribute and the passed in JSON number.

Example

          
            {"experian_business_facts.sales_revenue": { "$multiply": 1.5 }}
          
        
$divide

Returns the quotient of the attribute and the passed in JSON number.

Example

          
            {"experian_business_facts.sales_revenue": { "$divide": 12 }}
          
        

Top level functions

$firstOf

Returns the first non-empty attribute of list of attributes

Arguments

  • A list of paths to connectors + attributes

Returns

  • An attribute, which will be serialized in JSON

Example

              
                { "$firstOf": [ "equifax_austin_tetra_details.primary_sic.sic_code",
                                "hosted_infogroup_business_places.results[0].primary_sic_code_id" ] }
              
            
$allEmpty

Returns true if all of the given attributes are empty

Arguments

  • A list of paths to connectors + attributes

Returns

  • true

Example

              
                { "$allEmpty": [ "equifax_austin_tetra_details.primary_sic.sic_code",
                                "hosted_infogroup_business_places.results[0].primary_sic_code_id" ] }
              
            
$anyEmpty

Returns true if any of the given attributes are empty

Arguments

  • A list of paths to connectors + attributes

Returns

  • true

Example

              
                { "$anyEmpty": [ "equifax_austin_tetra_details.primary_sic.sic_code",
                                "hosted_infogroup_business_places.results[0].primary_sic_code_id" ] }
              
            
$map

Replace upstream attribute values with custom ones. Usable in combination with other top-level functions.

Example - Mapping with the output of $firstOf

              
              {
                "$firstOf": [
                  "domain_from_email.details.demographics.living_status",
                  "provider_5.home_owner_status"
                ],
                "$map": {
                  "r":         "renter",
                  "rent":      "renter",
                  "o":         "owner",
                  "homeowner": "owner",
                  "own":       "owner"
                }
              }
              
              
            

Example - distinct attribute values with similar codes

This update allows us to map the output of the individual attributes that are passed into $firstOf. The example below shows that for the same code, C006 from Quantarium and Attom, different values could be mapped.

              
              {
                "providers": {
                  "attomdata_attom_id": {
                    "version": "$latest"
                  },
                  "hosted_attom_residential_tax_assessor": {
                    "version": "$latest",
                    "inputs": {
                      "attom_id": "attomdata_attom_id.attom_id"
                    },
                    "when": {
                      "attomdata_attom_id": "$isMatch"
                    }
                  },
                  "hosted_quantarium_open_lien": {
                    "version": "$latest"
                  }
                },
                "inputs": {
                  "street": "221 Clinton Ave",
                  "city": "brooklyn",
                  "state": "ny",
                  "post_code": "11205",
                  "country": "us"
                },
                "refine": {
                  "route_demo": {
                    "$firstOf": [
                      {
                        "$at": "hosted_attom_residential_tax_assessor.results[0].contact_owner_mail_address_crrt",
                        "$map": {
                          "C006": "City"
                        }
                      },
                      {
                        "$at": "hosted_quantarium_open_lien.results[0].pa_carrier_route",
                        "$map": {
                          "C006": "County"
                        }
                      }
                    ],
                    "$map": {
                      "C001": "Street",
                      "C002": "Road"
                    }
                  }
                }
              }
              
            

Arguments

  • A mapping from upstream → custom attributes.

Returns

  • The custom attribute, or the upstream attribute if no argument matched.

Notes

  • $map needs to be called together with another function. If you only want to change a single attribute, you can use the $at function.

$at

Returns the attribute at the given path.

Example

              
                {
                  "$at": "fullcontact_person_enrich.details.demographics.living_status", 
                    "$map": {
                      "r": "renter",
                      "rent": "renter",
                      "o": "owner",
                      "homeowner": "owner",
                      "own": "owner"
                    }
                }
              
            

Arguments

  • A mapping from upstream → custom attributes.

Returns

              
                "providers": {
                  "equifax_austin_tetra_discovery": {
                    "error_simulation": "upstream_structure",
                    "version": "$latest"
                  }
                }
              
            
  • The custom attribute. Useful together with the $map function.

Error Simulations: simulation

The simulation option allows to simulate an error for the provider in the request

Syntax

  • A JSON object from a connector to an error type

    Example

    { "acxiom_place": "upstream_structure"}

    Returns

    type and message for the error raised

Example

          
            "simulation": {
              "acxiom_place": "upstream_structure",
              "hazardhub_risks_and_enhanced_property2": "upstream_http"
            }
          
        

Per Provider

For each provider in the request, you can simulate an error without having to add `simulate` option separately. The error in the response will remain the same

Example

          
            "providers": {
              "equifax_austin_tetra_discovery": {
                    "error_simulation": "upstream_structure",
                    "version": "$latest"
                  }
            }
          
        

Syntax

  • A JSON object from error_simulation to an error type within the provider object

    Example

                  
                    { "error_simulation": "upstream_structure"}
                  
                

    Returns

    type and message for the error raised

Errors Library

upstream_structure

XML/JSON parsing failure, unexpected elements in upstream response or unexpected upstream structure

          
            "error": {
              "type": "unexpected_upstream_structure",
              "message": "We’ve encountered an issue while communicating with an upstream connector."
            }
          
        
upstream_http

Server error from provider, example - HTTP status code 500 unexpected

          
            "error": {
              "type": "unexpected_upstream_http_status",
              "message": "HTTP status code 500 with meaning 'Internal Server Error. A generic error message, given when an unexpected condition was encountered and no more specific message is suitable.' unexpected."
            }
          
        
insufficient_credentials

The data provider(s) you tried to reach did not receive sufficient valid credentials to execute.

          
            "error": {
              "type": "insufficient_credentials",
              "message": "Missing credentials to run this provider."
            }
          
        
invalid_credentials

The credentials either provided by the client or by Demyst to access to a particular data provider are incorrect or out of date.

          
            "error": {
              "type": "invalid_credentials",
              "message": "Request was not authorized, the credentials you provided for this data source are likely invalid."
            }
          
        
upstream_service_unavailable

The credentials either provided by the client or by Demyst to access to a particular data provider are incorrect or out of date.

          
            "error": {
              "type": "upstream_service_unavailable",
              "message": "The data source is experiencing issues"
            }
          
        
rate_limit_exceeded

Providers having monthly / concurrency rate limit, if any, exceeded

          
            "error": {
              "type": "rate_limit_exceeded",
              "message": "Rate limit exceeded"
            }
          
        
sample_data_not_implemented

Provider does not have SampleData and errors out using test key/sample mode

          
            "error": {
              "type": "sample_data_not_implemented",
              "message": "Provider experian_business_judgments does not have SampleData"
            }
          
        
unexpected_provider_error

The data provider you are attempting to access threw an unexpected error.

          
            "error": {
              "type": "unexpected_error",
              "message": "An unexpected issue occurred"
            }
          
        
timeout

Timeout exceeded. Default timeout is 100 seconds, can be extended

          
            "error": {
              "type": "timeout",
              "message": "Timeout of 1000 ns exceeded"
            }
          
        

Debugging Config API Coded Errors

waterfall_dependency_error

User has specified a provider as an input / when condition, but this provider is not part of the providers in the request / saved data API

          
          {
            "transaction_id": "bebbe9b8-7a40-4e1e-8cb0-d3c8432617d4",
            "error": {
                "type": "waterfall_dependency_error",
                "message": "Provider experian_business_facts referenced a non-existing provider hosted_experian_cpdb"
            }
          }
          
       
refine_error

Happens before the providers are executed if a function doesn’t exist, or wrong parameters were passed

          
          {
            "transaction_id": "2a3ca703-cdf3-4f29-a4d4-57b24003a2e6",
            "error": {
                "type": "refine_error",
                "message": "Unknown function 'atis' on Transaction. Did you mean $at?"
            }
          }
          
        
decoding_error

An incorrect syntax, an unwanted character within the API call that won't let the request be evaluated until corrected will be propagated as a decoding_error

          
          {
            "transaction_id": "1c6b6350-4550-4cd1-a7cd-29055e2a791c",
            "error": {
                "type": "decoding_error",
                "message": "expected \" got '}\n    ...' (line 6, column 9)"
            }
          }
          
          {
           "transaction_id": "a9e2f44b-509c-4fc9-aa9e-01bb439c8f96",
           "error": {
            "type": "decoding_error",
            "message": "Not a string value: DownField(providers)"
            }
          }
        
          {
            "transaction_id": "ede8349c-d760-455a-9fdd-7997aef0041e",
            "error": {
                "type": "decoding_error",
                "message": ".providers: While parsing connector-attribute-path 'true': connector-attribute path requires a '.'"
            }
          }
          
        
dependency_error

This error occurs when a provider is set to receive inputs from the output of another provider, but that provider was Skipped.

          
          {
            "transaction_id": "6c48013f-bdee-4ef0-a2e7-5b2e531df15e",
            "output": {
            "PROVIDER_NAME": {
              "error": {
                "type": "dependency_error",
                "message": "Connector 'PROVIDER_NAME' had run state: 'Skipped'"
                }
              }
            }
          }