Overview
The Demyst platform provides accelerated deployment of external data solutions for the world's leading banks, insurers and fintechs. Demyst's platform eliminates the friction associated with data procurement, testing, and deployment to unlock innovation and drive business growth. Personalized solutions for every business use case are made possible with access to 1,000's of data providers worldwide via a single tool. Getting started is easy, access your free trial at www.demyst.com/sign-up.
Users can easily access data by simply creating a Data API from our data dictionaries found in the Dataset Catalog or Recipe Catalog. From there, you can insert the required inputs into the Data API and run your request to get an output consisting of a set of attributes.
Data selection: refine
The refine option allows the user to define custom attributes, which will be evaluated and returned in the response.
Example
The left-hand side of the JSON object is always an alphanumeric attribute key.
The right-hand side can be either of the following.
-
A JSON string — a path to an attribute.
The custom attribute will contain the value at this path.
-
A JSON object from connector to a function
Example
{ "domain_from_email": "$isMatch" }
Returns
true if the connector function matches
-
A JSON object from connector to a function
Example
{ "seon_fraud_v2.data.fraud_score": { "$gte": 0.5 }}
Returns
true if the connector function matches
Example
{
"refine": {
"bic": "experian_business_search_api.results[0].bic", // select attribute at path
"matched": { "domain_from_email": "$isMatch" }, // function on connector, returns true / false
"is_fraud": { "seon_fraud_v2.data.fraud_score": { "$gte": 0.5 }}, // function on an attribute, returns true / false
"sic_code": { "$firstOf": [ "equifax_austin_tetra_details.primary_sic.sic_code", // function on list of attributes
"hosted_infogroup_business_places.results[0].primary_sic_code_id" ] }
}
}
Connector preconditions: when
Connections with a when expression are only executed if the expression evaluates.
Syntax
An arbitrary expression on a connector or attribute level, identical with those options in refine
-
A JSON object from connector to a function
Example
{ "domain_from_email": "$isMatch" }
Returns
true if the connector function matches
-
A JSON object from an attribute to a function
Example
{ "seon_fraud_v2.data.fraud_score": { "$gte": 0.5 }}
Returns
true if the connector function matches
Example
{
"providers": {
"domain_from_email": {
"version": "$latest"
},
"seon_fraud_v2": {
"version": "$latest",
"when": {
"domain_from_email": "$failed"
}
}
}
}
seon_fraud_v2 will only be called after the domain_from_email connector failed.
Entity Resolution: match_key_conditions
The match_key_conditions option is available for hosted connectors (Connector IDs beginning with 'hosted') to specify the matching criteria from the inputs. This is helpful when the accuracy of your inputs are uncertain. For example, there may be a business that has changed locations but you only know their city and state. You can filter your matches to only return results with matching city and state.
If match_key_conditions is provided, during entity resolution we will only make queries that use the key groupings in the specified match key conditions. Without this specification, we will make all possible entity resolution queries.
Example
Use a key-value pair within the config section to implement match_key_conditions. Below is an example in which we are confident in the business_name, city, and state, while the post_code and street could be inaccurate since the business office may have moved recently.
{
"providers": {
"hosted_experian_cpdb": {
"version": "$latest"
}
},
"inputs": {
"business_name": "DemystData",
"street": "28 W 25th St",
"city": "New York",
"state": "NY"
"post_code": "10010"
},
"config": {
"match_key_conditions": ["business_name, city, state"]
}
}
The value associated with the match_key_conditions key is a JSON array. This array can include the string of comma-separated attributes as seen in the above example, or their equivalent 2-character encoding. Multiple conditions should be formatted as comma-separated strings.
{
...
"config": {
"match_key_conditions": [
"business_name, city, state",
"phone"
]
}
}
Below are a set of possible values that a match_key_conditions array may contain.
2-character encoding, comma-separated string of attributes
"00", "ssn"
"01", "first_name, last_name, dob, whole_address"
"02", "first_name, last_name, dob, post_code"
"03", "first_name, last_name, whole_address"
"04", "first_name, last_name, post_code"
"05", "first_name, last_name, city, state"
"06", "last_name, post_code, dob"
"07", "whole_address, dob"
"08", "first_name, email_address"
"09", "first_name, phone"
"10", "last_name, email_address"
"11", "business_name, whole_address"
"12", "business_name, post_code"
"13", "business_name, city, state"
"14", "phone"
"15", "ein"
"16", "domain"
As you can see from the 2-character encoding
"match_key_conditions": ["04", "07"]
AND
"match_key_conditions": ["first_name, last_name, post_code", "whole_address, dob"]
are exactly the same! Use whichever best suits you.
Result
Whether or not match_key_conditions are provided as input qualifiers, hosted connectors will specify upon which attributes matching occurred through an output key named match_keys. Match keys might be Demyst Types, or attributes specific to the connector.
If we matched on a person’s nickname or a business name alias, rather than the literal queried value for that field (i.e. first_name or business_name), we will output "common_nickname" or "business_name_aka" in the match_keys string instead of "first_name" or "business_name". This will apply to cases where an input “first_name” was Tom instead of Thomas, and so on.
Look for something like the following in your Demyst API responses to find what keys we matched on from your inputs.
{
"transaction_id": "123abc-456xyz",
"output": {
"hosted_connector": {
"data": {
"is_hit": true,
"results": {
...
"match_keys": "common_nickname,
last_name,
post_code,
primary_contact_first_name,
primary_contact_last_name",
...
},
},
},
},
}
Input waterfalls
For search products we often want to pass search results to another provider, we can solve this by specifying which attributes map to which input parameter. You could also pass a mix of attributes - from another provider output and an attribute from the initial inputs section
Syntax
-
A JSON object
-
The left-hand side will always be an attribute name
-
The right-hand side is a path to an attribute from an arbitrary connector or the attribute from inputs section
Example
{
"providers": {
"tloxp_business_search": { "version": 2 },
"tloxp_business_report": {
"version": "$latest",
"inputs": {
"business_id": "tloxp_business_search.business.id" // will be merged with other inputs given in the request
"country": "inputs.country"
}
}
},
"inputs" : {
"country" : "US"
}
}
Wildcard Queries
Add a wildcard to a list/array structure to return the first filled attribute. This removes the dependency to find and specify location/index of the item in the list/array
Syntax
-
The symbol * included in the indexing operator
-
indexing operator must be within the right side in a path to an attribute for a connector
Example
"refine": {
"sic" : "hosted_infogroup_business_places.results[*].primary_sic_code_id"
}
Function Library
Functions on connectors
$succeededReturns true if the connector succeeded
Example
{"domain_from_email": "$succeeded"}
$failed
Returns true if the connector failed
Example
{"domain_from_email": "$failed"}
$isMatch
Returns true if the connector succeeded, and has an is_hit attribute containing the true value
Example
{"domain_from_email": "$isMatch"}
$noMatch
Returns true if the connector failed, or has succeeded but doesn't have an is_hit attribute which contained the true value
Example
{"domain_from_email": "$noMatch"}
Functions on attributes
$orCombines multiple conditions on connectors, returning true / false
Example
"""
Let's say we have a client looking for any home
that has either a pool or a chicken coop.
"""
{
...,
"refine": {
"has_pool_or_chickens": {
"$or": [
{ "hosted_attom_residential_tax_assessor.results[*].poultry_house_flag": { "$eq": true } },
{ "hosted_attom_residential_tax_assessor.results[*].pool_house_flag": { "$eq": true } }
]
}
}
}
$and
Combines multiple conditions on connectors, returning true / false
Example
"""
Perhaps we want to make sure we're properly insuring a home, so lets
verify that it has 3 or more bedrooms and falls within a square footage range.
"""
{
...,
"refine": {
"building_requirements": {
"$and": [
{ "housecanary_property_details_enhanced.details_enhanced.public_record.number_of_bedrooms": { "$gte": 3 } },
{ "housecanary_property_details_enhanced.details_enhanced.public_record.building_area_sqft": { "$between": [1900, 2700] } }
]
}
}
}
Example - Using $and and $or
"""
In this example we want to identify charities invested in advancing
education and social/public welfare, or education and health.
"""
{
...,
"refine": {
"education_and_health_or_spw": {
"$and": [
{ "hosted_acnc_registered_charities.results[*].advancing_education": { "$eq": true } },
{ "$or": [
{ "hosted_acnc_registered_charities.results[*].advancing_health": { "$eq": true } },
{ "hosted_acnc_registered_charities.results[*].advancing_social_or_public_welfare": { "$eq": true } }
]
}
]
}
}
}
$empty
Returns true if the attribute was empty.
Example
{ "experian_business_facts.ticker_symbol": "$empty" }
$nonEmpty
Returns true if the attribute was not empty.
Example
{ "experian_business_facts.ticker_symbol": "$nonEmpty" }
$eq json
Returns true if the attribute was not empty and equal to the passed in JSON node.
Example
{ "domain_from_email.host": { "$eq": "demystdata.com" }}
$in [json]
Returns true if the attribute was not empty and was equal to any of the given arguments.
Example
{ "hazardhub_risks_and_enhanced_property2.risks.wildfire.score": { "$in": ["A", "B", "C"] }}
$notIn
Example
{ "hazardhub_risks_and_enhanced_property2.risks.wildfire.score": {"$notIn": ["D", "E", "F"] }}
$gt json-number
Returns true if the attribute was not empty and > to the passed in JSON number.
Example
{ "experian_business_facts.employee_size": { "$gt": 75 }}
$gte json-number
Returns true if the attribute was not empty and ≥ to the passed in JSON number.
Example
{ "experian_business_facts.employee_size": { "$gte": 75 }}
$lt json-number
Returns true if the attribute was not empty and to the passed in JSON number.
Example
{ "experian_business_facts.employee_size": { "$lt": 20 }}
$lte json-number Returns true if the attribute was not empty and ≤ to the passed in JSON number.
Example
{ "experian_business_facts.employee_size": { "$lte": 20 }}
$between [json-number, json-number]
Returns true if the attribute was not empty and in the defined inclusive range.
Example
{ "experian_business_facts.employee_size": { "$between": [20, 75] }}
Interpolated Strings
Demyst allows combining of results together into one attribute within refine (and within the input waterfalls section). Separators can be added right between the attributes.
Example
{
...,
"refine": {
"complete_address": "${companies_house_company_profile.registered_office_address.address_line_1}, ${companies_house_company_profile.registered_office_address.address_line_2}"
}
}
Result
{
"refine": {
"complete_address": "Cms Cameron Mckenna Llp Cannon Place, 78 Cannon Street"
}
}
Should you be working within $extract or $dictionary, this functionality can take over the $at attribute path within select.
Example
{
...,
"refine": {
"my_schema": {
"$extract": {
"list_path": "companies_house_company_search.items",
"select": {
"full_address": "${address.address_line_1}, ${address.address_line_2}"
}
}
}
}
}
You can also mix in any constant values you'd like outside of brackets. This may be used with or without separators.
{
...,
"refine": {
"email_address": "${domain_from_email.user}@gmail.com"
}
}
Arithmetic functions
$addReturns the sum of the attribute and the passed in JSON number.
Example
{"experian_business_facts.employee_size": { "$add": 10 }}
$subtract
Returns the difference of the attribute and the passed in JSON number.
Example
{"experian_business_facts.employee_size": { "$subtract": 5 }}
$multiply
Returns the product of the attribute and the passed in JSON number.
Example
{"experian_business_facts.sales_revenue": { "$multiply": 1.5 }}
$divide
Returns the quotient of the attribute and the passed in JSON number.
Example
{"experian_business_facts.sales_revenue": { "$divide": 12 }}
Top level functions
$firstOfReturns the first non-empty attribute of list of attributes
Arguments
-
A list of paths to connectors + attributes
Returns
-
An attribute, which will be serialized in JSON
Example
{ "$firstOf": [ "equifax_austin_tetra_details.primary_sic.sic_code",
"hosted_infogroup_business_places.results[0].primary_sic_code_id" ] }
$allEmpty
Returns true if all of the given attributes are empty
Arguments
-
A list of paths to connectors + attributes
Returns
-
true
Example
{ "$allEmpty": [ "equifax_austin_tetra_details.primary_sic.sic_code",
"hosted_infogroup_business_places.results[0].primary_sic_code_id" ] }
$anyEmpty
Returns true if any of the given attributes are empty
Arguments
-
A list of paths to connectors + attributes
Returns
-
true
Example
{ "$anyEmpty": [ "equifax_austin_tetra_details.primary_sic.sic_code",
"hosted_infogroup_business_places.results[0].primary_sic_code_id" ] }
$map
Replace upstream attribute values with custom ones. Usable in combination with other top-level functions.
Example - Mapping with the output of $firstOf
{
"$firstOf": [
"domain_from_email.details.demographics.living_status",
"provider_5.home_owner_status"
],
"$map": {
"r": "renter",
"rent": "renter",
"o": "owner",
"homeowner": "owner",
"own": "owner"
}
}
Example - distinct attribute values with similar codes
This update allows us to map the output of the individual attributes that are passed into $firstOf. The example below shows that for the same code, C006 from Quantarium and Attom, different values could be mapped.
{
"providers": {
"attomdata_attom_id": {
"version": "$latest"
},
"hosted_attom_residential_tax_assessor": {
"version": "$latest",
"inputs": {
"attom_id": "attomdata_attom_id.attom_id"
},
"when": {
"attomdata_attom_id": "$isMatch"
}
},
"hosted_quantarium_open_lien": {
"version": "$latest"
}
},
"inputs": {
"street": "221 Clinton Ave",
"city": "brooklyn",
"state": "ny",
"post_code": "11205",
"country": "us"
},
"refine": {
"route_demo": {
"$firstOf": [
{
"$at": "hosted_attom_residential_tax_assessor.results[0].contact_owner_mail_address_crrt",
"$map": {
"C006": "City"
}
},
{
"$at": "hosted_quantarium_open_lien.results[0].pa_carrier_route",
"$map": {
"C006": "County"
}
}
],
"$map": {
"C001": "Street",
"C002": "Road"
}
}
}
}
Arguments
-
A mapping from upstream → custom attributes.
Returns
-
The custom attribute, or the upstream attribute if no argument matched.
Notes
-
$map needs to be called together with another function. If you only want to change a single attribute, you can use the $at function.
Returns the attribute at the given path.
Arguments
-
An attribute path in full, i.e. connector + attribute
Returns
-
The attribute
-
This function is particularly useful used together with the $map function
-
Example - With $map
This example uses a connector which has a construction type attributes that has many possible values. These values and their descriptions are within the Data Dictionary. Instead of frequently referring back to the Data Dictionary, we can use $at with $map for easier access to all or a subset of this enumeration.
"""
Let's use the construction attribute from hosted_attom_commercial_tax_assessor
and identify immediately construction types of particular importance that are
returned in any of the connector's matched results.
"""
{
...,
"refine": {
"construction_types": {
"$at": "hosted_attom_commercial_tax_assessor.results[*].construction",
"$map": {
"0": "Unknown construction type",
"1": "Type isn't specified",
"10": "Wooden construction",
"31": "Brick construction"
}
}
}
}
"""
The refined output informs you immediately of what construction types
were returned at the attribute, for all matched results.
"""
{
"refine": {
"construction_types": "Wooden construction"
}
}
$extract
Extracts attributes from a list, while allowing the user to rename them and optionally apply a function to them
Arguments
-
Required
-
list_path: A path to a list
-
May not contain wildcards
-
-
select: A JSON object (map, dictionary, ...) containing either
-
The $at function, which takes an attribute path
-
Nested attributes may be retrieved with dot notation, i.e. address.post_code
-
-
The $const function, which takes a scalar JSON node (string, number, boolean, or null)
-
This will be applied to each list item
-
-
-
-
Optional
-
drop_nulls: A Boolean indicating that keys whose values are null (empty) be removed from the dictionary
-
flatten: A key in the dictionary to be selected and returned within a list, without wrapping as a dictionary
-
Returns
-
The custom schema
Example - Using only $at
"""
Assuming the following connector output:
"""
{
"companies_house_company_search": {
"data": {
"items": [
{ "company_status": "dissolved", "company_type": "private-unlimited-nsc" },
{ "company_status": "closed", "company_type": "oversea-company" }
]
}
}
}
"""
The $extract function is called like so, in refine:
"""
{
...,
"refine": {
"new_schema": {
"$extract": {
"list_path": "companies_house_company_search.items",
"select": {
"status": { "$at": "company_status" },
"type": { "$at": "company_type" }
}
}
}
}
}
"""
To yield the below final result:
"""
{
"refine": {
"new_schema": [
{ "status": "dissolved", "type": "private-unlimited-nsc" },
{ "status": "closed", "type": "oversea-company" }
]
}
}
Example - Using$at with $const
"""
Assuming a similar connector output as in the above example,
we could call $extract in the refine section, like this:
"""
{
...,
"refine": {
"new_schema": {
"$extract": {
"list_path": "companies_house_company_search.items",
"select": {
"new_attribute_name": { "$const": "Companies House - Demyst" },
"number": { "$at": "company_number" },
"status": { "$at": "company_status" }
}
}
}
}
}
"""
The result will look as follows:
"""
{
"refine": {
"new_schema": [
{ "new_attribute_name": "Companies House - Demyst",
"number": "123456789",
"status": "dissolved"
},
{ "new_attribute_name": "Companies House - Demyst",
"number": "987654321",
"status": "closed"
}
]
}
}
Example - Using$at with flatten
"""
Still assuming a similar connector output as in the above example,
we could call $extract in the refine section, like this:
"""
{
...,
"refine": {
"my_b2b_schema": {
"$extract": {
"list_path": "companies_house_company_search.items",
"select": { "send_business_mail_to": { "$at": "address_snippet" } },
"flatten": "send_business_mail_to" },
"my_kyb_schema": {
"$extract": {
"list_path": "companies_house_company_search.items",
"select": {
"new_attribute_name": { "$at": "title" },
"number": { "$at": "company_number" },
"status": { "$at": "company_status" }
}
}
}
}
}
"""
The response would then look something like the following!
"""
{
"refine": {
"my_b2b_schema": [
"123 First Line Address Rd, New Yorkie, NY, USA",
"ABC 2nd Street #ZYX, Big State, TX, USA"
],
"my_kyb_schema": [
{ "new_attribute_name": "DEMYST LIMITED",
"number": "123456789",
"status": "active"
},
{ "new_attribute_name": "DEMYST CONSULTING LIMITED",
"number": "987654321",
"status": "active"
}
]
}
}
$dictionary
$dictionary introduces a new level of nesting inside of $refine, which we currently can’t support natively without using $dictionary due to syntactic ambiguity
Arguments
-
Supported right-hand side arguments are the Top level functions
-
After the first call of $dictionary, make sure to use the keyword dictionary (without $) after the colon, for type casting
Returns
-
A new dictionary
Example
"""
Let's keep with the companies_house_company_search output assumed
in previous examples. We can really push the limits of how we want
'refine' to behave.
"""
{
...,
"refine": {
"my_special_dict:dictionary": {
"$dictionary": {
"company_likely_name": {
"$dictionary": {
"take_first_title": { "$at": "companies_house_company_search.items[0].title" }
}
},
"my_sub_dict": {
"$dictionary": {
"my_sub_2_dict": {
"$dictionary": {
"my_sub_3_dict": {
"$extract": {
"list_path": "companies_house_company_search.items",
"select": { "number": { "$at": "company_number" } },
"flatten": "number"
}
}
}
}
}
}
}
}
}
}
"""
The above logic yields an output like so:
"""
{
"refine": {
"my_special_dict": {
"company_likely_name": {
"take_first_title": "DEMYST LIMITED"
},
"my_sub_dict": {
"my_sub_2_dict": {
"my_sub_3_dict": [
"0123456789",
"9876543210",
"0192837465"
]
}
}
}
}
}
Error Simulations: simulation
The simulation option allows to simulate an error for the provider in the request
Syntax
-
A JSON object from a connector to an error type
Example
{ "acxiom_place": "upstream_structure"}
Returns
type and message for the error raised
Example
"simulation": {
"acxiom_place": "upstream_structure",
"hazardhub_risks_and_enhanced_property2": "upstream_http"
}
Per Provider
For each provider in the request, you can simulate an error without having to add `simulate` option separately. The error in the response will remain the same
Example
"providers": {
"equifax_austin_tetra_discovery": {
"error_simulation": "upstream_structure",
"version": "$latest"
}
}
Syntax
-
A JSON object from error_simulation to an error type within the provider object
Example
{ "error_simulation": "upstream_structure"}Returns
type and message for the error raised
Errors Library
upstream_structureXML/JSON parsing failure, unexpected elements in upstream response or unexpected upstream structure
"error": {
"type": "unexpected_upstream_structure",
"message": "We've encountered an issue while communicating with an upstream connector."
}
upstream_http
Server error from provider, example - HTTP status code 500 unexpected
"error": {
"type": "unexpected_upstream_http_status",
"message": "HTTP status code 500 with meaning 'Internal Server Error. A generic error message, given when an unexpected condition was encountered and no more specific message is suitable.' unexpected."
}
insufficient_credentials
The data provider(s) you tried to reach did not receive sufficient valid credentials to execute.
"error": {
"type": "insufficient_credentials",
"message": "Missing credentials to run this provider."
}
invalid_credentials
The credentials either provided by the client or by Demyst to access to a particular data provider are incorrect or out of date.
"error": {
"type": "invalid_credentials",
"message": "Request was not authorized, the credentials you provided for this data source are likely invalid."
}
The credentials either provided by the client or by Demyst to access to a particular data provider are incorrect or out of date.
"error": {
"type": "upstream_service_unavailable",
"message": "The data source is experiencing issues"
}
rate_limit_exceeded
Providers having monthly / concurrency rate limit, if any, exceeded
"error": {
"type": "rate_limit_exceeded",
"message": "Rate limit exceeded"
}
sample_data_not_implemented
Provider does not have SampleData and errors out using test key/sample mode
"error": {
"type": "sample_data_not_implemented",
"message": "Provider experian_business_judgments does not have SampleData"
}
unexpected_provider_error
The data provider you are attempting to access threw an unexpected error.
"error": {
"type": "unexpected_error",
"message": "An unexpected issue occurred"
}
timeout
Timeout exceeded. Default timeout is 100 seconds, can be extended
"error": {
"type": "timeout",
"message": "Timeout of 1000 ns exceeded"
}
Debugging Config API Coded Errors
waterfall_dependency_errorUser has specified a provider as an input / when condition, but this provider is not part of the providers in the request / saved data API
{
"transaction_id": "bebbe9b8-7a40-4e1e-8cb0-d3c8432617d4",
"error": {
"type": "waterfall_dependency_error",
"message": "Provider experian_business_facts referenced a non-existing provider hosted_experian_cpdb"
}
}
refine_error
Happens before the providers are executed if a function doesn't exist, or wrong parameters were passed
{
"transaction_id": "2a3ca703-cdf3-4f29-a4d4-57b24003a2e6",
"error": {
"type": "refine_error",
"message": "Unknown function 'atis' on Transaction. Did you mean $at?"
}
}
decoding_error
An incorrect syntax, an unwanted character within the API call that won't let the request be evaluated until corrected will be propagated as a decoding_error
{
"transaction_id": "1c6b6350-4550-4cd1-a7cd-29055e2a791c",
"error": {
"type": "decoding_error",
"message": "expected \" got '}\n ...' (line 6, column 9)"
}
}
{
"transaction_id": "a9e2f44b-509c-4fc9-aa9e-01bb439c8f96",
"error": {
"type": "decoding_error",
"message": "Not a string value: DownField(providers)"
}
}
{
"transaction_id": "ede8349c-d760-455a-9fdd-7997aef0041e",
"error": {
"type": "decoding_error",
"message": ".providers: While parsing connector-attribute-path 'true': connector-attribute path requires a '.'"
}
}
dependency_error
This error occurs when a provider is set to receive inputs from the output of another provider, but that provider was Skipped.
{
"transaction_id": "6c48013f-bdee-4ef0-a2e7-5b2e531df15e",
"output": {
"PROVIDER_NAME": {
"error": {
"type": "dependency_error",
"message": "Connector 'PROVIDER_NAME' had run state: 'Skipped'"
}
}
}
}