Sample Request and Response

Demyst API configurations are saved within the platform.

A user that creates a data API from a recipe or a data product gains access to the single lookup interface, the configuration edit interface, metrics and tracking screens, and programmatic access to the Data API.

An API request to a channel returns the same response that appears in the single lookup results view on the platform.

This part of the documentation shows how to create API requests and how to make sense of the output.

Request Details

URL and Request Type

The Demyst API is accessed at https://gw.CC.demystdata.com/v2/channel, with CC replaced by the region in which your execution servers are running (us, au, eu, sg). Requests to the API should be sent as HTTPS POST requests.

Authentication

Requests require an API Key for authentication. View more details on API Keys here.

Channel ID

The “id” is the Channel ID number from your saved data API on the Demyst platform. This is shown at Data APIs > Your Data API > Settings. The Channel ID sends inputs to that specific configuration so that the correct data is appended.

Inputs

The inputs use the same information that would be used on the single lookup page. Use the exact same keys that appear on that screen (in the example above, “country”, “state”, “city”, “street”, “post_code”, “first_name”, “last_name”).

Connection

All requests to Demyst are made over secure TLS 1.2 channels using HTTPS. The host you communicate with depends on the region that you are working with.

Sample Request

POST

https://gw.us.demystdata.com/v2/channel

arrows curl
                
                curl -X POST \
                  https://gw.us.demystdata.com/v2/channel \
                  -H 'Content-Type: application/json' \
                  -d '{
                    "api_key": "xxxxxxxxxxxxxxxxxxxxxx",
                    "id": 1020,
                    "inputs": {
                      "country": "US", 
                      "state": "NY",
                      "city": "New York",
                      "street": "100 Main Street", 
                      "post_code": "10001", 
                      "first_name": "JOHN", 
                      "last_name": "DOE"
                    }
                  }'
                
                

Response Details

Transaction ID

The transaction_id is a unique value that references the API transaction. This reference ID can be used to communicate with Demyst about the transaction without sharing any sensitive data.

Refine

The “refine” section of the response displays the attributes that are defined in the refine section of the saved configuration. These attributes use the raw response data from providers to derive new attributes, or simply rename and highlight them.

The full response from the providers can include hundreds of attributes and complex structuring. The “refine” section gives users a way to customize a Data API by sectioning off the important, cleaned data. For more details on how to define refine attributes, see our Data APIs page.

Channel ID

The full response from the providers can include hundreds of attributes and complex structuring. The “refine” section gives users a way to customize a Data API by sectioning off the important, cleaned data. For more details on how to define refine attributes, see our title--small page.

Refine Errors

Because the refine configuration is freely written by users, errors will occur — especially in the development phase. The “refine_errors” section provides feedback on specific attributes when an error occurs.

Output

The “output” is a container for subsections from each provider that was run. In the example above, only “hazardhub_risks_and_enhanced_property2” was run.

Data

The “data” section is Demyst’s parsing of the raw response from the data provider. The example only provides three data points, but data providers can return hundreds. Demyst preserves as much of the original response’s structure as possible. In this case, the “enhanced_property” and “assessment” sections are included in Demyst’s presentation of the data.

Flattened Data (optional)

This section takes everything in “data” and presents it as a single level of key-value pairs. It allows the data to be entered as a single row in a table, showing which subsection contains the data point by looking for the key name. This section is not included in responses by default. To receive this section, the “config” section of the configuration must include the setting: “return_flattened_data”:true . See the config page for more details.

Raw Data (optional)

The “raw_data” section returns the data exactly as Demyst receives it from the data provider, in the spirit of full transparency. This section is not included in responses by default. To receive this section, the “config” section of the configuration must include the setting: “return_raw_data”:true . See the config page for more details.

Cached

This shows whether the response was fetched live or fetched from the cache. A response will only be fetched from the cache if the request is made in cache mode and an identical request has been made recently. See more about cache mode and Demyst’s caching in the mode section.

Response Time

This value is the time Demyst needed to fetch the data from the provider. It is collected and displayed in a more usable way on the Data API’s dashboard, but it can also be collected and analyzed through this field in the API response.

Version

This value is a simple reference to the version of the data product used in the request.

Data providers occasionally add or remove data from their responses, or they allow new inputs to be queried. Whenever Demyst updates its connection with a data provider, a new version is created.

By default, Demyst uses the latest implementation and version of all data products. However, a static version is often configured for production purposes, which means that unexpected schema changes can occur.

Pipes

This is a legacy section. In most cases, it can safely be ignored.

Sample Response

GET

https://gw.us.demystdata.com/v2/channel

arrows json
                  
                  {
                    "transaction_id" : "291ada41-d4e4-42c2-8696-25d9c8041b9e",
                    "refine" : {
                      "approve_applicant":true,
                      "applicant_segment":"B",
                    },
                    "refine_errors" : {
                      
                    },
                    "output" : {
                      "hazardhub_risks_and_enhanced_property2" : {
                        "data" : {
                          "enhanced_property": {
                            "assessment": {
                              "assessed_improvement_value": 206160,
                              "assessed_land_value": 39360,
                              "assessment_year": "2021"
                            }
                          }
                        },
                        "cached" : true,
                        "response_time" : 363,
                        "raw_data": {
                          "body": "{\"enhanced_property\":{\"assessment\":{\"Assessed_Improvement_Value\":206160,\"Assessed_Land_Value\":39360,\"Assessment_Year\":\"2021\"}}}"
                          "mime_type": "application/json; charset=utf-8"
                        },
                        "flattened_data" : {
                          "enhanced_property.assessment.assessed_improvement_value": 206160,
                          "enhanced_property.assessment.assessed_land_value": 39360,
                          "enhanced_property.assessment.assessment_year": "2021"
                        },
                        "version" : 8
                      },
                    },
                    "pipes" : {
                      
                    }
                  }
                  
              

Demyst Types

Demyst validates all received inputs before sending to provider.


Required formats are as follows:

Data Type Description Example
blob Base64-encoded binary data RGVteXN0
business_name The name of a company Demyst Data Ltd.
city The name of a city New York City
country Must be a 2 or 3 character iso code https://en.wikipedia.org/wiki/ISO_3166-1_alpha-3 or https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 US, AU, SG
domain An internet domain name demyst.com
email_address An email address support@demyst.com
first_name A first name John
full_name A full name John Doe
gender A gender or abbreviation m, male, f, female
ip4 IP address (version 4) 192.168.0.1
last_name A last name Smith
latitude Number between -90.090.0 40.7
longitude Number between -180.0180.0 -73.9
marital_status A marital status or abbreviation m, married, s, single, ...
middle_name A middle name Rupert
number A number. Supports integral and decimal numbers of arbitrary size and precision 42
percentage A number between 0.0 and 100.0 99%, 99
phone Country dependent, for US must be 10 digits without leading one or 11 digits with, area code must be valid 917-475-1881
post_code If US 5 or 9 digit postcode, dash or no dash separating. other countries need be non empty 10001
sic_code A Standard Industrial Classification code. 4 digit character string 2024
state If US it must be a valid 2 character state code or state name. Empty otherwise NY, New York
street Non-empty. A street name 100 Main St
string A character string foo
url A Uniform Resource Locator. Starts with http: or https: https://www.demyst.com
us_ein An Employer Identification Number. Dashes and spaces stripped from input by us, must be 9 numeric character string 12-3456789
us_ssn A Social Security Number. Dashes and spaces stripped from input by us, must be 9 numeric character string 078-05-1120
us_ssn4 The last four digits of a Social Security Number 1120
year_month A particular month of a year. In format yyyy-MM 2019-01
year A year 2019

Authorization — Using API keys and generating tokens

Authentication

The Demyst API supports three secure, industry-standard methods of authorization so that organizations can choose the best fit for their preferred information security policies.

API Keys

Users can generate, control, and destroy REST API keys through the Demyst platform to conform with their organization's rotation policies. Demyst will not store API keys in plain text, which means that keys must be saved when they are generated.

Sample cURL request with token generation:

POST

https://gw.us.demystdata.com/v2/channel

arrows curl
                
                    curl -X POST \
                    https://gw.us.demystdata.com/v2/channel \
                    -H 'Content-Type: application/json' \
                    -d '{
                      "api_key": "xxxxxxxxxxxxxxxxxxxxxx",
                      "id": 1020,
                      "inputs": {
                      }
                    }'
                
              

JWT

JSON Web Tokens can be used in compliance with the industry-standard RFC 7519 method.

Tokens can be generated with the email address (username) and password of an account that has provisioned access to the Demyst platform.

Passwords expire after 90 days, and tokens are valid for 24 hours.

Sample cURL request with token generation:

POST

https://console.demystdata.com/jwt/create

arrows curl
              
                  curl -X POST \
                    https://console.demystdata.com/jwt/create \
                    -H 'Content-Type: application/json' \
                    -d '{
                      "email_address": "EMAIL",
                      "password": "PASS"
                  }'
                  
                  #Use the response token from above in below
                  curl -X POST \
                    https://gw.us.demystdata.com/v2/channel \
                    -H 'Content-Type: application/json' \
                    -d '{
                      "api_key": "JWT_FROM_ABOVE",
                      "id": 1020,
                      "inputs": {
                      }
                  }'
              
              

OAuth 2.0

Demyst supports resource owner credentials OAuth 2.0 grant (RFC 6749, §4.3) and refresh token OAuth 2.0 grant (RFC 6749, §1.5, §5).

Demyst will issue an access token with a TTL that is suitable for the client's needs along with a refresh token that can be used to retrieve another access token.

Sample cURL request with token generation:

POST

https://gw.us.demystdata.com/oauth/token

arrows curl
              
                  curl -X POST \
                    https://gw.us.demystdata.com/oauth/token \
                    -d 'grant_type=password&client_id=DEMYST_GENERATED_ID&client_secret=DEMYST_GENERATED_SECRET&username=CLIENT_PROVIDED_USER&password=CLIENT_PROVIDED_PASS'
                  
                  #Sample Response
                  {
                    "token_type": "Bearer",
                    "access_token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
                    "expires_at": "2020-11-17T15:29:38.721873Z",
                    "refresh_token": "xxxxxxxx"
                  }
                  
                  #Generate access_token using refresh token
                  curl -X POST \
                    https://gw.us.demystdata.com/oauth/token \
                    -d 'grant_type=refresh_token&client_id=DEMYST_GENERATED_ID&client_secret=DEMYST_GENERATED_SECRET&refresh_token=REFRESH_TOKEN_FROM_ABOVE'
                    
                    #Use access_token to get data
                    curl -X POST \
                    https://gw.us.demystdata.com/v2/channel \
                      -H 'Content-Type: application/json' \
                      -d '{
                      "api_key": "ACCESS_TOKEN_FROM_ABOVE",
                      "id": 1020,
                      "inputs": {
                      }
                  }'
                
              

The token can be used in a subsequent call:

POST

https://gw.us.demystdata.com/v2/channel

arrows curl
                
                    curl -X POST \
                      https://gw.us.demystdata.com/v2/channel \
                      -H 'Content-Type: application/json' \
                      -d '{
                        "api_key": "eyJhbGciOiJSUzI1NiJ9.eyJ1c2VyX25hbWUiOiJIYXJzaGl0IFNpbmdoIiwidXNlcl9pZCI6ODA2LCJlbWFpbF9hZGRyZXNzIjoiaHNpbmdoQGRlbXlzdGRhdGEuY29tIiwib3JnYW5pemF0aW9uX2lkIjoyLCJkYXRhIjoidGVzdCIsImlhdCI6MTU5ODYyNDgyMCwiZXhwIjoxNTk4NjY4MDIwfQ.qVXr0g5cIVUKVEuhQqXG6gx7615kk0OVZOaX7Jj1Wn1pG_IMdjcysVoO2V_Ossnn-wIVuke36KiciDR1Dp6Be6OfegSFOQCCSAaAZNp9PIbmgcHFTv3p4ONFXj07YKRdRxOm5SrUtOesOxgv7vmPrWf8Kz3yaujeoV8AdBOJjA8TfeGToo8FXpYKcuP1Vs0O63nSPr4p4eDXs0k_kT2AMmxYVGnyVG_XbDQKV5mrdEw5lS7hlr_huIwRyp2JwlgnCWBd4jz2xuroc5FeBPQUtx_5GaI4abA27ITy10RGCHJmFR-dGuIyec6sB0fAxJPzzR4X8Ck4qoIILothgH8Ndw",
                        "id": 2,            
                        "inputs": {
                            "business_name": "Demystdata",
                            "street": "28 W 25th St",
                            "city": "New York",
                            "state": "NY",
                            "post_code": "10010",
                            "country": "US"
                        }
                    }'
                
              

Note that test keys should only be used with Sample mode. Production keys should be used for every other mode.

Config Section

The “config” section — distinct within an overall API configuration — can be included at the end of a full API configuration or in the API request itself. It should be included in a saved API configuration.

The following example is a full configuration of Demyst’s Business Firmographics Recipe.

The “config” section at the bottom dictates specific API behaviors.

Mode

Adding "config": { "mode" : MODE } to the JSON request payload can run a transaction in an alternate execution mode.

  • sample By default, this mode runs each provider using Demyst’s internally saved static sample responses. This is the only allowed mode for a test API key; it does not need to be explicitly requested.

  • cache this mode checks Demyst’s provider cache for a response before making a call to a data provider. If a user has called a provider in cache mode with identical inputs at some point in the previous 7 days, the response will be in the cache. Responses are added to the cache when a provider is called in cache mode and the response is not already in the cache.

  • cache_read_only This mode searches Demyst’s provider cache for a response. It does not attempt a live call to any provider, and it returns an error if the response is not already present in the cache.

  • live This is the default mode for production keys, and it does not need to be explicitly requested. Live mode always calls new data from data sources.

Raw Data

The return_raw_data flag creates a new subsection under the response from each data provider. A true setting will return the raw data directly from the data provider. See more details on that section under the API Response description.

Flattened Data

The return_flattened_data flag creates a new subsection under the response from each data provider. A true setting will return the data parsed by Demyst and flattened down to a single level of key-value pairs. See more details on that section under the API Response description.

Fixed List Size

Data providers often return large lists of data, and the size of these lists can be arbitrary. By limiting the size of the lists, and therefore the overall number of attributes, responses can be simplified or made easier to use and more predictable for downstream processes.

The fixed_list_size information in the example above limits any list size to 3 — any element after that will be cut out of the raw data and parsed data responses. The default number is 10.

              
                {
                  "providers": {
                    "everstring_company_enrichment": {
                      "version": "$latest"
                    },
                    "hosted_experian_cpdb": {
                      "version": "$latest"
                    },
                    "hosted_equifax_mds": {
                      "version": "$latest"
                    },
                    "hosted_infogroup_business_places": {
                      "version": "$latest"
                    },
                    "hosted_infogroup_business_places": {
                      "version": "$latest"
                    }
                  },
                  "refine": {
                    "sic_code": {
                      "$firstOf": [
                        "hosted_infogroup_business_places.results[0].primary_sic_code_id",
                        "hosted_equifax_mds.results[0].efx_primsic",
                        "hosted_experian_cpdb.results[0].prim_sic_code",
                        "everstring_company_enrichment.data[0].es_sic_4"
                      ]
                    },
                    "naics_code": {
                      "$firstOf": [
                        "hosted_infogroup_business_places.results[0].primary_naics_code_id",
                        "hosted_equifax_mds.results[0].efx_primnaicscode",
                        "hosted_experian_cpdb.results[0].prim_naics_code",
                        "everstring_company_enrichment.data[0].es_naics_6"
                      ]
                    },
                    "year_business_start": {
                      "$firstOf": [
                        "hosted_infogroup_business_places.results[0].estimated_opened_for_business_lower",
                        "hosted_equifax_mds.results[0].efx_yrest",
                        "hosted_experian_cpdb.results[0].year_business_started",
                        "everstring_company_enrichment.data[0].es_year_started"
                      ]
                    },
                    "sales": {
                      "$firstOf": [
                        "hosted_infogroup_business_places.results[0].location_sales_volume",
                        "hosted_equifax_mds.results[0].efx_corpamount",
                        "hosted_experian_cpdb.results[0].est_annual_sales_amt",
                        "everstring_company_enrichment.data[0].es_revenue"
                      ]
                    },
                    "no_of_employees": {
                      "$firstOf": [
                        "hosted_infogroup_business_places.results[0].estimated_location_employee_count",
                        "hosted_equifax_mds.results[0].efx_locempcnt",
                        "hosted_experian_cpdb.results[0].est_num_of_employees",
                        "everstring_company_enrichment.data[0].es_employee"
                      ]
                    },
                    "phone": {
                      "$firstOf": [
                        "hosted_infogroup_business_places.results[0].phone",
                        "hosted_equifax_mds.results[0].efx_phone",
                        "hosted_experian_cpdb.results[0].phone",
                        "everstring_company_enrichment.data[0].es_company_phone"
                      ]
                    },
                    "website": {
                      "$firstOf": [
                        "hosted_infogroup_business_places.results[0].website",
                        "hosted_equifax_mds.results[0].efx_web",
                        "hosted_experian_cpdb.results[0].url",
                        "everstring_company_enrichment.data[0].es_primary_website"
                      ]
                    }
                  },
                  "config": {
                    "mode": "cache",
                    "return_raw_data":true
                    "return_flattened_data":true,
                    "fixed_list_size":3
                  }
                }
              
            

Errors

error_type current_message error_category description http_status
blackbelly_communications_link_failure internal_error An internal error has occurred. The Demyst team should be contacted so that it can be resolved. 0
blackbelly_query_timeout timeout The data query took too long to process, for one of several possible reasons. 0
blackbelly_table_does_not_exist internal_error An internal error has occurred. The Demyst team should be contacted so that it can be resolved. 0
channel_access_denied permission_error The Demyst team should be contacted to ensure that the user can access the channel used by the request. 0
channel_not_found Channel with id 1103 not found channel_not_found The user should verify that they provided the correct channel for their request. 0
data_function_error internal_error An internal error has occurred. The Demyst team should be contacted so that it can be resolved. 0
data_function_misses_result_key internal_error An internal error has occurred. The Demyst team should be contacted to fix a client-specific customization. 200
data_function_response_error Unexpected response received from DataFunction:
data function did not return a result key, in pipe
with id: Some(691)
internal_error No results were returned from a custom data function. Demyst support should be contacted. 200
decoding_error upstream_structure_error The data provider returned an unexpected response. Demyst support should be contacted. 422
insufficient_api_key_permissions The given apiKey is for test use only. It is only
authorized for use with Sample mode
permission_error The API key provided by the user can only be used for test purposes in sample mode. 400
insufficient_channel_permissions Organization with id 2 does not have permission
to run Channel with id 1052
permission_error The Demyst team should be contacted to ensure that the user can access the channel used by the request. 403
insufficient_credentials insufficient_credentials The Demyst team should be contacted to fix an access issue with a specific data provider. 200
insufficient_input Did not have sufficient valid input to run.
Given valid inputs:
InputField(email_address,Some(EmailAddress),
\"ladylockett86@icloud.com\"),
InputField(state,Some(State),\"QLD\"),
InputField(country,Some(Country),\"AU\"), InputField(post_code,Some(PostCode),\"4159\"), InputField(last_name,Some(LastName),\"Schiffke\"),
InputField(ip6,Some(Ip6),
\"2001:8003:e122:4800:edac:8c6:c71e:bbd4\"), InputField(street,Some(Street),\"4 Clive Rd\"), InputField(phone,Some(Phone),\"+61467558685\"), InputField(first_name,Some(FirstName),\"Leah\"), InputField(city,Some(City),\"Birkdale\")"
insufficient_input The attempted configuration requires specific inputs to execute successfully. Verify that the correct inputs that were provided to the connector. 200
invalid_api_key apiKey 236dfh453756... is invalid permission_error The user’s API key is invalid, inactive, or missing a character. 400
invalid_credentials Request was not authorized, the credentials you
provided for this data source are likely invalid.
invalid_credentials The credentials — provided by either the client or by Demyst — for accessing a specific data provider are incorrect or out of date. 200
ip_whitelist_error "Request came from 116.14.20.220, which
is not on the ip whitelist for this organization.
permission_error The request came from an IP address that is not associated with the user’s organization. The Demyst team should be contacted to update the IP whitelist. 0
logical_input_error logical_input_error The attempted query requires specific inputs that were not in the correct format. 200
missing_provider_grants permission_error The user requested information from a data provider that requires special access privileges. Contact the Demyst team to discuss accessing this provider. 0
network_error upstream_error A network error has occurred. The Demyst team should be contacted to resolve it. 200
no_api_key_given permission_error The user did not provide the credentials that are necessary to access a specific data provider. 0
provider_version_not_found provider_version_not_found The query attempted to access a data provider that was not recognized. The information in the original query should be checked for errors. 400
rate_limit_exceeded rate_limit_exceeded A data provider has been contacted too frequently in too short a period of time; their rate limit has been exceeded. 200
sample_data_not_implemented sample_data_not_implemented The API attempted to request sample data from a data provider that is not available. Contact Demyst support to discuss implementing sample data for this data provider. 200
timeout Timeout of 5 s exceeded. Cause: -" timeout The Demyst team will need to check the response to determine why the default timeout period has been exceeded. 200
uncaught_defect internal_error An unexpected error has occurred. The Demyst team should be contacted so that it can be resolved. 200
unexpected_batch_record_error internal_error An unexpected error has occurred. The Demyst team should be contacted so that it can be resolved. 200
unexpected_blackbelly_sql_error internal_error An unexpected error has occurred. The Demyst team should be contacted so that it can be resolved. 200
unexpected_data_function_error Data Function Exception: An error occurred
and the request cannot be processed.
(Service: AWSLambda; Status Code: 500;
Error Code: ServiceException Data Function Exception:
Unable to execute HTTP request:
Timeout waiting for connection from pool,
in pipe with id: Some(601)
internal_error The Demyst team will need to resolve the issue by identifying why the response from the provider has resulted in an error. 0
unexpected_provider_error internal_error The Demyst team will need to resolve the issue by identifying why the response from the provider has resulted in an error. 200
unexpected_upstream_http_status upstream_error The Demyst team will need to resolve the issue by identifying why the response from the provider has resulted in an error. 200
unexpected_upstream_structure upstream_structure_error The Demyst team will need to resolve the issue by identifying why the response from the provider has resulted in an XML/JSON parsing failure. 200
upstream_application_error Unknown Error or... Unknown city:
PEACHTREE CORNERS, GA
upstream_application_error Either the requested data provider is down, or it returned an unexpected error. (Similar to upstream_service_unavailable) 200
upstream_service_unavailable HTTP status code 503 unexpected
|| HTTP status code 500 unexpected
upstream_error Either the requested data provider is down, or it returned an unexpected error. (Similar to upstream_application_error) 200