Demyst Catalog - RESTful API

Overview

This page details the RESTful API for pulling catalog details that can aid with catalog integration with a third-party tool (such as ckan, data.world) or any other bespoke destination.

API Calls for Catalog

- GET JWT TOKEN

To get the JWT token required for calls to our API, the following cURL request can be used (or imported to an API platform like Postman as a raw test or can be put into the terminal of your computer). Substitute the dummy variables [EMAIL] and [PASSWORD] with your email address and password credentials for accessing our API.

    
    curl --location --request POST 'https://console.demystdata.com/jwt/create' \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "email_address": "[EMAIL]",
      "password": "[PASSWORD]"
    }'
    
  

The response of this request will replace the variable [BEARER] in the calls below, copy and paste it exactly as it is output.

- List of Connectors and Basic Metadata

The endpoint to access our entire list of available providers/APIs/connectors provides general metadata rather than the heavier payload of a call to a specific provider. This provides a queryable overview of the list of products we offer.

This process is two-part: the cURL request for accessing the total number of providers (an ever-changing and growing value), and the cURL request to use that total number of providers for an unpaginated view of the list of products.
Part1:

    
    curl --location --request GET 'https://console.demystdata.com/catalog/providers' \
    --header 'Authorization: Bearer [BEARER]' \
    --header 'Accept: application/json' \
    --header 'Content-Type: application/json'
    
    
    

The total number of accessible providers is the value total_entries in the “pagination” dictionary at the top of the response. Use this value to replace the variable [TOTAL_ENTRIES] in Part 2.
Part2:

    
      curl --location --request GET 'https://console.demystdata.com/catalog/providers?per_page=[TOTAL_ENTRIES]' \
      --header 'Authorization: Bearer [BEARER]' \
      --header 'Accept: application/json' \
      --header 'Content-Type: application/json'
    
    

From this list, we can use the id or name of a specific product to make the below calls.

- List Input and Output fields/Schema of a single Connector

When you have identified the id value for the product of which you’d like more than basic metadata (or specifically, you need the input and output schema), use the below request and replace [ID].

    
    curl --location --request GET 'https://console.demystdata.com/catalog/providers/[ID]' \
    --header 'Authorization: Bearer [BEARER]' \
    --header 'Accept: application/json' \
    --header 'Content-Type: application/json'
    
    
    

For example let’s use the provider of the name hosted_equifax_ixi_ability_to_pay which can be viewed on our platform here: Demyst: The Data Deployment Platform. From the aforementioned cURL request of the entire list of providers with basic metadata, we see that this product has identifier id: 1490. Substituted into the directly-above request, this will become:

    
    curl --location --request GET 'https://console.demystdata.com/catalog/providers/1490' \
    --header 'Authorization: Bearer [BEARER]' \
    --header 'Accept: application/json' \
    --header 'Content-Type: application/json'
    
    

The entire sample output will be like below. The schema can be found at the following locations:

  • Input Schema Location →

    .latest_version.schema[0].input

  • Output Schema Location → .latest_version.schema[0].output

If you are more specifically interested in required or optional input attributes, these can be found at:

  • Required Inputs → .latest_version.schema[0].input.required
  • Optional Inputs → .latest_version.schema[0].input.optional

Entire sample output:

    		
      {
        "adx_flag":false,
        "alias":"IXI - Ability to Pay Index",
        "data_update_frequency":null,
        "description":"Ability to Pay (ATP) Index™ provides insight into a household’s estimated financial position by ranking households by their likely economic capacity. It is delivered as a household-level score of 1-1000 which ranks households based on their financial ability to pay financial obligations or incur new ones.  ATP Index incorporates estimated total income, discretionary spending, and aggregated credit to provide a more complete measure of capacity than using only one of these measures alone. It is intended for use in non-FCRA marketing applications only.",
        "diligence_tag":"Conditionally Approved",
        "dppa":false,
        "fcra":false,
        "featured":false,
        "glba":false,
        "id":1490,
        "logo":"https://console.demystdata.com/assets/equifax-e721c85699facdb021ce0491fa03126e97ae914c18f38a6f90c8c47ae86809ec.svg",
        "name":"hosted_equifax_ixi_ability_to_pay",
        "notes":"Internal Diligence Notes - Data may not be used a factor in establishing an individual’s eligibility for credit, insurance, employment, or any other permissible purpose for which a consumer report may be used under the FCRA or in any way for the purpose of taking “adverse action” against a consumer, as defined in the Equal Credit Opportunity Act (ECOA) and Regulation B (vendor)",
        "product_type":"Full Release",
        "text_conditionally_approved":"Conditions: For use in non-FCRA B2C marketing uses cases.",
        "credit_price":"7.0",
        "error_rate":1.0,
        "hit_rate":0.0,
        "data_source":{
          "name":"Equifax",
          "website":"https://www.equifax.com"
        },
        "data_category":{
          "name":"People"
        },
        "data_regions":[
          {
            "name":"United States"
          }
        ],
        "tags":[
          {
            "name":"Consumer behavioral",
            "description":"Consumer behavioral data (i.e. related to activities, interests & intent)",
            "category":"SubEntity(Data-Category)"
          },
          {
            "name":"Consumer transactional and financial",
            "description":"Consumer historical transactional & financial data",
            "category":"SubEntity(Data-Category)"
          },
          {
            "name":"Application Pre-fill",
            "description":"In order to assess the risk associated with a particular business, individual or property, an insurer must collect information about the insured. Whilst a short list of factors is easily to collect from a customer, it is typically not enough to be able to assess the risk associated to them and so an underwriting process often has a long list of items that must be known about an entity seeking insurance. Without having external data in order to make this assessment, the customer is forced to provide a large amount of information upfront which undermines their experience, causing drop-offs in applications and is also prone to errors and omissions. By having access to external data about an individual, an insurer is able to pre-fill an application with information for confirmation by an individually rather than through self-declaration which can be prone to both incorrect inputs and 'too-much-effort' drop-offs.",
            "category":"Use-Case"
          },
          {
            "name":"Lead Targeting",
            "description":"A FI’s outbound new customer acquisition is a function of the amount and quality of leads. Low conversion rates are typically a result of a lack of data enrichment resulting in poorly optimised marketing campaigns that include customers outside the FI’s target market. Specific marketing channels (e.g. digital vs call centre) will also favour different types of leads, but without more characteristics on the lead, no triage can take place. By leveraging external data sources, a FI can access a more expansive list of potential customers, enrich them with additional attributes and filtered to those with desirable characteristics. This can be enhanced with machine learning techniques where available data about converted leads can be used to inform a lead scoring model specific to the FI’s experience, improving conversion rates even more over time. Finally, based off a lead score and carefully chosen characteristics, channel triaging can occur to direct leads to the channel most likely to result in their conversion.",
            "category":"Use-Case"
          },
          {
            "name":"Portfolio Monitoring",
            "description":"A FI’s existing customer retention and cross-sell strategy is a function of the amount of information available about an existing customer. Knowing which customers are likely to churn or which you are not capturing their entire share of wallet is necessary to not waste retention marketing resources, or impact customer experience by conducting outreach to non-churning customers. By leveraging external data sources, a bank or insurer can monitor changes in a customer's enriched profile, such as lifestage, lifestyle or property attributes as indications of new cross-sell opportunities or risks of retention. Using an extended set of variables about each customer and actual churn experiences, a bank or insurer's can develop propensity to churn models specific to the bank's experience to further focus retention efforts with optimal offers, prompts and assistance.",
            "category":"Use-Case"
          },
          {
            "name":"eCommerce",
            "description":"",
            "category":"Industry"
          }
        ],
        "latest_version":{
          "id":10368,
          "schema":[
            {
              "input":{
                "optional":[
                  {
                    "name":"age_band",
                    "type":"Integer"
                  },
                  {
                    "name":"earliest_snapshot",
                    "type":"Date"
                  },
                  {
                    "name":"latest_snapshot",
                    "type":"Date"
                  }
                ],
                "required":[
                  [
                    {
                      "name":"post_code",
                      "type":"PostCode"
                    },
                    {
                      "name":"post_code_extension",
                      "type":"String"
                    },
                    {
                      "name":"country",
                      "type":"Country"
                    }
                  ]
                ]
              },
              "output":[
                {
                  "name":"is_hit",
                  "type":"Boolean"
                },
                {
                  "name":"results",
                  "type":"List",
                  "children":[
                    {
                      "name":"post_code",
                      "type":"PostCode"
                    },
                    {
                      "name":"post_code_extension",
                      "type":"String"
                    },
                    {
                      "name":"age_band",
                      "type":"Integer"
                    },
                    {
                      "name":"atp",
                      "type":"Integer"
                    },
                    {
                      "name":"country",
                      "type":"Country"
                    },
                    {
                      "name":"date",
                      "type":"Date"
                    }
                  ]
                }
              ]
            }
          ],
          "version":9,
          "input_fields":[
            {
              "description":"4 digit ZIP + 4",
              "log_flattened_name":"post_code_extension",
              "name":"post_code_extension",
              "type":"String"
            },
            {
              "description":null,
              "log_flattened_name":"latest_snapshot",
              "name":"latest_snapshot",
              "type":"Date"
            },
            {
              "description":null,
              "log_flattened_name":"earliest_snapshot",
              "name":"earliest_snapshot",
              "type":"Date"
            },
            {
              "description":"5 digit Zip code",
              "log_flattened_name":"post_code",
              "name":"post_code",
              "type":"PostCode"
            },
            {
              "description":"ISO 2 code for country",
              "log_flattened_name":"country",
              "name":"country",
              "type":"Country"
            },
            {
              "description":"Age band value from 0 to 7. 0 - Default. 1 - 18 to 24. 2 - 25 to 34. 3 - 35 to 44. 4 - 45 to 54. 5- 55 to 64. 6 - 65 to 74, 7 - 75 to 111",
              "log_flattened_name":"age_band",
              "name":"age_band",
              "type":"Integer"
            }
          ],
          "output_fields":[
            {
              "description":"Boolean indicating the match",
              "field_is_populated_rate":0.0,
              "log_flattened_name":"is_hit",
              "name":"is_hit",
              "type":"Boolean"
            },
            {
              "description":"5 digit Zip code",
              "field_is_populated_rate":1.0,
              "log_flattened_name":"results[*].post_code",
              "name":"post_code",
              "type":"PostCode"
            },
            {
              "description":"4 digit ZIP + 4",
              "field_is_populated_rate":1.0,
              "log_flattened_name":"results[*].post_code_extension",
              "name":"post_code_extension",
              "type":"String"
            },
            {
              "description":"Age band value from 0 to 7. 0 - Default. 1 - 18 to 24. 2 - 25 to 34. 3 - 35 to 44. 4 - 45 to 54. 5- 55 to 64. 6 - 65 to 74, 7 - 75 to 111",
              "field_is_populated_rate":1.0,
              "log_flattened_name":"results[*].age_band",
              "name":"age_band",
              "type":"Integer"
            },
            {
              "description":"A modeled continuous score of 1-1000 that ranks customers based on their estimated financial capacity and/or ability to pay debt(s), with a rating of 1000 being the most likely to be able to pay financial obligations.  Derived from IXI’s Financial Assets Database, it examines the relationship between spending, income and other variables to assess consumers’ ability to pay.",
              "field_is_populated_rate":1.0,
              "log_flattened_name":"results[*].atp",
              "name":"atp",
              "type":"Integer"
            },
            {
              "description":"ISO 2 code for country",
              "field_is_populated_rate":1.0,
              "log_flattened_name":"results[*].country",
              "name":"country",
              "type":"Country"
            },
            {
              "description":"Date the data was delivered and ingested by Demyst",
              "field_is_populated_rate":1.0,
              "log_flattened_name":"results[*].date",
              "name":"date",
              "type":"Date"
            },
            {
              "description":"Data container",
              "field_is_populated_rate":null,
              "log_flattened_name":"results",
              "name":"results",
              "type":"List"
            }
          ]
        },
        "output_fields":[
          {
            "description":"Boolean indicating the match",
            "field_is_populated_rate":0.0,
            "log_flattened_name":"is_hit",
            "name":"is_hit",
            "type":"Boolean"
          },
          {
            "description":"5 digit Zip code",
            "field_is_populated_rate":1.0,
            "log_flattened_name":"results[*].post_code",
            "name":"post_code",
            "type":"PostCode"
          },
          {
            "description":"4 digit ZIP + 4",
            "field_is_populated_rate":1.0,
            "log_flattened_name":"results[*].post_code_extension",
            "name":"post_code_extension",
            "type":"String"
          },
          {
            "description":"Age band value from 0 to 7. 0 - Default. 1 - 18 to 24. 2 - 25 to 34. 3 - 35 to 44. 4 - 45 to 54. 5- 55 to 64. 6 - 65 to 74, 7 - 75 to 111",
            "field_is_populated_rate":1.0,
            "log_flattened_name":"results[*].age_band",
            "name":"age_band",
            "type":"Integer"
          },
          {
            "description":"A modeled continuous score of 1-1000 that ranks customers based on their estimated financial capacity and/or ability to pay debt(s), with a rating of 1000 being the most likely to be able to pay financial obligations.  Derived from IXI’s Financial Assets Database, it examines the relationship between spending, income and other variables to assess consumers’ ability to pay.",
            "field_is_populated_rate":1.0,
            "log_flattened_name":"results[*].atp",
            "name":"atp",
            "type":"Integer"
          },
          {
            "description":"ISO 2 code for country",
            "field_is_populated_rate":1.0,
            "log_flattened_name":"results[*].country",
            "name":"country",
            "type":"Country"
          },
          {
            "description":"Date the data was delivered and ingested by Demyst",
            "field_is_populated_rate":1.0,
            "log_flattened_name":"results[*].date",
            "name":"date",
            "type":"Date"
          },
          {
            "description":"Data container",
            "field_is_populated_rate":null,
            "log_flattened_name":"results",
            "name":"results",
            "type":"List"
          }
        ],
        "current_organizations_credit_price":null
      }
    
    

Pull the CSV File Link for sample data for a single Connector

To pull the URL from which to get the sample data of a single connector, use the following cURL request:

    
      curl --location --request POST 'https://console.demystdata.com/table_providers/[ID]/download_sample_data' \
      --header 'Authorization: Bearer [BEARER]' \
      --header 'Accept: application/json' \
      --header 'Content-Type: application/json'
      
      
    

We will again use the provider with name hosted_equifax_ixi_ability_to_pay (viewable on our platform here: Demyst: The Data Deployment Platform) and id: 1490. Replacing [ID] becomes:

    
      curl --location --request POST 'https://console.demystdata.com/table_providers/1490/download_sample_data' \
      --header 'Authorization: Bearer [BEARER]' \
      --header 'Accept: application/json' \
      --header 'Content-Type: application/json'		
      
    
    

Within the output of this request will be a URL that will look something like "url": "[SAMPLE_DATA_URL]". Copy the entirety of the link provided in the response and use the following:

    
    curl --location --request GET '[SAMPLE_DATA_URL]'

    
    

To view CSV formatted sample data for the product.