How to setup a polling integration
Issue
The events poller allows a customer to pro-actively retrieve events by way of polling an API for data. As the CDP is an event driven system, the events-api is reactive in getting it’s events from third-party system. After the events poller loads in it’s data, it forwards them to the events-api.
As such, the following requirements are part of it’s responsibilities:
- Being able to load in data from an API;
- Being able to add standard authorization to these API calls
- Having support for pagination;
- Being able to verify your dataset.
Product
- Mobile Marketing Cloud
- Customer Data Platform
Resolution
How to setup an event poller
In an account, start with adding an EventsPoller integration.
We will start by getting cat facts into the CDP. For that, we will use the following example API:
https://dogapi.dog/api/v2/breedsWhile we support both HTTP and HTTPS, we do expect these HTTPS calls to have a valid and active SSL certificate behind it.
When using the “Skip SSL certification” option, you can skip this validation on the SSL certificate. This is generally discouraged and should be only be used for testing purposes. Skipping the SSL certificate by default could incur the risk of man-in-the-middle attacks on the customer data.
We currently have the following 3 options available for authentication:
- No authentication (generally used for public API’s);
- Token (generally used to access other CM.com API’s. Please refer to acquiring an API token for those services).
- Depending on the service, you will need to specify if the token needs to be part of the URL (default) or part of the Header.
- OAuth2
- We support authorization with OAuth2. The basis for OAuth2 is that one first needs to retrieve a valid token indicating access to a specific API resource. With that token, we are able to perform an authorized request to the API.
Note: We will attempt to authorize with the token where either the test of a previous run succeeded with. When the system cannot load the data, the authorization will be made invalid and a new token is attempted to retrieve. A maximum of 3 attempts to call the API is attempted per interval.
After setting up the API details, it’s time to give the event type a name. As our example is in loading dog breeds, the event name would be “DogBreed”.
As the system will be polling for data, we will need to give it a time between the interval it needs to do so. This heavily depends on the integration what is required.
We will not attempt to load data, while the previous attempt has not finished.
Some parties choose to rate limit the number of allowed requests. Please verify with a API owner if such restrictions apply.
There is an option for an advanced interval setup using CRON. This advanced setup is based on UTC, but will allow you to better finetune your setup. For example, it’s possible to setup a hourly setup that only executes on Monday, Tuesday and Saturday.
You will also need to setup the data container name. By default, the system will search for the result field in the result. Should the API result contain multiple result items, this will translate into multiple events sent to the events-api. Should the data not be specified in the field result field but in another field, please adjust accordingly.
In our dog breed example above, the data we need is not in the result field but in the datafield. This will result in our setup looking as follows:
Before completing the integration, you will need to test the intergration. If your API result returns an empty list and/or empty data, the issue could be in either the authentication or an incorrect data container name.
Result
The output of the intergration is that data is sent towards the events-api with the specified setting. By default, an empty event type is created where the data is sent to. Adding scripts and/or output event types and/or mappings are currently not generated. This will require more actions by our professional services department or your customer success manager to make sure that the event ends up in the profile.
Pagination
When working with restful API’s, it’s possible that loading in a specific resource may return slow and large payloads. To be able to better support such situations, an option is added to support pagination.
Setting | Purpose | Example |
page parameter name | Set’s the name of the query parameter, where the current page is set to. | When using value pagina, the URI will have the following when on page 5. www.example.com?pagina=5 |
Currentpage parameter name | We expect for a paginated response, the result to contain the total number of pages we will need to result. This sets the field name for where we can find the number of pages | When using the example as above. Setting the value to “total_pages” would mean that 10 pages in total will be loaded. (With the page name being in the URI in the page parameter name) |
Pagelimit parameter name | Some API’s allow you to specify the number of items to be returned per page. This setting allows you to set the query parameter name for such limit | When using value pagina_limiet, the url will have the following with page limit size 123 www.example.com?pagina_limiet=123 |
Page limit | Closely correlated to the page limit parameter name. This sets the actual value of the page limit | When using value 123, the url will have the following with page limit size parameter pagina_limiet www.example.com?pagina_limiet=123 |
Api uses datetime
If you want to specify a specific start date for your poller or if your API expects a start/end date, you can check the ‘Api uses datetime’ option.
Setting | Purpose | Example |
Parameter date_start name | Sets the name of the query parameter that specifies the start date | 'startDate' |
Parameter date_end name | Sets the name of the query parameter that specifies the end date | ‘endDate’ This will result in:
We always set this value with the current date, at the moment of executing. |
Start date | Set a specific start date from when the poller should poll data. | 2024-01-01 |
