๐Setting up an HTTP API connector
Overview
This is a generic connector to get data from any HTTP APIs (REST style APIs). Itโs really customizable and versatile but it implies a more complex configuration.
This type of data source combines the features of Pythonโs requests library to get data from any API with the filtering langage jq for flexible transformations of the responses. Optionally, an xpath string can be provided to first parse the XML response and then the jq filter is be applied to get the data in tabular format.
Information
The connector doesn't handle paginated results.
Configuring the connector
To configure this connector you will need to use the documentation of the API you need to connect to.
Responsetype
The type of response the connector has to expect from the queried API.
Make sure you use the correct responsetype, based on the queried APIโs documentation. Currently JSON & XML are supported, the default being JSON.
Retrypolicy
Defines how the connector should behave when the network is unreachable:
MAX ATTEMPTS
: number of attempts to do before aborting the connexionMAX DELAY
: total time to wait before aborting the connexionWAIT TIME
: time to wait between each attempt
Certificate
If the connector must use a certificate to establish the connexion, you can provide the path to the certificate.
Auth
The authentication method that the connector should use to query the data. AUTHTYPE
Can be:
basic
: username password, you can provide them inpositional arguments
: input your username and password in the right ordernamed arguments
: input them this way {โusernameโ:โmyusernameโ, โpasswordโ:โmypasswordโ}
digest
: same as aboveoAuth1
:positional arguments
: input client_id (sometimes named client_key) and client_secret. Both are provided by the service you are trying to accessnamed arguments
: input {โclient_idโ:your_client_id, โclient_secretโ: your_client_secret}.
oAUth2
:positional arguments
: enter one by one (in the right order), the URL to access to the authentication endpoint (e.g. https://login.mywebsite.com/oauth2/token), the โclient_IDโ (sometimes named โclient_keyโ) and the โclient_secretโ. These informations are provided by the service you are trying to accessnamed arguments
: input {โclient_idโ:your_client_id, โclient_secretโ: your_client_secret}.
CustomTokenServer:
provides a flexible mechanism for authenticating API requests using a custom token server. the token you get is then sent in the theAuthorization
header prefixed with "Bearer
{{your_token}}"
. In thenamed arguments
section you have to fill as a json dict the required elements to get your token:method
: The HTTP method to use when requesting the token (e.g., 'GET', 'POST').url
: The URL to get the token server.params
(optional): Query parameters to include in the token request.data
(optional): Form data to include in the token request body.headers
(optional): Additional headers to include in the token request.json
(optional): JSON payload to include in the token request body.filter
(optional): A JQ-style filter to extract the token from the response. Defaults to "." (root of the JSON response).
Template
You can use this object to avoid repetition in data sources. The values of the three attributes will be used or overridden by all data sources using this connector.
json
: a JSON object of parameters to send in the body of every HTTP request made using the configured connector. Example: { โoffsetโ: 100, โlimitโ: 50 }headers
: a JSON object of parameters to send in the header of every HTTP request made using the configured connector. Example: { โcontent-typeโ: โapplication/xmlโ }params
: a JSON object of parameters to send in the query string of every HTTP request made using the configured connector. Example: { โoffsetโ: 100, โlimitโ: 50}proxies
: JSON object expressing a mapping of protocol or host to corresponding proxy. Example {โhttpโ: โfoo.bar:3128โ, โhttp://host.nameโ: โfoo.bar:4012โ}
Selecting data from the API
parameters
: A JSON object that will be used for variables interpolation in the query string. For testing purpose only. In production mode, it should be left blank as variable interpolation will be handled by the app requester.url
: The APIโs endpoint you want to query, it will be appended to the baseroute URL defined in the connector โ ๏ธ as it cannot be empty in the case when the API doesnโt have endpoint, you can split the baseroute url defined in the connector and put the last part in the datasource. Ex: https://example.com/API in connector and /v1 in datasourceMethod
: Defines the http method you want the datasource to perfom, GET, POST or PUT. Default is GET. You can find the method you need in the documentation of the API you want to queryheaders
: a JSON object of parameters to send in the header of every HTTP request made using the configured connector. Example: { โcontent-typeโ: โapplication/xmlโ }. Overwrites the headerโs parameter in Templateparams
: a JSON object of parameters to send in the query string of every HTTP request made using the configured connector. Example: { โoffsetโ: 100, โlimitโ: 50} Overwrites the params parameter in Templatejson
: a JSON object of parameters to send in the body of every HTTP request made using the configured connector. Example: { โoffsetโ: 100, โlimitโ: 50 } Overwrites the JSON parameter in Templateproxies
: JSON object expressing a mapping of protocol or host to corresponding proxy. Example {โhttpโ: โfoo.bar:3128โ, โhttp://host.nameโ: โfoo.bar:4012โ} Overwrites the proxies parameter in Templateflatten column
: optional field where you can specify the name of a column that contains nested rows. the column names in the resulting DataFrame will be prefixed with the original column name. Specified more parameters using a,
delimiter. If specified, the nested rows will be flattened into separate columns in the resulting data frame. Example if you have a column orders: [{"id": 3, "product": "Notebook", "price": 5.99}] results will be separated in orders_id, orders_product and orders_pricedata
: Two options, Type1 for a simple string, Type2 for a JSON field. ๐ก you can send XML data with Type1 optionxpath
: If the reply from the API contains XML data you can parse it with an xpath string. See documentation: xpath Example:
In the connector weโll have a response like this:
And we can then apply a:
filter
: String containing a jq filter applied to the data to get them in tabular format. See documentation: jq Example:
Letโs take the JSON defined above
We apply the filter โ.bookstore.book[]โ which means that it will extract the book
list from the bookstore
So we end up with a table of results looking like this:
Note: the reason to have a filter
option is to allow you to take any API response and transform it into something that fits into a column based data frame.
Example of connection to Open Data Paris
Setting up the connection to Open Data Paris
Selecting data from Open Data Paris
The JSON response looks like this:
We apply the filter .records[].fields
which means that for every entry in the records
property, it will extract all the properties of the fields
object. So we end up with a table of results looking like this (Iโm skipping columns in this example, but you see the point):
Note: the reason to have a filter
option is to allow you to take any API response and transform it into something that fits into a column-based data frame. jq is designed to be concise and easy for simple tasks, but if you dig a little deeper, youโll find a feature functional programming language hiding underneath.
Performance If the HTTP API connector is used in a live context, make sure that the API is performant enough and is able to retrieve data fast. In order to have suitable performance, make sure to retrieve a limited amount of data since its need additional transformation in order to unnest the data (in the case of json response).
After selecting data from your connector you will be able to create a dataset thanks to YouPrep using the selection as "source step".
Last updated