Datasources

A datasource is a connection to an external data system. Pipeline uses datasources to fetch time-series data for your feeds.

Datasources can connect to supported file transfer sources, such as SFTP or FTPS servers, or to approved external API endpoints. Once a datasource has been added and tested, Pipeline can use it to create scheduled feeds.

Without a datasource, Pipeline has no way to reach your data. Datasources are therefore the starting point for automated Pipeline workflows.

Datasources

Adding a datasource

1. Go to Datasources in the left sidebar.
2. Click Add Datasource.
3. Choose the datasource type.
4. Enter the required connection details.
5. Select Next to run a connection test.
6. Review the data preview.
7. Add a name and description.
8. Select Save Datasource.

Choose Datasource Type

Pipeline currently supports datasource setup for file transfer sources and external API endpoints. The fields shown in the setup form depend on the datasource type selected.

SFTP datasource

Field	Description
Host	The public hostname of the SFTP server.
Port	The port used to connect to the server. SFTP usually uses port 22.
Username	Your SFTP login username.
Password / Private key	Depending on your server's authentication method.
File Path	The full path to the file on the server.

FTP via TLS datasource

Use an FTP via TLS datasource when your data is available through an FTPS server.

Field	Description
Host	The public hostname of the FTPS server.
Port	The port used to connect to the server. Use 21 for explicit FTPS, or 990 if your server requires implicit FTPS.
Username	The username used to connect to the server.
Password	The password used to connect to the server.
File Path	The full path to the file on the server.

Advanced TLS settings

For FTP via TLS datasources, you can optionally provide an Expected TLS hostname.

Use this only when the FTPS server certificate is issued for a different canonical hostname than the host you connect to. Pipeline still connects to the host entered above, but verifies the TLS certificate against the expected hostname.

Advanced TLS Settings

External API datasource

Use an External API datasource when Pipeline should retrieve data from a third-party API endpoint.

External API datasources are useful when your data is available through an HTTPS API rather than through a file transfer server. Pipeline can connect to the endpoint, test the response, detect available fields, and use the response data to create Pipeline feeds.

Note:

External API requirements can vary depending on the API provider. The API must return data in a supported structure, and the endpoint must be reachable by RI Platform.

Field	Description
Endpoint URL	The public HTTPS API endpoint that returns the data.
HTTP Method	The request method used by the API. GET is commonly used for read-only endpoints. JSON POST can be used where supported.
Expected response format	The expected response format. Auto-detect is recommended where available.
Authentication	The authentication method required by the API, such as no authentication, API key, bearer token, or basic authentication.
Advanced options	Optional headers, query parameters, incremental sync, pagination, and response parsing settings.

External API Authentication

Authentication

External API datasources can be configured with different authentication methods depending on what the API requires.

Supported options include:

• No Authentication - Use this when the endpoint is public or already protected by other network rules.
• API Key - Send an API key in a header or query parameter.
• Bearer Token - Send an authorisation bearer token.
• Basic Auth - Send a username and password using basic authentication.

Avoid placing secrets directly in the endpoint URL or request body. Use the authentication fields provided by the datasource setup form wherever possible.

Advanced options

Some APIs require additional configuration before Pipeline can retrieve data correctly.

Advanced options can include:

• Optional headers - Add fixed request headers required by the API.
• Optional query parameters - Add fixed query parameters, such as site IDs, meter IDs, filters, or response format options.
• Timestamp-based incremental sync - Add start and end time parameters where the API supports time filtering.
• Pagination - Configure page-based requests where the API returns results across multiple pages.

External API Advanced Options

Note:

Only enable incremental sync or pagination when the API supports the required request parameters. If an API does not support those parameters, the request may fail or return no data.

Testing the connection

A connection test is required before you can save a new datasource.

During the test, Pipeline connects to the datasource and checks that it can retrieve data successfully. If the test passes, Pipeline shows a preview of the data, including the columns and the first few rows.

For External API datasources, the connection test also helps Pipeline understand the API response. Pipeline checks the response format, identifies possible record locations, previews returned data, and detects fields that may later be used as feeds.

• The columns it found (headers).
• Whether it could identify a timestamp column.
• A preview of the first few rows.

File Transfer Connection Test

External API Test Success

For APIs that return nested JSON, Pipeline asks which part of the response contains the list of records. The selected records location tells Pipeline which part of the response should be treated as rows of data.

Select Records Location

If the test fails, an error message is shown. See Datasource connection issues below for specific guidance.

Saving a datasource

After the connection test succeeds, add a name and description for the datasource.

Use a clear name that helps you and your team recognise the datasource later. The description should explain what the datasource contains and how it will be used.

For External API datasources, the save step also summarises the configured endpoint, response format, records location, authentication status, and detected fields.

Save Datasource

Save External API Datasource

Editing a datasource

1. On the Datasources page, click the Update icon on the datasource row.
2. Update the fields you want to change.
3. Re-test the connection if required.
4. Save the updated datasource.

Note:

Changing the connection type may replace the stored column headers with those returned from the new test. Any feeds already mapped to the old headers are not automatically remapped.

Deleting a datasource

1. Click the Delete icon on the datasource row.
2. Confirm the deletion in the dialog.

Warning:

Deleting a datasource does not automatically delete the feeds created from it, but those feeds will stop receiving new data.

Datasource connection issues

External API test failed

• Confirm that the endpoint URL is correct and uses HTTPS.
• Confirm that the API endpoint is publicly reachable where required.
• Confirm that the selected HTTP method matches the API documentation.
• Confirm that any required API key, bearer token, basic authentication details, headers, or query parameters have been entered correctly.
• If using JSON POST, confirm that the request body is valid JSON.

External API response has no records

• Confirm that the API response contains a list of records.
• Check whether the correct records location has been selected.
• Confirm that any filters, query parameters, time ranges, or pagination settings are not excluding the expected data.
• Re-test the datasource after changing the configuration.

External API fields look wrong

• Confirm that the selected records location points to the repeating data records, not to a summary object.
• Check whether the API returns nested fields that need to be flattened before they can be used as feed columns.
• Confirm that the response format is correct.
• If the API sometimes returns different fields, test using a representative request.

Test failed - connection refused or timeout

• Confirm the host and port are correct.
• Confirm the server can be reached from Pipeline.
• For SFTP or FTP via TLS, check that the server allows connections from the Pipeline server.

Test failed - authentication error

• Double-check your credentials.
• Confirm that the account has permission to access the file.
• Confirm that the selected connection type matches the server configuration.

Test passed but data looks wrong

• Confirm that the file path points to the correct file.
• If the preview rows look empty or garbled, the endpoint may be returning a format that Pipeline cannot parse.
• Confirm that the file contains the expected columns.
• Confirm that the file includes a timestamp column.
• If the timestamp column is not detected, check that the column name contains the words timestamp, date, time, or datetime.

Connection was working and now fails

• The datasource may have changed its authentication credentials or URL.
• The file path may have changed.
• The datasource may be temporarily unavailable.
• Check the Alerts panel for a recent Critical or Major alert on this datasource.