Toucan 3.0
YouPrep documentationHelp centerGet a demo
  • Welcome
    • 👋Welcome to Toucan
    • ⚙️Technical resources
      • ⚙️Toucan stack
      • Setup mode
        • Toucan SaaS mode
      • ⚙️Security
        • Application Security
        • Source Code Quality
        • Global Security Practices
        • Security of Docker Images
  • TUTORIALS
    • 📊Getting Started : Embedded Analytics
    • 🤓Advanced tutorials
      • Embedding a story with user attributes
        • Dynamic filter with user attributes
        • Dynamic Tables
        • Dynamic Database
        • Dynamic Host
      • Using the HTTP API connector in advanced use cases
      • Using advanced syntax for SQL queries
      • Merging filters with our tool
      • Deep customization chart (CSS)
        • Homepage customization
        • Chart customization
        • Dashboard customization
  • Data Management
    • 🧮Overview of Data In Toucan
    • 📡Datasources in Toucan
      • 🔌Managing Connectors
        • 🔌Creating, editing and deleting a connector
        • 🔌Set up OAuth2 credentials for your platform
        • 🔌Setting up a connector
          • 🔌Generic Connectors
            • 🔌Setting up an HTTP API connector
            • 🔌Setting up an ODBC Connector
          • 🔌Database and data warehouse Connectors
            • 🔌Setting up an AWS Redshift Connector
            • 🔌Setting up a Snowflake Connector
            • 🔌Setting up a PostgreSQL Connector
            • 🔌Setting up a Google Big Query Connector
            • 🔌Setting up an AWS Athena connector
            • 🔌Setting up a MySQL connector
            • 🔌Setting up a MongoDB connector
            • 🔌Setting up a Microsoft SQL Server connector
            • 🔌Setting up an Azure SQL connector
            • 🔌Setting a Databricks Connector
            • 🔌Setting up a ElasticSearch Connector
            • 🔌Setting up a Clickhouse Connector
          • 🔌Online services connectors
            • 🔌Setting up a Sharepoint Connector
            • 🔌Setting up a Google Sheets Connector
            • 🔌Setting up a Salesforce Connector
            • 🔌Setting up a Hubspot Connector
          • 🔌Setting up an AWS S3 connector
      • 🗄️Managing remote file storages
        • 🗂️Setting up a file storage
          • SFTP
        • 📄Create a dataset from a file
      • 📁Managing Files
        • 📁Adding, editing and deleting local files
        • 📂Using advanced file settings
        • 📁Adding and combining remote files in Toucan
    • 🔢Datasets in Toucan
      • 🔢Stored and Live Datasets
      • 💿Managing datasets
        • 🔢Creating datasets
        • 🔢Editing, Duplicating and Deleting a dataset
        • 🔢Refreshing and Publishing Datasets
        • 📈Optimize data performance
        • 🗂️Adding indexes to stored datasets
        • 👩‍💻Code mode and single mode
      • 🛑Setting permissions on dataset
      • 🗃️Maintaining Data
        • 🗃️Tagging datasets
        • 🗃️Identifying datasets dependencies
        • 🗃️Set validation rules
    • 🧑‍🍳Preparing data
      • Overview of YouPrep™
        • 🎹Column header
          • Rename column
          • Duplicate column
          • Fill null values
          • Replace values
          • Sort values
          • Convert columns data types
        • Add
          • Add text column
          • Add formula column
          • Add conditional column
        • Filter
          • Delete columns
          • Keep columns
          • Filter rows
          • Top N rows
          • ArgMax
          • ArgMin
        • Aggregate
          • Group by
          • Add total rows
          • Hierarchical rollup
          • Get unique groups/values
        • Compute
          • Compute evolution
          • Cumulated sum
          • Percentage of total
          • Rank
          • Moving average
          • Compute statistics
          • Absolute value
        • Text
          • Concatenate
          • Split column
          • Extract substring
          • To lowercase
          • To uppercase
          • Compare text columns
          • Trim spaces
          • Replace text
        • Date
          • Convert text to date
          • Convert date to text
          • Extract date information
          • Add missing dates
          • Compute duration
          • Normalize date granularity
        • Reshape
          • Pivot
          • Unpivot
          • Waterfall
        • Combine
          • Append datasets
          • Join datasets
        • Geo
          • Geographic dissolve
          • Geographic hierarchy
          • Geographic simplify
          • Prepare geo data (with basemap)
      • YouPrep™ Native SQL
      • Hybrid pipeline
    • ➿Managing variables in Toucan
      • ➿Variables hub
      • ♈Use variables in YouPrep™
      • ➿Easy reference to variables
    • 🧞Using advanced data concepts
      • 🧞Data personnalisation with user attributes
        • Connector setup with a user attribute
        • Database selection with a user attribute
        • YouPrep data filtering with a user attribute
        • Filter data in SQL with a user attribute
      • 🧞Advanced syntax for variables
      • 🧞Data cache
  • Visualizations and Layouts
    • 📺Apps
      • 📺Managing Apps
        • ➕Creating Apps
        • 📄Duplicating Apps
        • 🖨️Publishing Apps
        • 🚮Deleting Apps
        • ✍️Editing within an App
      • 🖌️Customizing Apps
        • Customizing chart color elements
        • Customizing the app's font
        • Adding Assets
        • Adding Glossary
        • Setting up, Managing and testing custom visibilities
        • Customizing the "no data error" message
        • Creating a dynamic background based on an Filter's column
      • 🏠Home
        • Creating the Home
        • Creating Tiles
          • Tile Dynamic Value
          • Tile Leaderboard
          • Tile Line
          • Tile Scorecard
          • Tile Bullet
          • Tile Heatmap
          • Tile PDF
          • Tile Video
          • Tile Image
          • Tile Text
          • Tile HTML
          • Tile Separator
      • ✨Stories
        • Creating a Story
        • KPIs
        • Narrative
        • Crossfilter
      • 📽️Filters
        • Managing Filters
          • Creating, reusing and editing Filters
          • Applying Filters
          • Unpinning and deleting Filters
        • Type of Filters
          • Dropdown
          • Checkboxes
          • Buttons
          • Date Range
          • Hierarchical
          • Slider
        • Templating from Filters' values
        • Dependant Filters
      • 📈PDF Report
      • 🎡Datawall
      • 🏗️Dashboard Builder
        • Create a Dashboard Builder
        • Embed a Dashboard Builder
        • Dashboard export options
      • 🌟MyFavorites
    • 📊Creating Visualizations
      • 🤩Viz Gallery
        • ✨New Viz Designs
          • Barchart
          • Barlinechart
          • Bubblechart
          • Circularchart
          • Heatmap
          • Leaderboard
          • Linechart
          • Stacked Barchart
          • Tablechart
        • 📁Legacy Viz Designs
          • Bulletchart
          • Funnelchart
          • Gantt chart
          • HTML
          • Leaderboard Centered Average
          • Mapchart
            • Configure a drill
          • Mediachart
          • Radarchart
          • Timeline
          • Versuschart
          • Waterfallchart
          • Score Card
      • 🧠Common Chart Configuration
      • 💅Customizing chart colors
      • 🧞‍♂️Advanced chart configuration
        • Templating from chart's dataset
        • Add units, precisions and sentiments
        • Adding Tutorials
        • Add sparklines
        • Navigate with stories
        • Group informations in your stories
        • Multiple Charts in One Story
        • Manage dates
        • Customize tiles' sources
        • Add stars to tiles' title
        • Manage data order in your tiles
        • Navigate with tiles
    • 👩‍💻Embedding
      • 🔐Authentication
      • 🖇️Integration
        • Generate and manage embeds
        • Customize embeds
        • Embedding a Toucan App Using iFrames
        • Passing Extra Variables to Your Toucan Embed
      • ⚙️Embed SDK
        • Embed SDK Authentication
      • ❓FAQ
    • 🙋Self-Service
      • Self-Service Dashboard
        • The embedded Self-Service Dashboard
        • The in-platform Self-Service Dashboard
      • Self-service PDF Report
  • Collaboration
    • ⏰Creating alerts
    • 📧Managing notifications
    • ➕Enriching a story with descriptions
    • 💌Sharing content
    • 💬Adding comments to stories
  • Administration
    • ⚙️Instance Management
      • ⚙️Managing operations in SaaS
      • ⚙️Customizing your instance (whitelabel)
    • 👥Managing Users
      • 👥Users
      • 👥Managing user groups
      • 👥Managing user properties
      • 👥Setting up permissions and visibilities
    • 🌐Managing languages in Toucan (internationalisation)
    • 📈Monitoring Engagement with User Analytics
      • 🎛️How to Filter your User Analytics?
      • 🖥️Understanding your User Analytics Dashboards
  • Additional Ressources
    • 📚External documentation
    • 🚁Support for App-builders
    • 🆕Latest releases
      • 🎁2025 Releases
      • 🎁2024 Releases
      • 🎁2023 Releases
    • 🔧Troubleshooting
      • Troubleshoot:: DataHub
      • Cross-Site Cookies
      • How to :: read the inspector error
      • How to :: troubleshoot the toucan way
Powered by GitBook
On this page
  • Adding a remote file in Toucan
  • FTP Server
  • S3 Bucket
  • Combining a remote file in Toucan

Was this helpful?

  1. Data Management
  2. Datasources in Toucan
  3. Managing Files

Adding and combining remote files in Toucan

Last updated 10 months ago

Was this helpful?

Toucan ca read files over a network connection for a large variety of protocols.

Adding a remote file in Toucan

In order to add a remote file in Toucan, use the code mode of file settings.

Follow the following steps to access it:

  • Upload a csv empty file (or any small random csv file) within Toucan

  • Switch to code mode within the configuration interface

  • Replace the fields of the code block depending on the distant file server and configuration associated. Refer to the sections below the fields to fill.

  • Save File settings. A new file should appear in the listing of files (in datasources). A dataset will be also automatically created.

Example with a CSV file on Dropbox.

domain: 'my_remote_data'
type: 'csv'
file: 'https://www.dropbox.com/s/9yu9ekfjk8kmjlm/fake_data.csv?dl=1'

We support:

  • ftp (as well as sftp or ftps),

  • http (and https),

  • S3 and

  • a long list of other schemes (‘mms’, ‘hdl’, ‘telnet’, ‘rsync’, ‘gopher’, ‘prospero’, ‘shttp’, ‘ws’, ‘https’, ‘http’, ‘sftp’, ‘rtsp’, ‘nfs’, ‘rtspu’, ‘svn’, ‘git’, ‘sip’, ‘snews’, ‘tel’, ‘nntp’, ‘wais’, ‘svn+ssh’, ‘ftp’, ‘ftps’, ‘file’, ‘sips’, ‘git+ssh’, ‘imap’, ‘wss’).

FTP Server

  • Mandatory: access to a FTP server and to Toucan staging mode on your workspace

  • Open Filezilla or any FTP client

  • Copy the URL corresponding to the location of the file on the FTP server it should look like this:

ftp://user:password@example.com/pub/file.txt

"ftp" is the protocol used, "user" and "password" are the login credentials "example.com" is the domain of the server, and "/pub/file.txt" is the full path to the file on the server.

Important

💡Contact us via help@toucantoco.com or your Delivery contact to set up a hidden password in the URL

  • Paste this url in the file field and modify the other configuration fields configuration as explained above in Adding a remote file in Toucan

  • A new file should appear in the listing of files (in datasources). A dataset will be also automatically created.

domain: 'db_test'
type: 'csv'
file: 'ftps://<login>:<password>@ftps.toucantoco.com:990/my_db.csv'
separator: ";"

Toucan Toco FTP Server

You can send data to Toucan Toco FTP Server with the following credentials:

  • Host: ftps.toucantoco.com

  • Port: 990 (for the connection) and range 64000-64321 (for data transfert)

  • Protocol: FTPS (if you use FileZilla it’s implicit FTP over TLS)

  • Mode: Passive Mode

  • User : Given by the Toucan Toco Team

  • Password: Given by the Toucan Toco Team

S3 Bucket

The access key and secret key for your data files hosted on S3 buckets can be configured this way:

s3://<access key>:<secret key>@mybucket/filename'

For example:

domain: 'my_data'
type: 'csv'
file: 's3://<access key>:<secret key>@mybucket/my_data.csv'
separator: ";"

Note

If your access key or secret key contains special characters such as “/”, “@” or “:” you have to encode them. URL encoding converts special characters into a format that can be transmitted over the Internet. You will find more infos about this topic here (as well as an automatic encoder).

Toucan Toco can provide a S3 bucket with a dedicated AWS IAM user related to your instance.

Thus you will be able to configure your datasources block with a special configuration as following:

domain: 'my_data'
type: 'csv'
file: "{{ secrets.extra.s3.s3_uri_auth_encoded }}/my_data.csv"
separator: ";"

Note

If you are using a custom domain name for your S3 bucket using minio per example. Here is the syntax you should use

domain: 'my_data'
type: 'csv'
file: 's3://<access key>:<secret key>@mybucket/my_data.csv'
separator: ";"
fetcher_kwargs:
    client_kwargs:
        endpoint_url: "https://endpoint.mydomain.com:9000"

Combining a remote file in Toucan

In the previous page, we saw how to add remote files in Toucan. Read the previous page first, before going further with this one.

In this page, we will discover how to combine several remote files into one file.

You can load multiple files - uploaded on our server or on a FTP/S3 server - in a unique file with the option match: true. The dataset that will be created from the file will contain a column __filename__ corresponding to the origin file of the row.

Tutorial

Your corporation has now a new file of data each month : data-product-corporation-201801.csv, data-product-corporation-201802.csv … You want them to be loaded in a single domain called data-product-corpo

  • Find the regular expression (regex) that matches your files with regex101.com.

data-product-corporation-\d{6}\.csv

  • Don’t forget to use ‘^’ and ‘$’ to be more restrictive.

^data-product-corporation-\d{6}\.csv$

  • Add a backslach to escape backslaches.

^data-product-corporation-\\d{6}\\.csv$

  • Copy your regular expression in the "file" option of your datasource block

  • Add the option match: true

domain: 'data-product-corpo'
file: '^data-product-corporation-\\d{6}\\.csv$'
skip_rows: 0
separator: ','
encoding: 'utf-8'
type: 'csv'
match: true

Example of content for FTP (with authentication):

domain: 'db_test'
type: 'csv'
file: 'ftps://<login>:<password>@ftps.toucantoco.com:990/^data-product-corporation-\\d{6}\\.csv$'
separator: ";"

Example of content for S3 (with authentication):

domain: 'my_data'
type: 'csv'
file: 's3://<access key>:<secret key>@mybucket/^data-product-corporation-\\d{6}\\.csv$'
separator: ";"
fetcher_kwargs:
    client_kwargs:
        endpoint_url: "https://endpoint.mydomain.com:9000"

📡
📁
📁