The Streaming Load API is used to read data from your local files and load it into Databend.
Creating an API Request
To create a request with the Streaming Load API, follow the format below:
curl -H "insert_sql:<value>" -F "upload=@<file_location>" [-F "upload=@<file_location>"] -XPUT http://<user_name>:[password]@<http_handler_host>:<http_handler_port>/v1/streaming_load
insert_sql is required and must include an INSERT statement as well as the
FILE_FORMAT parameter that specifies the file formats. For details about
FILE_FORMAT, see Input & Output File Formats.
|insert_sql||[INSERT_statement] + [FILE_FORMAT]||All||-H "insert_sql: insert into ontime file_format = (type = 'CSV' skip_header = 1 compression = 'bz2')"|
Alternatives to Streaming Load API
The COPY INTO command enables you to load data from files using insecure protocols, such as HTTP. This simplifies the data loading in some specific scenarios, for example, Databend is installed on-premises with MinIO. In such cases, you can load data from local files with the COPY INTO command.
COPY INTO ontime200 FROM 'fs://<file_path>/ontime_200.csv' FILE_FORMAT = (type = 'CSV' field_delimiter = ',' record_delimiter = '\n' skip_header = 1);
To do so, you must add the setting
allow_insecure to the configuration file
databend-query.toml as indicated below and set it to
# fs | s3 | azblob | obs
type = "fs"
allow_insecure = true
For security reasons, Databend does NOT recommend insecure protocols for data loading. Use them for tests only. DO NOT set
true in any production environment.