Databend is a powerful cloud data warehouse. Built for elasticity and efficiency. Free and open. Also available in the cloud: https://app.databend.com .
What's New
Check out what we've done this week to make Databend even better for you.
Features & Improvements ✨
Multiple Catalogs
- extends show databases SQL (#9152)
Stage
- support select from URI (#9247)
Streaming Load
- support
file_format
syntax in streaming load insert sql (#9063)
Planner
- push down
limit
tounion
(#9210)
查询
Storage
- use distinct count calculated by the xor hash function (#9159)
read_parquet
read meta before read data (#9154)- push down filter to parquet reader (#9199)
- prune row groups before reading (#9228)
Open Sharing
- add prototype open sharing and add sharing stateful tests (#9177)
Code Refactoring 🎉
*
- simplify the global data registry logic (#9187)
Storage
- refactor deletion (#8824)
Build/Testing/CI Infra Changes 🔌
Bug Fixes 🔧
Format
- support ASCII control code hex as format field delimiter (#9160)
Planner
- prewhere_column empty and predicate is not const will return empty (#9116)
- don't push down topk to Merge when it's child is Aggregate (#9183)
- fix nullable column validity not equal (#9220)
查询
- address unit test hang on test_insert (#9242)
Storage
What's On In Databend
Stay connected with the latest news about Databend.
Breaking Change: Unified File Format Options
To simplify, we're rolling out a set of unified file format options as follows for the COPY INTO command, the Streaming Load API, and all the other cases where users need to describe their file formats:
[ FILE_FORMAT = ( TYPE = { CSV | TSV | NDJSON | PARQUET | XML} [ formatTypeOptions ] ) ]
- Please note that the current format options starting with
format_*
will be deprecated. ... FORMAT CSV ...
will still be accepted by the ClickHouse handler.- Support for customized formats created by
CREATE FILE FORMAT ...
will be added in a future release:... FILE_FORMAT = (format_name = 'MyCustomCSV') ....
.
Learn More
- Issue | unify format options and remove format_ settings
- PR | support file_format syntax in streaming load insert sql
Open Sharing
Open Sharing is a simple and secure data-sharing protocol designed for databend-query nodes running in a multi-cloud environment.
- Simple & Free: Open Sharing is open-source and basically a RESTful API implementation.
- Secure: Open Sharing verifies incoming requesters' identities and access permissions, and provides an audit log.
- Multi-Cloud: Open Sharing supports a variety of public cloud platforms, including AWS, Azure, GCP, etc.
Learn More
What's Up Next
We're always open to cutting-edge technologies and innovative ideas. You're more than welcome to join the community and bring them to Databend.
Refactoring Stage-Related Tests
We're about to run stage-related tests again using the Streaming Load API to move files to a stage instead of an AWS command like this:
aws --endpoint-url ${STORAGE_S3_ENDPOINT_URL} s3 cp s3://testbucket/admin/data/ontime_200.csv s3://testbucket/admin/stage/internal/s1/ontime_200.csv >/dev/null 2>&1
This is because Databend users do not need to take care of, or do not even know the stage paths that the AWS command requires.
Issue 8528: refactor stage related tests
Please let us know if you're interested in contributing to this issue, or pick up a good first issue at https://link.databend.rs/i-m-feeling-lucky to get started.
Changelog
You can check the changelog of Databend Nightly for details about our latest developments.
- v0.8.152-nightly
- v0.8.151-nightly
- v0.8.150-nightly
- v0.8.149-nightly
- v0.8.148-nightly
- v0.8.147-nightly
- v0.8.146-nightly
- v0.8.145-nightly
Contributors
Thanks a lot to the contributors for their excellent work this week.
ariesdevil | b41sh | BohuTANG | Chasen-Zhang | ClSlaid | dantengsky |
drmingdrmer | hantmac | lichuang | mergify[bot] | PsiACE | RinChanNOWWW |
soyeric128 | sundy-li | wubx | Xuanwo | xudong963 | youngsofun |
ZhiHanZ | zhyass | zzzdong |
Connect With Us
We'd love to hear from you. Feel free to run the code and see if Databend works for you. Submit an issue with your problem if you need help.
DatafuseLabs Community is open to everyone who loves data warehouses. Please join the community and share your thoughts.
- Databend Official Website
- GitHub Discussions (Feature requests, bug reports, and contributions)
- Twitter (Stay in the know)
- Slack Channel (Chat with the community)