In this video, we'll learn how to continuously load data into ClickHouse from AWS S3, using the S3Queue table engine. We start by exploring the config required on the serve to enable this functionality, before examining the S3 bucket that we're going to load into ClickHouse. We walk through the two modes of ingestion that we can use and adjust the flush frequency so that data is available in ClickHouse quicker. Finally, we use shadowtraffic.io to generate some more data and stream it into our S3 bucket, before checking that it's made its way into ClickHouse.
#Clickhouse #AWS #s3 #dataengineering #awss3
Resources
S3Queue docs - https://clickhouse.com/docs/en/engines/table-engines/integrations/s3queue
Configuring AWS credentials - https://clickhouse.com/docs/en/integrations/s3#managing-credentials
Data generation tool - https://shadowtraffic.io/
Don't forget to give us a ⭐ on Github! https://github.com/clickhouse/clickhouse
