Data lakehouse architecture: streaming ingestion and zero-copy data mirroring with Qlik

Qlik has enhanced its Open Lakehouse platform with streaming data ingestion capabilities, real-time transformations, and extended integration with the Apache Iceberg ecosystem. This is a response to the growing demands of teams building analytical and AI architectures on open formats, without vendor lock-in, and with full control over data in their own cloud.

What’s New in Qlik Open Lakehouse

The most significant changes concern three areas: ingestion, transformations, and integrations. First: streaming ingestion. Qlik Open Lakehouse now supports streaming data ingestion from Apache Kafka, Amazon Kinesis, and Amazon S3 directly into managed Apache Iceberg tables. Data lands in the customer’s cloud (Amazon S3) without engaging data warehouse compute resources. Ingestion occurs on dedicated, cost-effective compute, which lowers costs and increases scalability. Second: streaming transformations. Data can be processed on the fly – cleaned, filtered, normalized, and nested structures flattened – even before it reaches the target Iceberg table. This eliminates the need for additional ETL layers and shortens data availability time for downstream consumers. Third: automatic Iceberg optimization. Qlik manages file compaction and metadata updates, which is crucial for maintaining query performance with large volumes of streaming data. Users do not need to manually run OPTIMIZE or VACUUM operations – the platform does it automatically.

Integrations with the Iceberg Ecosystem

Qlik has extended support for tools and platforms that work with Apache Iceberg:

Snowflake Open Catalog – in addition to AWS Glue, Qlik now also supports the Snowflake catalog
Zero-copy mirroring to Databricks and Amazon Redshift – data in Iceberg is available in these platforms without duplication, analogous to existing mirroring to Snowflake
Extended compatibility with Apache Spark – seamless access to current Iceberg tables
Support for Amazon Athena, Amazon SageMaker Studio, Trino, Presto – data written once is queryable by multiple engines

Zero-copy mirroring is a key feature for hybrid lakehouse-warehouse architectures. Data physically remains in Iceberg on S3 but is visible and accessible in Snowflake, Databricks, or Redshift without additional copies. This reduces storage costs and simplifies management.

Governance and Data Quality

Qlik automatically applies governance to data in Open Lakehouse. Each Iceberg table has:

Data lineage – full traceability of data origin, from source to consumer
Qlik Trust Score – an automatically calculated indicator of data quality and reliability
Catalog services – automatic metadata catalog updates
Data quality checks – data validation in pipelines

Governance here is not an overlay or a separate tool – it is built into the Qlik Talend Cloud platform and operates at the level of pipelines, tables, and mirrored datasets.

Architecture and Deployment

Qlik Open Lakehouse manages Apache Iceberg tables on Amazon S3 in the customer’s environment. Streaming pipelines write events and apply transformations as data flows. Compression and metadata updates occur automatically, maintaining performance at scale. Data is written once, in the open Iceberg format, and is available to multiple analytical engines without duplication. This is a classic lakehouse architecture, but with management that eliminates typical problems: lack of optimization, catalog chaos, missing lineage, and lack of quality control.

Availability

The new streaming ingestion and streaming transformations features are expected to be generally available in Q1 2026 for Qlik Talend Cloud customers. Support for Snowflake Open Catalog, extended compatibility with Apache Spark, and zero-copy mirroring to Databricks and Amazon Redshift will be rolled out in phases, starting from Q1 2026, with a regional schedule announced as features become available. For teams building modern data architectures on Apache Iceberg, Qlik Open Lakehouse offers a managed platform with full governance, without sacrificing openness and control over data. This is a combination that is difficult to achieve in practice – and precisely why it is worth noting.