Summary
PyArrow 21.0.0 is now required and brings preliminary variant type support; this release also fixes regressions in MERGE operations, partition column changes, S3 GIL contention, and Unity Catalog operations. New features include typed custom metadata support in CommitProperties and Musl arm64 wheels.
Summary from the captured archive. Follow the original source for full context.
More from delta-io/delta-rs
Delta-rs 1.0.0 removes the deprecated S3 DynamoDB log store and adds support for column mapping creation, V2 checkpoints, and deletion vectors in Change Data Feed. The release also introduces custom token credentials for Unity Catalog and fixes Spark interoperability bugs affecting OPTIMIZE and MERGE operations.
Python bindings now include a streaming history API and accept Path or PathLike objects in DeltaTable.is_deltatable. Key fixes resolve a Change Data Feed merge regression and preserve exact int64 decimal statistics at scale zero.
This release adds directory partitioning support to convert_to_delta and fixes encoding in column properties, partition detection accuracy, and naive datetime timezone handling. DeltaTableConfig is now deprecated and scheduled for removal.
Python v1.6.4 adds DeltaTable.scan for lazy engine-native reads, column mapping creation, writer features like row-group boundary rolling, and performance improvements in partition splitting and zero-copy uploads. The release removes PartitionFilter in favor of kernel predicates (a breaking change) and fixes stability issues in schema merging for partitioned tables, deletion vector handling, memory leaks during uploads, and metadata-only scan batching.
Added support for dropping NOT NULL column constraints and custom token credentials in Unity Catalog, plus lazy snapshot materialization and replay capabilities. Fixed delete operations on string columns and OPTIMIZE failures on Spark-written tables; Change Data Feed now respects commit timestamps and supports deletion vectors.
Python-delta v1.6.2 consolidates table URL parsing logic across platforms and exposes partition_values_map as a new public API for accessing partition metadata. The release also fixes merge operation barriers, preserves container values when null counts are invalid, and resolves SFTP dependency issues on Windows.