HF Buckets: Upload Only What Changed with Xet
Hugging Face Xet uses content-defined chunking to upload only changed file bytes
“Think Git, but for pieces of a file instead of whole files, and there's nothing to install.”
Hugging Face's Xet storage layer splits files into ~64KB content-defined chunks, fingerprints each piece, and only uploads chunks the Hub doesn't already have — eliminating redundant bandwidth on incremental dataset updates. The feature is built into the HF CLI with no separate install, and works with Parquet files when PyArrow's content-defined chunking option is enabled. Relevant for ML engineers managing large, frequently-updated datasets, but represents an infrastructure improvement rather than a strategic industry shift.