AWS ADOP uses build-time AI agents on Bedrock to compress data engineering from weeks to hours
“the model fills in the blueprint. It doesn't draw it.”
data-engineering
28 tracked signals on data-engineering.
F1's agentic AI system cut data source onboarding from 8 weeks to 40 minutes
“For the first time, we have end-to-end visibility across the entire MarTech platform with data lineage and root cause analysis, not just dashboards full of alerts”
An RL agent autonomously diagnoses and remediates ETL pipeline failures with safety bounds, cutting manual recovery from ~2.5 days.
“The central question is simply whether an agent can act, but whether it can act usefully, explainably, and within the boundaries that an operation would actually trust.”
Databricks pitches Lakeflow as a unified platform to simplify fragmented data engineering for AI and agents.
“All these apps, agents, AI, everything needs data, and you might be thinking, "My job got a little bit harder."”
Databricks Lakebase enables Git-style database branching for CI/CD data workflows in production
AWS showcases an AI agent on Amazon EMR that troubleshoots and upgrades Apache Spark workloads.
DeepLearning.AI and AWS launch a Professional Certification Program in Data Engineering
Databricks LakeFlow Designer offers no-code visual pipeline building that generates real Python/SQL code
“I think the short answer is everybody could”
Databricks now supports running and debugging workloads directly from local IDEs
Databricks introduces Zerobus Ingest for petabyte-scale telemetry data ingestion.
Databricks makes Spatial SQL generally available with AI/BI Maps, Delta Sharing, and Iceberg v3 support
AI is reshaping the data engineer role across Fabric, VS Code, and CLI workflows.
“AI is changing the way that engineers work, not just by generating code, but by helping us rethink the full developer workflow across Fabric, VS Code and CLI.”
Databricks introduces declarative ETL patterns for Lakehouse SQL workflows
Databricks Variant data type for semi-structured ingestion is now generally available
Databricks offers a decision framework for migrating legacy ETL stored procedures and schedulers to its platform.
England's Office for Students cut a 300-million-record data job from 8 hours to minutes using Databricks.
Databricks details database branching with Lakebase to enable evolutionary, refactoring-based database development.
Databricks details database branching with Lakebase to enable evolutionary database development practices.
Databricks introduces Apache Spark Real-Time Mode for low-latency real-time sessionization in gaming.
“In the gaming industry, every millisecond counts.”
Databricks introduces database branching with Lakebase to enable evolutionary database development.
Databricks highlights its engineering and research presence at SIGMOD 2026.
Databricks IP Functions reach GA for high-performance network analytics in the Lakehouse
Databricks expands AUTO CDC capabilities to address complex real-world data engineering scenarios
Dow built a carbon footprint ledger on Databricks to scale sustainability tracking
Databricks showcases sports analytics app converting tracking data into coaching intelligence
A practical guide arguing range-based partitioning is the only good way to partition large Postgres tables.
“If your tables are around 100 gigabytes, that's a good moment to start to think about partitioning.”
Data pipeline architecture is the end-to-end design of how data is collected and processed.
Databricks offers guidance on optimizing BI dashboard performance and reducing TCO