Gen AI has created unprecedented demand for reliable data orchestration.
Enterprise AI initiatives require consistent access to clean, properly formatted data from multiple sources. Enterprises typically have data in multiple locations, technologies and formats and bringing all that data together into a data pipeline that can then be ingested and used for AI training, inference and retrieval augmented generation (RAG) is not a trivial task. There are many different tools in the space including the widely used open source Apache Airflow technology and its commercial implementation with Astronomer.
While Airflow is widely used, it's not the only open data orchestration tool. Today Kestra 1.0 is being released providing a solid competitive open source option. Kestra takes a somewhat different approach to data orchestration than Airflow that aims to solve a broader set of enterprise data requirements. Though the technology is only officially hitting its 1.0 status today, Kestra already has deployments already running production workloads at Apple, Toyota, Bloomberg and JPMorgan Chase. The platform's 1.0 release introduces AI-generated workflows that maintain enterprise governance controls.
"We pioneered declarative orchestration," Emmanuel Darras, Kestra's CEO and co-founder, told VentureBeat in an exclusive interview. "With 1.0, we elevate that paradigm through AI, enabling users to express intent in natural language while maintaining full governance."
Why build another enterprise data orchestration platform
Kestra emerged from practical frustration with existing tools in enterprise environments. Darras encountered Airflow limitations during a deployment at a large European retail corporation four years ago, leading to the decision to design an orchestration platform with a fundamentally different architecture.
"Our mission is simple, to help any organizations to modernize and simplify their stack by unifying all automation across data and AI and infrastructure and business operation into a single orchestration logic," Darras explained.
The approach diverges from data-centric orchestration tools by targeting broader enterprise automation needs. While Airflow historically focused on data engineering teams, Kestra's architecture addresses infrastructure automation, business process management and data workflows through a single platform.
"Airflow has historically, been a data engineering oriented orchestrator, it's very powerful and useful," Darras said. "It has shown its limits in terms of simplicity for and scale and governance for many engineers and our approach is to talk to all engineers, and not only data engineers."
Technical architecture: Code vs. declarative approaches for data orchestration
The core element of Airflow is the DAG (Directed Acyclic Graph), which is written in Python as a way to encapsulate what data needs to be orchestrated. Kestra takes a different approach, rather than using code, it uses declarative YAML instead of Python code for workflow definition.
This architectural choice draws inspiration from software engineering and DevOps practices rather than data engineering conventions. The declarative approach enables version control, automated testing and CI/CD integration without requiring programming expertise for workflow modifications.
The declarative foundation becomes strategically important as enterprises integrate AI capabilities. Traditional code-based orchestration requires programming expertise to modify workflows for new AI use cases. Declarative approaches enable faster iteration and broader team participation in AI data pipeline development, setting the stage for automated workflow generation.
Kestra 1.0 delivers AI-native data orchestration
Kestra waited years to declare version 1.0, despite running production workloads at major enterprises for multiple years. The 1.0 designation represents both technical maturity and a fundamental shift toward AI-integrated orchestration.
The 1.0 release introduces Declarative Agentic Orchestration. The AI implementation works at two levels. First, an AI copilot generates YAML workflows directly from natural language prompts, accelerating workflow creation for technical teams.
More significantly, the platform enables intent-driven automation. Users can declare the final intention, the goal in a prompt. So for example, the user can input a prompt like 'give me how many customers bought this product last month'. The AI agents in Kestra are then able to generate, optimize and execute the entire workflow behind the scenes to enable that request.
This approach addresses a critical enterprise concern about AI automation: maintaining governance and auditability. Unlike black-box AI systems, Kestra's agents generate the same YAML workflows that human developers would create, ensuring every automated decision follows existing approval processes.
How an automotive startup navigated data orchestration platform selection
Foundation Direct, an automotive analytics company processing data for thousands of car dealers, recently completed an orchestration migration that illustrates both common evaluation pitfalls and successful selection criteria.
Mike Heidner, SVP of Analytics and Business Intelligence at Foundation Direct explained to VentureBeat that his firm brings cloud computing and data analytics to automotive dealers who often lack modern data infrastructure. The company processes data from multiple sources including dealer management systems (DMS) that can be 15-20 years old.
Foundation initially managed workflows through cron (scheduled server) jobs running DBT transformations with a separate no-code extraction tool. The process was bulky and not as accurate as the company needed it to be.
Jack Perry, Lead Engineer at Foundation Direct explained to VentureBeat that he evaluated orchestration platforms with specific criteria that many enterprise teams overlook. Perry noted that he had previous experience with the Prefect data orchestration tool from a prior role but recognized it wouldn't meet Foundation's scaling requirements.
The evaluation focused on reliability, language flexibility, UI accessibility and open-source testing capability. These criteria proved more predictive of long-term success than traditional feature comparisons.
"We were able to set it up in like an hour or so with Docker compose locally, and just have it running and that was really appealing," Perry said.
Additional criteria included separate development and production environments and open-source availability for testing without licensing commitments.
"I wanted to make sure we had an open source option so we could test it without committing to it, and just getting that full blown test," Perry noted.
The team also deliberately avoided Airflow despite its market dominance.
"I've heard great things about Airflow," Perry said. "But I've also heard it has a learning curve, and that's something I wasn't ready to commit to."
The migration delivered quantifiable improvements. Success rates improved from 80% to 97% consistently. The platform's queuing features solved API rate limiting issues when multiple team members triggered workflows simultaneously.
The operational efficiency gains extended beyond engineering. Foundation implemented self-service capabilities allowing their tagging and measurement team to refresh pipelines without engineering intervention.
"Nobody even has to talk to Jack, they fill out their parameters in the application, and then they execute it," Heidner said.
Data orchestration strategic evaluation framework
For enterprises evaluating orchestration platforms, technical flexibility should take priority over feature lists.
Key evaluation criteria should include:
Language compatibility: Can the platform execute workflows in your required programming languages without architectural constraints?
Governance integration: Does the platform support your existing approval processes, access controls and audit requirements?
Operational efficiency: Can non-technical team members perform routine operations without engineering intervention?
Testing capability: Can you validate the platform with realistic workloads before making enterprise commitments?
Deployment flexibility: Are both self-hosted and managed versions available to match your operational preferences?
The orchestration platform decision has become strategic infrastructure for AI-driven enterprises. Organizations continuing to manage data workflows through multiple point solutions face increasing complexity as AI initiatives scale. The choice between consolidated platforms and tool sprawl will determine whether enterprises can deliver AI capabilities rapidly or remain constrained by integration overhead.
For enterprises building AI capabilities, the orchestration evaluation should prioritize governance, reliability and team accessibility over feature lists. The platform that enables business users to modify workflows safely while maintaining audit trails will accelerate AI adoption more than tools that require specialized expertise for routine changes.
