There is a lot of enterprise data trapped in PDF documents. To be sure, gen AI tools have been able to ingest and analyze PDFs, but accuracy, time and cost have been less than ideal. New technology from Databricks could change that.
The company this week detailed its "ai_parse_document" technology, now integrated with Databricks' Agent Bricks platform. The technology addresses a critical bottleneck in enterprise AI adoption: Approximately 80% of enterprise knowledge remains locked in PDFs, reports and diagrams that AI systems struggle to accurately process and understand.
"It's a common assumption that parsing PDFs is a solved problem, but in reality, it isn't," Erich Elsen, principal research scientist at Databricks, told VentureBeat. "The challenge isn't just that documents are unstructured; it's that enterprise PDFs are inherently complex. They mix digital-native content with scanned pages and photos of physical documents, alongside tables, charts and irregular layouts, and most existing tools fail to capture that information accurately."
The hidden complexity behind document parsing
While optical character recognition (OCR) has existed for decades, Elsen argues that extracting usable, structured data from real-world enterprise documents remains fundamentally unsolved.
Key elements such as tables with merged cells, figure captions and spatial relationships between document elements are routinely dropped or misread by existing tools, making downstream AI applications, retrieval-augmented generation (RAG) systems or business intelligence dashboards unreliable.
The typical enterprise workaround has been to stack multiple imperfect tools together: One service for layout detection, another for OCR, a third for table extraction, as well as additional APIs for figure analysis. This approach requires months of custom data engineering and ongoing maintenance as document formats evolve.
"To compensate, teams have had to stack multiple imperfect tools or build extensive custom pipelines, spending months on data engineering instead of innovation," Elsen said. "ai_parse_document solves that by extracting complete, structured data from real-world documents — so organizations can finally trust and query unstructured data directly within Databricks."
Technical approach: End-to-end training vs. pipeline stacking
There are multiple services in the market today for parsing PDFs, including AWS Textract, Google Document AI and Azure Document Intelligence, among others. Elsen argued that instead of just reading text, the tool uses a system of modern AI components trained to end-to-end to extract structured context with state-of-the-art quality.
The function goes beyond basic extraction to capture:
-
Tables preserved exactly as they appear, including merged cells and nested structures
-
Figures and diagrams with AI-generated captions and descriptions
-
Spatial metadata and bounding boxes for precise element location
-
Optional image outputs for multimodal search applications
All results are stored directly in the Databricks Unity Catalog as Delta tables, meaning parsed documents become queryable structured data without leaving the Databricks environment. This is a key differentiator from cloud services that require exporting data for processing.
"Through data-centric training and optimized inference, we've achieved 3–5x lower cost while matching or exceeding leading systems like Textract, Document AI and Azure Document Intelligence," Elsen said.
Early enterprise adoption across manufacturing and industrial sectors
Several major enterprises have already deployed ai_parse_document in production with use cases spanning data science workflow optimization, democratization of document processing and RAG application development.
For example, Elsen noted that Rockwell Automation uses ai_parse_document to reduce configuration overhead for its data scientists.
"What once required significant setup to support complex solutions is now streamlined, letting their teams spend more time innovating and less time managing infrastructure," he said.
TE Connectivity, meanwhile, is using ai_parse_document to democratize unstructured data processing.
"Previously, extracting tables, text and metadata from documents required complex, code-heavy workflows," Elsen said. "With Databricks, they’ve condensed all of that into a single SQL function, making advanced document processing accessible to every data team, not just data scientists."
Emerson Electric is another early adopter. The company is using ai_parse_document for a RAG use case. Elsen explained that by enabling parallel document parsing directly within Delta tables, Emerson has made building RAG applications both fast and simple, all within its existing Databricks environment.
The platform integration play
While Databricks has a long history with open source, the ai_parse_document technology is a proprietary component of the Databricks platform.
Unlike standalone document intelligence APIs, ai_parse_document is deeply integrated with Databricks' Agent Bricks platform, which is a collection of AI functions and orchestration capabilities for building production AI agents.
The function works with Databricks' broader data infrastructure, including:
-
Spark Declarative Pipelines: Provide automatic incremental processing, meaning new documents arriving in SharePoint, S3 or Azure Data Lake Storage are parsed automatically without manual orchestration.
-
Unity Catalog: Governs permissions, audit trails and data lineage for parsed content exactly as it does for structured data.
-
Vector Search: Indexes parsed document elements including text, tables and figures with captions for multimodal RAG applications.
-
AI function chaining: Allows developers to pipe ai_parse_document output directly to ai_extract (entity extraction), ai_classify (document categorization) and ai_summarize (content summarization) within a single SQL query.
-
Multi-Agent Supervisor: Coordinates document-processing agents with other specialized agents for complex workflows.
"Parsing is only the beginning and rarely an end unto itself," Elsen said. "The goal is to allow customers to chain our ai_functions, like ai_extract and ai_classify, together with ai_parse_document to turn their documents into actionable data and insights. We also aim to make it seamless to turn a corpus of documents into a knowledge database for use in RAG or other information retrieval agents."
What this means for enterprise AI strategy
For enterprises building AI agent systems, it's critical to understand how PDF documents are actually used and understood by systems.
The Databricks approach sheds new light on an issue that many might have considered to be a solved problem. It challenges existing expectations with a new architecture that could benefit multiple types of workflows. However, this is a platform-specific capability that requires careful evaluation for organizations not already using Databricks.
For technical decision-makers evaluating AI agent platforms, the key takeaway is that document intelligence is shifting from a specialized external service to an integrated platform capability.
