Geotab, a leader in fleet telematics, has taken a bold step forward in simplifying complex fleet data management. By leveraging generative AI, Geotab introduced its cutting-edge agent, Ace, designed to help fleet managers extract actionable insights from vast and complex datasets with ease. Here’s a closer look at the challenges they faced, the innovative solutions they built, and the lessons they learned along the way.
Executive Summary
Managing fleet data is a massive challenge, with Geotab handling data from over 4 million connected vehicles, totaling 26 million trips daily. Traditional tools required hours—or even days—of effort to extract actionable insights, leaving many users frustrated. Recognizing this gap, Geotab developed Ace, a generative AI agent powered by a robust retrieval-augmented generation (RAG) architecture.
With Ace, fleet managers can now go from asking a natural language question to receiving actionable, AI-generated SQL queries and insights in seconds. The integration of Arize AI for observability and evaluation ensures the system continually improves while maintaining reliability and trust.
The Problem
Geotab’s vast telematics platform generates immense amounts of data, spanning 200 tables and billions of rows, making it challenging for users to derive insights effectively. Key challenges included:
- Complexity of Data: Lack of schema standardization made querying data overwhelming for non-expert users.
- Time-Intensive Querying: Generating insights required hours of manual effort, with many users giving up before reaching their goals.
- High Error Rates: Trial-and-error query processes often led to inaccuracies, further reducing user confidence.
- Opaque Workflows: Users struggled to trust AI-generated outputs without visibility into the reasoning behind results.
Solutions
To address the challenges of managing complex fleet data and empower users with actionable insights, Geotab implemented several innovative solutions with its Ace agent. Here’s how each solution contributed to transforming fleet management:
1. Retrieval-Augmented Generation (RAG) Architecture
Ace’s architecture is designed to retrieve domain-specific knowledge dynamically, ensuring accurate and context-aware responses:
- Multi-Source Retrieval: Ace uses multiple vector databases to pull relevant schemas, example queries, and company domain knowledge. This enables the system to understand and respond to ambiguous user queries effectively.
- Entity Matching: Resolves ambiguities like identifying whether “Bob” refers to a driver or a vehicle, ensuring precise query generation.
- SQL Validation Loop: Iteratively refines SQL queries, learning from errors to output accurate and actionable insights.
2. Automated Reasoning and Transparency
To build user trust, Ace generates reasoning reports alongside query results:
- Explanations for Outputs: Each report outlines the steps taken, assumptions made, and logic behind the generated SQL.
- Validation Steps: Checks for hallucinations or inconsistencies in reasoning, ensuring users can rely on the insights provided.
3. Iterative Evaluation with Arize AI
Geotab leveraged Arize AI’s observability tools to ensure continuous improvement in Ace’s performance:
- Offline Metrics: Evaluated predicted SQL for relevance and execution accuracy, modifying standard metrics to account for real-world nuances like extra rows or renamed columns.
- Online Metrics: Used Net Promoter Score (NPS), token counts, and cost per query to monitor user satisfaction and system efficiency in production.
- Human-in-the-Loop Feedback: Weekly reviews of customer queries and feedback informed schema adjustments and improved SQL examples.
4. Schema Simplification
To reduce errors and improve AI comprehension, Geotab restructured its database schemas:
- Flattened Tables: Minimized joins and moved redundant logic to BigQuery views, making data easier for Ace to parse.
- Clear Schema Descriptions: Ensured that column descriptions were clear and accessible, even for non-technical users.
Key Results
Geotab’s Ace agent represents a transformative leap in fleet management, combining cutting-edge generative AI with Arize AI’s observability tools to address complex data challenges. By leveraging retrieval-augmented generation (RAG), Ace dynamically retrieves schema knowledge, example queries, and domain-specific insights, streamlining the process of generating SQL queries and actionable results.
The system delivered drastic efficiency gains, reducing query times from hours to seconds and empowering fleet managers to make smarter decisions faster. This efficiency translated directly into improved user satisfaction, with positive feedback from fleet managers who now access insights effortlessly through Ace’s intuitive interface. Enhanced accuracy was another key outcome, as iterative evaluation practices—including SQL validation and transparent reasoning—helped reduce errors stemming from ambiguous data or schema complexity.
Arize AI’s observability tools played a critical role in this transformation, enabling real-time monitoring and actionable insights that ensure continuous improvement. The combination of Ace’s reasoning reports, dynamic retrieval, and robust evaluation frameworks set a new benchmark for leveraging generative AI in data-intensive industries like fleet management.
Lessons Learned
The journey to building Ace provided valuable lessons that shaped its success.
- Geotab discovered the importance of high-quality examples. While initially planning for thousands of examples in its vector database, the team found that a few hundred well-crafted examples led to significantly better results. Subtle nuances in SQL queries could either elevate or degrade the quality of outputs, underscoring the need for precision.
- To enhance AI performance, Geotab simplified database schemas, contradicting traditional normalization principles. Flattening tables and minimizing joins made the data more interpretable for Ace, reducing hallucinations and improving query accuracy. Clear and detailed schema descriptions were equally critical, ensuring even novice users could understand the data.
- SQL evaluation presented its own challenges. Users often had subjective goals, making it difficult to define a single “correct” answer. Flexible evaluation metrics that accounted for extra rows, renamed columns, or slightly altered outputs were key to aligning system behavior with user expectations. Additionally, the iterative human-in-the-loop process helped refine prompts, schema mappings, and SQL examples to improve system reliability.
- Geotab’s experience also reinforced the importance of observability from day one. Tools like Arize AI allowed the team to monitor system performance, debug issues, and optimize workflows in real time. Prompt engineering, while labor-intensive, proved inevitable and essential, as tailored prompts with clear examples significantly improved the quality of generated outputs.
- Geotab recognized that adopting generative AI raises the bar for organizational practices. Clear documentation, robust data quality standards, and streamlined workflows became critical to ensuring Ace’s success and scalability.
