According to multiple estimates, a large share of data generated by businesses today is unstructured data such as text, images, or audio. This data has enormous potential for machine learning applications, but there is work to be done before it can be used directly in the form of feature extraction and dimension reduction. LLMs and vision models now consume unstructured inputs natively, yet pipelines still need chunking, labeling, eval harnesses, and monitoring tailored to messy real-world formats.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Unstructured data lacks a fixed column schema; content varies in length, encoding, and modality.
- ML pipelines extract structure via OCR, parsing, embeddings, or end-to-end neural encoders.
- LLMs handle text (and increasingly images) directly but still need retrieval, tools, and safety layers.
- Eval for unstructured inputs requires rubrics, human review, and task suites beyond tabular metrics.
- Monitor length distributions, language mix, parsing failures, and embedding drift for unstructured routes.
Examples by modality
Text: emails, tickets, chats, PDFs, HTML pages, code repos.
Images: photos, scans, UI screenshots, medical imaging (with domain compliance).
Audio: call center recordings, voice commands.
Video: often processed into frames or transcripts before modeling.
Each modality has preprocessing pitfalls (encoding, EXIF rotation, silence trimming).
From raw to model-ready
Classical path: tokenization, bag-of-words or embeddings, then classifiers or retrieval indexes.
Modern path: foundation models encode raw or lightly processed inputs; structured metadata joins by ids.
Hybrid: LLM extracts JSON fields from text into tabular columns for downstream tree models.
Document pipelines chunk text, extract tables, run OCR on scans, and attach source metadata for grounding.
Evaluation challenges
No single accuracy number; use task rubrics (groundedness, toxicity, task success).
Human labeling is expensive; active learning and LLM judges need calibration against humans.
Golden sets must reflect real file types and noise, not only clean paragraphs.
Slice by language, channel, and document source.
An evaluation harness versions prompts, corpora, and scorers together for unstructured tasks.
Platform guidance in LLM and agent evaluation platforms stores traces with raw inputs redacted per policy.
CI patterns in writing LLM evals as tests gate merges on unstructured golden tasks.
Monitoring
Input byte size, token counts, OCR confidence, parse error rate.
Embedding and retrieval metrics for RAG corpora.
Moderation and safety classifiers on sampled content.
Latency breakdown for preprocessing versus model inference.
Failure modes
Assuming UTF-8 everywhere; mojibake breaks tokenizers.
OCR errors on scans causing confident wrong answers.
Eval sets of clean text while production has HTML boilerplate and signatures.
Storing full unstructured payloads in logs without redaction or retention policy.
Retention and redaction
Define retention windows for raw attachments in eval stores separately from aggregated metrics; legal holds override default TTLs.
Modality-specific evals
Image evals need blur and compression augments; audio evals need background noise suites mirroring call center conditions.
Practitioner checklist
Before changing production settings, confirm labels, thresholds, and monitor windows match the definitions used in your last offline eval. Snapshot dashboards when incidents start so postmortems compare apples to apples.
Run slice-level reviews on high-traffic cohorts weekly even when global metrics look flat. Many failures appear first in one locale, product line, or prompt route.
Document model version, featurizer hash, and data window in every incident ticket. Future you needs that context to interpret drift charts.
When metrics disagree (offline vs online, precision vs recall), write down the business cost of each error type before picking a fix. Metrics are proxies; costs are the decision.
Share eval harness links in release notes so support and PM teams know which golden tasks must pass before they announce improvements.
FAQ
Is JSON unstructured?
JSON is structured syntactically but often semi-structured semantically; fields may contain free text needing NLP.
Do I still need feature extraction with LLMs?
Often less manual feature engineering for text, but chunking, retrieval, and metadata tags remain critical.
How does unstructured data relate to tabular data?
Many systems extract tabular features from unstructured sources or join embeddings to entity rows.
What privacy rules apply?
Redact PII in logs and eval stores; control access to raw attachments per regulation.
Can I monitor unstructured inputs without labels?
Track parsing success, length drift, language detection, retrieval scores, and proxy user feedback until labels arrive.
