Often used alongside model features, tags enable metadata support for slicing and cohorting. They are a convenient workaround to analyze groups of metadata that are important, but that an ML team might not want to send as an input to a model. In other words, tags avoid conflating two separate entities, features and other metadata, while enabling deep model analysis across cohorts of any kind.
Key takeaways
- Tags are metadata labels on predictions or traces used for filtering and slicing, not model inputs.
- Separating tags from features prevents leakage and keeps models from treating admin fields as signals.
- Common tags include experiment id, customer tier, prompt route, app version, and region.
- Tag cardinality and PII policies need governance like feature schemas.
- LLM and agent observability relies heavily on tags for multi-step debugging.
Tags versus features
Features enter the model during training and inference; the model learns weights from them.
Tags annotate events for analysis: which prompt template fired, which tool was available, which deployment cell served traffic.
Using tags as features without intent causes leakage (future information) or unfair use of protected attributes.
Document which fields are features, tags, or both in hybrid setups where tags become features after careful review.
Typical tag sources
Application request headers and session attributes.
Orchestration layer (agent name, planner version).
Infrastructure (Kubernetes pod, region, canary flag).
Human labels attached asynchronously for eval.
Business systems (account segment, product SKU family) joined by id after inference.
Uses in observability
Slice accuracy, latency, token usage, and eval scores by tag combinations.
Compare canary versus stable deployments on the same tag dimensions.
Debug agent failures filtered to tool_name=search and locale=de.
Feed tags into evaluation harness exports for regression diffs.
An evaluation harness should preserve tags on every row so CI comparisons remain cohort-aware.
Platforms in LLM and agent evaluation platforms index tags for trace search and dashboards.
CI eval guidance in writing LLM evals as tests encodes tag-scoped floors when product requires them.
Governance
High-cardinality tags (user id) explode storage and privacy risk; hash or roll up where possible.
Retention policies differ for tags with PII versus operational ids.
Access control: tags may expose sensitive segments restrict dashboard viewers accordingly.
Failure modes
Inconsistent tag naming across services breaks slice joins.
Missing tags default to null and hide incidents in “unknown” buckets.
Accidentally logging secrets as tag values.
Using tags as features without retrain documentation when policy changes.
Schema registry
Maintain a tag schema registry with owners, allowed values, and deprecation dates synced to OpenTelemetry semantic conventions where possible.
Cross-service joins
Standardize tenant id tag names across microservices so traces, model logs, and eval exports join without brittle rename maps.
Practitioner checklist
Before changing production settings, confirm labels, thresholds, and monitor windows match the definitions used in your last offline eval. Snapshot dashboards when incidents start so postmortems compare apples to apples.
Run slice-level reviews on high-traffic cohorts weekly even when global metrics look flat. Many failures appear first in one locale, product line, or prompt route.
Document model version, featurizer hash, and data window in every incident ticket. Future you needs that context to interpret drift charts.
When metrics disagree (offline vs online, precision vs recall), write down the business cost of each error type before picking a fix. Metrics are proxies; costs are the decision.
Share eval harness links in release notes so support and PM teams know which golden tasks must pass before they announce improvements.
Related concepts
Tags differ from model features but often define performance slices in dashboards. Align tag schemas with eval harness exports so offline and online cohort names match.
FAQ
Can a field be both tag and feature?
Rarely recommended. If promoted from tag to feature, version the model and document the change.
How many tags should I attach per event?
Enough for debugging and slicing, not hundreds of redundant keys. Standardize a core schema per service.
Do OpenTelemetry attributes map to tags?
Conceptually yes; align naming with your observability convention for cross-tool queries.
Should tags affect model predictions?
Not unless explicitly engineered as inputs with governance review.
How do tags help LLM agents?
They identify which agent, tool, prompt version, and retrieval index participated in each step of a trace.
