Attributing LLM Cost per Feature in a Production Product
Understand how to track and allocate LLM costs across features in a production system without overestimating or underestimating usage.

You're trying to figure out how much each feature in your product is costing you in terms of LLM usage, but your current tracking method either aggregates all calls into a single metric or splits them in a way that makes it impossible to accurately allocate costs to specific features. This leads to either underestimating the cost of certain features or overestimating others, which distorts your understanding of where to optimize.
Instrumenting Feature Usage
To attribute LLM cost per feature in a production product, the first step is to instrument each feature's usage at the request level. This means adding feature-specific metadata to every request that interacts with the LLM. The metadata should include a feature ID, user ID, and a timestamp. These fields are essential for later cost allocation and analysis. The feature ID allows you to map requests to specific features, while the user ID helps in understanding usage patterns across different user segments.
To implement this, you can use middleware or API gateways to log and categorize requests before they reach the LLM. Middleware is often the preferred choice in a full-stack environment because it allows you to intercept and modify requests at an early stage. For instance, in a Next.js application using Next.js API routes, you can add a middleware layer that wraps each request.
Here’s an example of how you might structure the middleware in a Next.js app:
import { NextApiRequest, NextApiResponse } from 'next';
export default function middleware(req: NextApiRequest, res: NextApiResponse) {
const featureId = req.headers['x-feature-id'] as string;
const userId = req.headers['x-user-id'] as string;
const timestamp = new Date().toISOString();
req.headers['x-llm-feature-id'] = featureId;
req.headers['x-llm-user-id'] = userId;
req.headers['x-llm-timestamp'] = timestamp;
return true;
}This middleware adds feature-specific headers to the request, which can be used later in the LLM service to categorize usage. The headers are then available for logging, monitoring, or further processing. The key is ensuring that these headers are consistently applied across all feature endpoints.
One trade-off is the overhead of adding middleware, which can introduce latency. However, in most cases, this is negligible compared to the value of accurate cost tracking. The headers should be validated and sanitized to prevent injection attacks or malformed data. For more details on middleware configuration, refer to the Next.js middleware documentation.
Cost Allocation by Feature
Once you've instrumented feature usage, the next step is to allocate the cost of that usage to the right feature. This is where tagging systems and metrics stores come into play. For each feature, you need to track both input and output tokens. These are the two primary contributors to cost in most LLM services. You can use Prometheus or Datadog to collect this data, but the key is ensuring that each request is tagged with the feature identifier.
The tagging system should map these identifiers to cost centers. For instance, if you have a feature called user_profile, you might tag it with feature:user_profile and then associate that tag with a cost center like engineering-ai. This mapping is typically done in your metrics configuration or via a central tagging service. The exact format depends on your metrics store, but the principle is consistent: every request must carry a tag that identifies which feature it belongs to.
You can then use tools like AWS Cost Explorer or Azure Cost Management to group costs by these tags. These tools allow you to filter and aggregate costs at the feature level, which is essential for budgeting and optimization. However, the effectiveness of these tools depends on the granularity and consistency of your tagging. If you're using AWS, for example, you can configure cost allocation tags in the console and then use the API to query cost data by tag.
It's worth noting that the process of tagging and aggregating can introduce some latency, especially if you're dealing with high-throughput systems. You should also consider the trade-off between the accuracy of your cost tracking and the overhead it imposes on your system. For example, if you're using Guardrails: Validating Input Before It Reaches the Model, you might already have a system in place to identify feature-specific requests, which can be leveraged for tagging.
Ultimately, the goal is to make sure that every token used by a feature is accounted for and attributed to the correct cost center. This requires careful configuration and ongoing validation to ensure the data you're collecting is both accurate and actionable.
Trade-Offs in Cost Tracking
Adding metadata to track llm usage per feature increases latency and memory usage. Every additional field added to a request or response adds overhead, and in a high-throughput system, this can compound. For example, adding a feature_id to every GraphQL request increases the payload size and may require additional processing on both the client and server. A typical trade-off is to balance the granularity of tracking with acceptable performance. If you're tracking 100 features, you might consider grouping low-usage features under a parent category to reduce overhead.
Over-tagging is a common pitfall in feature cost attribution. If you tag every minor interaction, you end up with a lot of noise. For instance, tagging a simple button click as a feature can obscure the real cost drivers. It's better to focus on high-impact features that directly influence cost. A good rule of thumb is to tag only features that are part of a core user flow or have a measurable impact on usage patterns.
For features with low usage, consider sampling or filtering. If a feature is used less than 1% of the time, tracking it at full fidelity may not be worth the overhead. Instead, you can sample requests or filter them out entirely. This approach reduces the volume of data collected and the associated costs. A simple configuration in your logging or metrics system can handle this. For example, setting a sample_rate of 0.01 in your feature tagging middleware ensures that only 1% of requests are tagged, which is often sufficient for analysis.
| Trade-Off | Impact | Mitigation |
|---|---|---|
| Metadata overhead | Increased latency, memory usage | Balance granularity with performance |
| Over-tagging | Data noise, analysis complexity | Focus on high-impact features |
| Low-usage features | Unnecessary overhead | Sample or filter requests |
For more on configuring feature tagging in GraphQL, see the Apollo Server documentation.
Common Gotchas
One of the most common gotchas in llm cost tracking is assuming that every request is feature-specific. In reality, a significant portion of traffic might be internal, such as health checks, background jobs, or test traffic. These requests are often not tied to any user-facing feature and should be excluded from cost allocation. Failing to do so can skew your feature cost attribution and make it difficult to justify budgeting decisions. You’ll need to instrument your application to distinguish between internal and external traffic, which usually involves adding a request header or query parameter that identifies the source.
Shared models introduce another layer of complexity. If multiple features use the same model, you can’t simply attribute cost by request count or token usage. You need a way to differentiate usage across features, which might involve adding feature-specific metadata to each request. For example, you could include a feature_id in the request headers or payload. This requires coordination with your API gateway or service mesh to ensure the metadata is consistently applied and logged.
Finally, avoid over-reliance on token counters. While they’re useful for estimation, they’re not a substitute for actual cost metrics. Cloud providers like AWS or Azure expose precise cost metrics through their APIs, which should be used instead. For example, you can query the AWS Cost Explorer API to get detailed usage by resource and tag. This approach is more accurate and aligns with the token counting is wrong: measure actual LLM cost principle.
Tools and Infrastructure
Cloud provider tools like AWS X-Ray or Azure Monitor are essential for tracing and cost attribution. They offer built-in metrics for compute, storage, and API usage, which can be leveraged to build a feature-level cost model. For example, AWS CloudWatch can be configured to track the number of tokens processed per feature, while X-Ray can trace requests through the application stack to associate them with feature flags. These tools require careful configuration, often involving custom metrics and dimensions that map features to specific services or endpoints.
Building a custom dashboard using Grafana or Kibana allows for real-time visualization of feature-level cost trends. Both tools support ingestion of metrics from cloud providers and can be configured with custom dashboards that filter data by feature. For instance, in Grafana, you'd set up a data source for Prometheus or CloudWatch and create panels that aggregate cost by feature, time range, and environment. This setup involves defining query rules and ensuring data is indexed properly for fast retrieval.
Integrating cost tracking into the CI/CD pipeline ensures that cost tracking is enforced in feature deployments. For example, a GitHub Actions workflow can be configured to check if a feature flag is present and if the corresponding cost tracking middleware is enabled before merging code into production. This requires a script that validates feature flags against a cost tracking configuration file, which is stored in a shared repository.
// Example GitHub Actions workflow snippet
const validateCostTracking = (featureFlags: string[]): boolean => {
const requiredFlags = ['cost-tracking-enabled'];
return requiredFlags.every(flag => featureFlags.includes(flag));
};This script checks if the cost-tracking-enabled flag is present in the feature list. If not, the pipeline fails, ensuring cost tracking is part of the deployment process. This approach introduces a small overhead in CI/CD but ensures that cost tracking is consistently applied across all features.
Key takeaways
- Tag every feature-specific request with a unique feature ID and user ID in your API gateway or middleware to enable granular cost attribution.
- Use Prometheus or Datadog to track input/output tokens per feature and map tags to cost centers using a centralized tagging system.
- Avoid over-tagging by focusing on high-impact features and apply sampling or filtering for low-usage features to reduce overhead.
- Use AWS X-Ray or Azure Monitor for tracing and integrate with Grafana or Kibana to build a custom dashboard for real-time feature cost analysis.
- Always validate that your feature tags are applied to actual user traffic and not internal or test requests, and be cautious with shared models.
Frequently asked questions
- How do I track LLM costs per feature without overcomplicating the system?
- Instrument requests with feature-specific metadata and aggregate costs using a tagging system. Avoid over-tagging and focus on high-impact features.
- Can I use cloud provider cost tools for feature-level attribution?
- Yes, use tools like AWS Cost Explorer or Azure Cost Management to group costs by feature tags. This works best with consistent tagging practices.
- What if a feature uses the same model as another?
- Use feature tags to distinguish usage, even if the model is shared. Track input/output tokens per tag to ensure accurate cost attribution.


