Skip to main content

You Audit Everything Downstream of the Cloud Bill. You Don't Review How You Were Billed.

Tyler

Tyler

Co-Founder & CEO

|
A billing statement showing $4,317,662.08 billed against $4,217,662.08 actually used, under the line Spot the difference. Your anomaly detector didn't.

At 7:46 PM Pacific on July 16, a configuration change introduced a unit pricing error into an AWS billing pipeline. Alarms detected the anomaly immediately. No engineer was paged.

For the next several hours, thousands of AWS accounts watched their month-to-date costs detach from reality. One account with typical usage under five dollars a month showed $1.7 billion. Another dashboard displayed $7.1 trillion. A customer posted a screenshot of $225,579,210,164.83. AWS did not learn about it from its own monitoring; it learned about it at 12:19 AM from customer escalations, roughly four and a half hours after the alarms had already fired and been ignored. Estimated bill generation was not paused until 8:24 AM. In AWS's own words, "Our alarms detected cost anomalies but failed to halt the estimated bill generation process or alert our engineering teams." As a precaution, AWS then disabled budget and cost anomaly alerts platform-wide.

The July 16 Timeline

  • 7:46 PM PDT: A configuration change introduces a unit pricing error. Alarms fire. Nobody is paged.
  • 12:19 AM PDT: AWS learns about it from customer escalations, roughly four and a half hours later.
  • 8:24 AM PDT: Estimated bill generation is finally paused.
  • Result: Estimates in the billions and trillions stayed live for more than 24 hours. No invoice was affected.

Nobody was actually invoiced. The estimates were wrong, the invoices were fine, AWS backfilled corrected data, and the story lasted about a news cycle. Most FinOps teams filed it under "funny screenshot" and moved on.

That is the wrong file.

The reason the number was absurd is the only reason anyone noticed. A unit error that turns 5 cents per gigabyte into 5 cents per byte produces a figure so ridiculous it screams. Move the decimal one place instead of nine and you get a number that looks completely normal, sits inside your historical variance, passes every anomaly detector you own, and gets paid. Twelve times a year.

The July incident did not reveal that AWS billing is broken. It revealed something more useful: the billing pipeline is a piece of software, software has defects, and the defect that made headlines was caught by its own magnitude rather than by any control designed to catch it. That is worth thinking about carefully, because the entire discipline of FinOps sits on top of the output of that pipeline and treats it as ground truth.

The Bill Is a Computation, Not a Measurement

There is a category error buried in how most organizations think about cloud spend. We talk about the bill as though it were a reading, like a water meter. It is not. It is the output of a long, opaque, multi-stage calculation, and almost none of that calculation is visible to the party paying for it.

Seven steps from usage to invoice, from the provider meter through the metering pipeline, rate lookup, discount engine, credits and tax, and channel re-rating, with only the final invoice fully verifiable by the customer

Walk the chain. A resource runs and emits usage into the provider's meter, which you cannot see. Raw events get aggregated into usage records by a pipeline you cannot inspect. Those records get joined to a SKU and a unit price. Discounts apply: Reserved Instances first, then EC2 Instance Savings Plans, then Compute Savings Plans, applied hourly to your highest-savings-percentage usage first, according to an algorithm AWS runs on your behalf and that you cannot redirect. Credits, taxes, and adjustments land at the billing-account level. If you buy through a channel partner, the whole thing gets rebuilt one more time before you see it. Then an invoice arrives.

Seven steps. You can independently verify roughly one of them, and it is the last one, which is the output rather than the computation.

This is not a new observation, and that is the uncomfortable part. Researchers at the University of Wisconsin ran controlled measurement studies against EC2, Google Compute Engine, and Rackspace and published the results at USENIX HotCloud in 2014.

What Independent Measurement Found in 2014

  • EC2 undercharging Internet outbound traffic by an average of 5.6%
  • A Rackspace storage bug where creating and deleting a 100GB drive produced charges 36.3% higher than expected
  • EC2 I/O operations where the measured count exceeded the charged count by as much as a factor of 4.6
  • The authors' conclusion: "Billing events mostly occur within the cloud infrastructure, making them largely unobservable to customers."

Their call was structural, not accusatory: a transparent billing model with real-time usage access, so customers could audit their own bills. Twelve years later, that API still does not exist in any meaningful form, and the industry has spent the intervening decade building sophisticated tooling on top of the unaudited output instead.

The providers are, to their credit, fairly honest about this in their documentation. Microsoft states plainly that "All included costs are estimated until an invoice is generated", that charges can continue to accrue and change until the fifth day after the billing period ends, and that "The price shown on your invoice might differ from the price used for cost estimation." Every dashboard your team stares at during the month is explicitly labeled as provisional by the vendor that built it.

There is one more asymmetry worth naming, because it shapes everything else. Standard cloud agreements give the provider a right to verify your compliance on notice. None of them give you a reciprocal right to verify the provider's metering. The provider can audit you. You cannot audit the provider.

Every One of These Was Found by Someone Other Than the Billing System

The AWS incident was not an outlier. It was the most visible member of a category.

Four billing incidents and how long the wrong numbers stayed visible, from 40 hours for the AWS Cost Explorer error to 13 days for Google Cloud BigQuery reservation discrepancies, with two found by customers and two found by the provider

In July 2022, a Google Compute Engine rollout created mismatched usage data that cascaded into duplicated charges. Some accounts saw up to double their actual instance usage reflected in their billing data. Google engineers discovered the delayed reporting roughly 18 hours after it started. Full recovery took nine days and nine hours.

In July 2023, customers using the BigQuery Reservation API in the US and EU multi-regions saw billing discrepancies on usage going back to July 6. The incident was not opened until July 18. Twelve days of misbilled usage data, live, before anyone opened a ticket about it.

Nine days before the AWS incident, a developer on Anthropic's free tier received an invoice for $1,669,875.90. Within 24 hours it had escalated to $16.6 million. The account had never incurred a charge and had no payment method on file. The emails came from a genuine Anthropic domain with working Stripe payment links. Two charge attempts were made and declined by his bank. It took roughly 18 support emails across four days to get written confirmation the invoices were void.

Look at the detection path in every one of these. Two were found by the provider, days into the incident. Two were found by customers. In none of them did an automated control on the customer side raise its hand, because customers do not have automated controls for this.

That last point generalizes further than billing pipeline bugs. In two separate incidents documented by InfoQ in July, autonomous agents with cloud credentials burned through thousands of dollars: a three-person agency with a typical $10 to $15 monthly AWS bill received a $14,000 charge in a single day after attackers used static access keys against Bedrock, and an agent given full AWS access spun up five m8g.12xlarge instances to port-scan a hobbyist network, reaching $6,531.30 before anyone noticed. Both were caught by credit card charges, not by AWS monitoring. The structural reason is simple: cloud billing data lags actual spend by up to 24 hours, and budget actions evaluate against that delayed data. Every budget-based control fires at invoice time. The money is already gone.

Anomaly Detection Is Not Billing Accuracy

Here is the part that should bother a mature FinOps practice most, because it is not a gap in vendor capability. It is a gap in the definition of the problem.

Read the FinOps Foundation's own framework. Anomaly Management is defined as "the ability to detect, identify, clarify, alert on, and manage unexpected cost events in a timely manner." And what is an anomaly? "In the context of FinOps, anomalies are levels of spending that are different (usually higher) than normal historical or expected spend."

Normal historical spend. That is the reference point. The capability page does not mention provider billing errors or invoice accuracy verification anywhere.

The tooling implements the definition faithfully. AWS Cost Anomaly Detection trains machine learning models on your historical spending patterns and computes cost impact as actual spend minus expected spend, where expected spend is derived from your own history. Severity is scored by "how abnormal a certain anomaly is accounting for historical spending patterns."

Follow that logic to its conclusion. If a rate has been misapplied to a service since the day you turned it on, that misapplied rate is your history. It is your baseline. It is the thing the model was trained to consider normal. A systematically wrong price is, by construction, invisible to a system that defines correctness as consistency with the past. So is a Savings Plan that stopped applying to the workload it was purchased for. So is a negotiated discount that quietly reverted to list on one SKU. So is a credit that was earned and never passed through.

Anomaly detection answers "did my spending change?" Billing accuracy answers "is this charge correct?" Those are different questions, and the second one has almost no tooling behind it.

The framework does gesture at the right activity. Its Invoicing and Chargeback capability lists "Periodic reconciliation of invoices to ensure rate, discounts, and credits are validated" as a measure of success, and describes the Walk maturity level as one where "Invoices are manually validated ensuring credits, discount rate, and one-off charges are all correct."

Manually validated. Against a rate card where the EC2 price list file alone runs to roughly 370 MB and over 10 million lines, before you touch the other hundred-plus services. Across three providers whose billing files, by the FinOps Foundation's own account of why FOCUS had to exist, each use "unique terminology, taxonomy, and metrics."

Nobody is doing that manually. What actually happens is that someone in Finance checks the invoice total against the forecast, confirms it is roughly what everyone expected, and codes it. That is not validation. That is confirmation that the number did not surprise anyone, which is precisely the check that a stable, systematic error is guaranteed to pass.

The Channel Layer Is a Second Pipeline

If you buy cloud through a reseller, distributor, or MSP, there is one more transformation between the meter and your invoice, and it deserves to be described accurately rather than darkly.

AWS ships a first-party service for it. AWS Billing Conductor is documented as "a custom billing service for AWS Channel Partners and organizations that have chargeback requirements," and its function is stated without euphemism: it "enables users to create a second version of their costs to share with their customers or account owners." Partners define pricing rules that "set global or specific markups or discounts," add custom line items as "one-time or recurring flat or percentage-based charges or credits," and produce pro forma cost and usage data that lives in a separate domain from the actual billable data. AWS notes that customers "will observe cost differences between billable costs (matching the AWS invoice) and pro forma costs (matching the Billing Conductor configuration)."

On the Microsoft side, the mechanics differ but the visibility outcome is similar. Under a CSP arrangement, Microsoft's documentation states that the cost visibility policy "must get enabled by the provider for the customer tenant," that customers then view consumption at pay-as-you-go rates, and, explicitly, that "Costs shown don't include any discounts or credits that the partner may have."

None of this is illegitimate. Resellers add real value, take real risk, and are entitled to a margin. Chargeback tooling exists because large enterprises genuinely need to allocate costs to internal P&Ls. The point is narrower and more important than an accusation: if you buy through a channel, the cost data you analyze is a constructed artifact by design, the margin structure inside it is invisible to you by default, and your ability to detect an error in it is lower than it would be on a direct account, not higher. You are auditing a rendering, not a record.

For any enterprise with a partner-intermediated agreement, the right question at the next QBR is not "are your rates competitive?" It is "can I see the underlying provider billing data alongside what you invoice me, for the same period, at line-item granularity?" The answer to that question tells you a great deal.

What This Is Actually Worth

Sizing this honestly requires admitting what is not known. There is no public, rigorous study of billing error rates across AWS, Azure, and GCP at enterprise scale. Anyone who quotes you a precise number is selling something.

What does exist are reference points from adjacent, better-instrumented domains.

In June 2026, a startup called Vaudit published results from instrumenting customer environments with an SDK to capture raw usage and compare it against invoices. Across 60 companies and $34 million in AI provider invoices reviewed between March and June, they identified roughly $1.7 million in overcharges, an error rate near 5%. Approximately 80% of the disputed amounts were credited back after challenge. The alleged failure modes are instructive: customers billed for newer, more expensive models than the ones they actually invoked, charges for incomplete requests and error responses, and failed agents generating retry storms. Anthropic and OpenAI both dispute the findings, and that disagreement is worth taking seriously rather than waving away. But note what made the audit possible at all: independent measurement on the customer side. That is the entire methodology.

Vega Cloud, writing about the same category, describes a customer whose misconfigured services drove an extra 6.7% every month, compounding into hundreds of thousands of dollars over a year.

And in telecom, a discipline roughly three decades older than cloud with a mature third-party audit industry built around it, commonly cited invoice error rates attributed to Gartner still land in the range of 7 to 12%. Telecom has had specialized expense management vendors, contract reconciliation tooling, and dedicated recovery auditors since the 1990s. It still runs at 7 to 12%. Cloud billing is younger, changes faster, has vastly more SKUs, and has essentially none of that verification infrastructure.

So rather than assert a rate, here is the arithmetic, and you can substitute your own assumption.

Scenario table of annual overpayment by cloud spend and error rate, ranging from $125,000 at $25 million of spend and a 0.5% error rate to $25 million at $500 million of spend and a 5% error rate

At $100 million in annual cloud spend, a 1% error rate is a million dollars a year. Half a percent, an error rate low enough that no dashboard would ever surface it and no CFO would ever question it, is half a million. At $500 million, one percent is five million dollars annually, recurring, invisible, and fully paid.

Now put that next to where the attention goes. Flexera's 2026 State of the Cloud Report puts wasted cloud spend at 29%, the highest in five years, and 85% of organizations name managing cloud spend as a top challenge. An enormous tooling industry exists to attack that 29%, and it should. Nobody is arguing that a possible 1% is bigger than a measured 29%.

The argument is about controls, not magnitude. The 29% has platforms, dashboards, dedicated headcount, executive reporting, and a foundation-published framework. The 1% has a person in Finance checking whether the invoice total looks about right. One of those numbers is instrumented and one is not, and the uninstrumented one is the one where you have no recourse at all if you never find it.

The Second-Order Problem: This Is a Financial Controls Gap

Step back from the dollars for a moment, because there is a governance argument here that lands harder with a CFO than any of the above.

Cloud is now, for many enterprises, one of the largest single vendor relationships in the business. Flexera reports that 76% of large enterprises spend more than $5 million per month on cloud. The FinOps Foundation's 2026 survey, covering 1,192 practitioners and more than $83 billion in annual spend, shows FinOps teams increasingly reporting to the CTO or CIO and expanding into SaaS, licensing, private cloud, and data center.

Now ask how that spend is controlled. In most organizations, the answer is: the provider computes the amount, sends an invoice, and the enterprise pays it. There is no independent measurement, no reconciliation to a customer-side source of truth, and no systematic dispute process. For a payables relationship of that size in any other category, that would not survive an audit committee conversation.

It is the same shape of problem as the one I wrote about last month, where reinsurers and regulators have already put a price on cloud dependency and enterprises still accept a service credit worth a percentage of one month's bill. In both cases the provider computes the number, and the customer accepts it without a measurement of their own.

The AI numbers make the exposure worse and more urgent. The FinOps Foundation's survey found 98% of practices now manage AI spend, up from 31% two years ago, and the community's top tooling request is granular AI spend monitoring. A practice does not ask for granular monitoring of something it can already see clearly. AI billing arrived faster than the instrumentation for it, on metering more opaque than compute: tokens counted inside a model server you cannot observe, priced per model version, with retries and error responses that may or may not be billable depending on the provider. That is the same asymmetry I described in July, now pointed at the invoice instead of the SLA.

Forecasting is not the problem. It is a measurement problem, and measurement problems do not get solved by better dashboards on top of the same unverified input.

What You Should Actually Be Doing Right Now

1. Instrument the customer side. Every credible finding in this space came from the same method: measure independently, then compare to the invoice. CloudTrail, Azure Activity Log, GCP audit logs, and application-level telemetry all record what you actually asked for. That is your source of truth. The billing file is the counterparty's claim about it. Right now most organizations have only the claim.

2. Separate rate validation from anomaly detection. These are different controls. Anomaly detection asks whether spend changed. Rate validation asks whether the price applied to a unit of usage matches the price you negotiated, and whether your commitments, discounts, and credits landed the way the contract says they should. Your existing platform almost certainly does the first and almost certainly does not do the second. Confirm that with your vendor directly rather than assuming.

3. Reconcile commitment and discount application every month, not at renewal. Savings Plan and RI application is provider-controlled and runs hourly. Enterprise Discount Program and private pricing terms are contractual. The gap between what your agreement entitles you to and what the discount engine actually applied is exactly the kind of error that stays stable, stays small, and never triggers an alert.

4. If you buy through a channel partner, ask for the underlying provider data. Not the partner's report. The provider's billing data for the same period at line-item granularity, alongside the partner invoice. If that request is difficult to fulfill, you have learned something important about your visibility, and you should decide whether you are comfortable with it.

5. Establish a dispute process before you need one. Find your dispute windows in each provider agreement and each partner contract. Know who owns the claim, what documentation is required, and how long you have. Vaudit reported roughly 80% of disputed overcharges being credited back once challenged, which is the useful half of the story: this money is recoverable, but only if someone finds it and files.

Things To Think About

The July AWS incident was not a scandal. Nobody was actually overcharged, AWS corrected it, and the invoices were right. That is genuinely the correct read of what happened.

It is also the reason it is worth paying attention to. The pipeline that produced $7.1 trillion in phantom charges is the same pipeline that produces your real invoice every month. It failed. Its own alarms fired and paged nobody. Customers found it first. And the only reason anyone noticed is that the error was too large to hide.

Everything in modern FinOps is built on the assumption that the number on the bill is correct. We optimize against it, allocate from it, forecast on it, and report it upward. It is the one input in the entire practice that receives no independent verification, from a counterparty that has every incentive to compute in its own favor when the calculation is ambiguous, and no obligation to show its work.

You do not need to believe your provider is acting in bad faith to want this control. You need only believe that billing systems are software, that software has defects, that a small stable error is worth real money at your scale, and that nobody currently checks. All four of those are true.

Trust your provider. Verify the bill.

That is the problem Next Signal exists to solve. We monitor cloud provider SLA compliance and billing accuracy across AWS, Azure, and GCP, and automate the detection and recovery process for enterprise customers. The bill is a claim. Treat it like one.

Wondering what an unverified cloud bill might be costing you? The ROI calculator at nextsignal.io gives you a rough estimate in under a minute.

Frequently Asked Questions

What happened in the July 2026 AWS billing incident?

At 7:46 PM Pacific on July 16, a configuration change introduced a unit pricing error into the AWS bill computation system, which relies on unit conversion data to calculate line item charges. Estimated month-to-date costs in the billions and trillions were displayed to thousands of accounts for more than 24 hours. AWS's alarms detected the anomaly but did not halt bill generation or page engineering. Actual invoices were never affected.

Why would cloud cost anomaly detection miss a billing error?

Because it is trained on your own spending history. AWS Cost Anomaly Detection computes cost impact as actual spend minus expected spend, where expected spend is derived from your past. A rate that has been misapplied since the day you turned a service on becomes part of that baseline, so it reads as normal. Anomaly detection answers whether spend changed; it does not answer whether a charge is correct.

Can you audit your cloud provider's billing?

Not directly. Standard cloud agreements give the provider a right to verify your compliance, with no reciprocal right for you to verify their metering, and there is still no real-time usage API that would let you reconstruct the bill. What you can do is instrument the customer side, using CloudTrail, Azure Activity Log, GCP audit logs, and application telemetry as an independent record, then compare that to the invoice.

Does buying cloud through a reseller reduce billing visibility?

It adds a transformation. AWS Billing Conductor is documented as letting partners "create a second version of their costs to share with their customers," with configurable markups, custom line items, and pro forma data held separately from actual billable data. Microsoft's CSP documentation notes that costs shown to customers "don't include any discounts or credits that the partner may have." The data you analyze is a constructed artifact, so ask for the underlying provider billing data at line-item granularity for the same period.

How much does a small cloud billing error actually cost?

It scales directly with spend. At $100 million in annual cloud spend, a 1% error rate is $1 million a year; at $500 million, it is $5 million a year, recurring. For reference points, an independent audit of $34 million in AI provider invoices reported an error rate near 5%, and telecom, with three decades of audit infrastructure behind it, still runs on commonly cited invoice error rates of 7 to 12%.

Sources

Industry data and reporting cited in this article: