AI agents: what's real and what's "agent washing"

Why everyone is selling an "agent" now

A year or two ago, the word was "copilot." Now it's "agent." Your ERP vendor has one. Your phone system has one. The startup that emailed your controller last week has three. The label sells, so it gets stuck on almost everything.

Gartner, the research firm, has a name for this. In a June 2025 forecast, it called it "agent washing": rebranding existing products, such as AI assistants, robotic process automation (RPA) and chatbots, without real agent abilities. Gartner estimates only about 130 of the thousands of agentic AI vendors are real.

That doesn't mean agents are fake. It means the word alone tells you nothing. This guide covers what an agent is, what the best independent tests found, where agents do earn their keep, and how to tell the difference in a sales meeting.

What an AI agent actually is

The clearest line comes from Anthropic, the company behind Claude. In its guide Building effective agents, it separates two things. Workflows are systems where AI and tools follow steps written ahead of time in code. Agents are "systems where LLMs dynamically direct their own processes and tool usage." In plain terms: a workflow follows your recipe. An agent picks its own next step.

Here's how that sorts the products you'll be pitched:

Gartner draws a similar line. It says AI assistants simplify tasks but depend on human input and don't act on their own, and that calling them agents is the most common form of agentwashing.

One honest note: a workflow isn't worse than an agent. For most office work, it's better. Anthropic's own advice is to find the simplest solution that works and add complexity only when needed. It even says that might mean not building agents at all.

What the tests found

Two research teams put agents through realistic business work in 2025. Both are worth knowing, with one caution: they tested models from early-to-mid 2025. Models have improved since. The pattern in the results still holds up as a guide to where agents struggle.

Carnegie Mellon's fake company. Researchers built TheAgentCompany, a simulated small software company with its own internal sites, files, chat and simulated coworkers, and gave agents 175 real-world tasks. In the latest version of the paper (Sept. 2025), the best agent, running on Google's Gemini 2.5 Pro, finished 30.3% of tasks on its own. With partial credit for half-finished work, it scored 39.3%. Claude 3.7 Sonnet came in at 26.3%, and GPT-4o at 8.6%.

The detail that matters for a back office is which tasks went worst:

Share of tasks the best agent finished, by department

Gemini 2.5 Pro in Carnegie Mellon's TheAgentCompany test, 175 tasks. Tested on 2025 models.

1. Project management28 tasks39.3%
2. Software engineering69 tasks37.7%
3. HR29 tasks34.5%
4. Data science14 tasks14.3%
5. Admin15 tasks13.3%
6. Finance12 tasks8.3%

Source: Xu et al., TheAgentCompany, arXiv 2412.14161 v3, Sept. 10, 2025, Table 5.

Finance and admin were the weakest. The researchers say some of those tasks meant building spreadsheets, collecting information from several people, or reading images that employees had scanned. That is a fair description of an AP desk. The failures were often mundane. One agent got stuck because it couldn't close a welcome popup. Another, unable to find the right coworker, renamed a different user to match the name it was looking for. That's a shortcut no employee would get away with.

One of the paper's authors later told The Register that a newer agent reached 34%. That figure isn't in the paper, but it shows the direction: better, not solved.

Salesforce's CRM test. Salesforce's own researchers built CRMArena-Pro, 19 tasks across sales, service and quoting. Leading agents succeeded about 58% of the time when a task took one step, and about 35% when it took several back-and-forth turns. But on plain workflow execution, top agents succeeded more than 83% of the time. The researchers also found agents had "near-zero inherent confidentiality awareness." Prompting them to be careful helped, but often hurt their task performance.

Agents do well on a narrow job with clear steps. They stumble when the job is long, vague, or spread across people.

Where the other side has a point

It would be easy to read those numbers and write agents off. That would be a mistake.

• Narrow work goes well. The same Salesforce paper that found 35% on multi-turn tasks found over 83% on workflow execution. Gartner, for all its warnings, predicts that 40% of enterprise apps will include task-specific agents by the end of 2026, up from less than 5% in 2025.
• The back office is where the payoff shows up. MIT's NANDA project, in a 2025 report best known for finding 95% of organizations got zero return from generative AI, also found that back-office automation often yields better returns than the sales and marketing projects that get half the budgets. (The report calls its findings preliminary, and its outcomes are self-reported.)
• Some vendors have real numbers. Salesforce says its own agent on its help site has handled more than five million conversations and resolves more than 63% of them instantly. That's a vendor claim about its own site, and CX Today notes Salesforce's figures are self-reported with no outside audit. But it is a real deployment at scale.

The fair reading: agents are real, but narrower than the marketing. Gartner's own analyst put it this way in the same release: many use cases sold as agentic today don't need an agent at all.

What the cancellations will be about

Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, due to "escalating costs, unclear business value or inadequate risk controls." In a May 2026 release, it added that by 2027, 40% of enterprises will demote or shut down autonomous agents because of governance gaps found only after something went wrong in production.

That second release offers a useful ladder for how much freedom an agent gets: observe (read-only), advise (a person does the work), act with approval, and act on its own. For a mid-size office, most jobs belong on the first three rungs. Paying a supplier, changing a vendor's bank details or emailing a customer belong with a person.

Companies your size are also moving more slowly than big ones, and that's fine. In McKinsey's 2026 State of AI survey, 40% of respondents at large organizations said they were scaling AI agents, while the share at smaller organizations stayed flat at 22%.

"AI washing" can be a legal problem, too

Regulators have already acted on overstated AI claims. In March 2024, the SEC settled charges against two investment advisers, Delphia and Global Predictions, for false statements about their use of AI. They paid $400,000 in total penalties. In September 2024, the FTC announced five actions under "Operation AI Comply," including DoNotPay, which had marketed "the world's first robot lawyer" and agreed to pay $193,000. Those cases were about selling to investors and consumers. They don't make a vendor's pitch to you illegal. But they show that "it's AI" is a claim someone can be asked to back up.

Six questions to ask before you buy an "agent"

1. What does it decide on its own, and what is a fixed step? If the answer is "nothing, it follows the steps," it's a workflow. That may be exactly what you need. Just don't pay agent prices for it.
2. Will you run it on 20 of our real documents, including the messy ones? Not the demo set. Your scanned packing slips, your customer who sends POs as photos, your supplier whose invoices change every quarter.
3. What does it do when it isn't sure? A good answer is "it stops and asks a person, and shows why." A bad answer is "it's 99% accurate."
4. What can it touch? Ask for the exact permissions. Read-only to start. Its own login, never a shared one.
5. What gets logged? Every action, with the document it came from, so your team can check its work.
6. What does it cost when volume doubles? Agents that plan and retry use more computing than a single answer. Gartner lists cost as the first reason projects get canceled.

The best agents in 2026 are the boring ones: a narrow job, clear limits, a person approving the steps that matter. If a vendor can show you that working on your own documents, the label matters less than the result. If they can't, the label is all you're buying.