VSP Demo?

Back to Blog

If Your Copilot Pilot Can't Fail, It Can't Tell You Anything

By Asif Rehmani
Updated November 16, 2026
If Your Copilot Pilot Cant Fail It Cant Tell You Anything
VisualSP
Blog
If Your Copilot Pilot Can't Fail, It Can't Tell You Anything

A small group got Copilot licenses months ago. A few were on the technology committee, a couple needed it for something specific, a few asked loudly enough. The instructions were roughly "try it and tell us what you think."

Now the renewal date is on the calendar, or someone in finance wants to know whether to buy more. And the honest answer to "so what did the pilot tell us?" is that people seem to like it.

That's the question I hear most from IT directors heading into a renewal or an expansion decision right now. We ran a pilot. Why don't we know anything?

Because the pilot was never set up to fail. If there's no result that would make you say "this didn't work," there's also no result that proves it did. A group of people trying a tool is a test drive. A pilot is an experiment with a question in it, and the answer is allowed to be no.

Why "give it to a few people and see" produces nothing

The story I heard over and over at TechCon365 Dallas earlier this month went like this. Buy Copilot licenses. Hand them to whoever asks. Give people an hour of training, if that. Call it a rollout. Then stare at the usage numbers.

It's common. When I polled 116 IT leaders in Seattle, 105 had Copilot licenses and 82 had no usage guidelines or training in place. Licenses go to whoever is curious or senior, nobody picks a job for Copilot to do, and the only data at the end is the usage report.

The usage report can't carry the decision. It tells you who opened Copilot and how often. It doesn't tell you whether anything about the work changed. Chad Doty, who led the Copilot rollout at Ecolab, put it plainly when we talked: "Our adoption numbers looked great, but it didn't really tell me anything." He also said "rolling it out is easy." The hard part is having something specific to point at when someone asks why you're keeping it.

There's a second problem. Curious volunteers are the wrong sample. They'll find a use for anything. When you expand to everyone else, the people who never raised their hand behave nothing like the pilot group, and the numbers you were counting on don't show up.

What a pilot that can fail looks like

Four decisions, made before anyone gets a license.

  1. One painful workflow. A named task that happens often and that people already complain about. "Productivity" doesn't count. The best example I've seen lately came from a finance team I talked with in Dallas. They picked a bank reconciliation that ate time every single day and got Copilot working on that one thing.
  2. A before number. How long the task takes today, how often it gets redone, how many people touch it. Write it down now, because you won't be able to reconstruct it in March. A rough number the team agrees on is fine. It only has to exist.
  3. The team that owns that workflow. Give the licenses to the people who actually do the work, and keep them together. They help each other get unstuck, and you get a clean read.
  4. A decision date with a rule. Pick a date 60 or 90 days out. Decide in advance what result means expand, what means fix and retry, and what means stop. Write the stop line too. If you can't write it, you haven't designed a pilot yet.

Before the licenses go out, finish this sentence: "We will know this worked if ____ by ____." If you can't fill in both blanks, the pilot can't give you an answer.

"We don't have time to run this like a science project"

Fair. Most people who own Copilot adoption have that job on top of another one. But this version is less work. One workflow instead of twelve. One team instead of a scattered group of volunteers. One page that says what you're testing and when you'll decide.

The vague pilot is the expensive one. It eats six months and leaves you exactly where you started, with a renewal decision and nothing to hand finance.

The other objection I hear is "what if it fails?" Then you learned something real for the cost of a small batch of licenses, and you learned it before you bought more. When a pilot like this comes back with a no, it usually means one of two things. The workflow was the wrong one, or people never got hands-on help with their actual work. Both are fixable, and now you know which one to fix.

What happens when the pilot has a question in it

The finance team's story in Dallas didn't end with the reconciliation. Other teams started asking how they did it. Nobody had to mandate anything. People started pulling licenses from them.

That matters for your business case. The expansion request now comes from the business, with a before and after attached, and IT isn't the one doing the selling.

It's also how you walk into a renewal with something to say. A usage percentage on its own tends to invite a cut. "This task got faster for this team, here's the before and after, and two other teams have asked for the same setup" invites a conversation about where to go next.

When we run Copilot Catalyst with a customer, this is where it starts: one group, their real work, weekly hands-on sessions, and a scorecard that tracks observable behavior change. You can do the setup above on your own, though. None of it requires buying anything.

Do this before Friday

Take the pilot you're running now, or the one you're about to start, and write that one sentence. If the blanks won't fill, pick one workflow this week, get the team that owns it to agree on a rough before number, and put the decision date on the calendar.

If you're heading into a renewal or an expansion decision and your pilot didn't give you an answer, reply or grab time with me. I'm happy to look at what you have and help you set up a version that does.

VisualSP accelerates digital adoption, digital transformation & user training.

Get a Demo
Table of Contents

Stop Pissing Off Your Software Users! There's a Better Way...

VisualSP makes in-app guidance simple.