When a Change Is a Signal (and When It Isn’t)

A school leadership team is reviewing their latest reading assessment data.

The latest data point increased compared to the point immediately before it.

Someone in the room says, “Looks like what we’re doing is working.”

That conclusion shows up in schools all the time.
And most of the time, it’s wrong.

But how do we know?

What if the previous five data points looked similar?
What if the system has been moving up and down like this for months or even years?

In that case, the most recent increase may not represent improvement at all.

It may simply be part of the pattern.

Looking at Data Over Time Isn’t Enough

Last month, I argued that we have to read data over time, not one point at a time.

That’s a critical first step. Plotting the dots helps us see how results behave, not just what happened most recently.

But even when leaders look at data over time, a harder question remains: when does a change actually mean something?

Because not every change in the data represents improvement or decline. When you plot data over time, you start to see something important: results move.

Attendance goes up and down.
Assessment scores rise and fall.
Behavior incidents spike and dip.

That movement is not necessarily a sign that something improved or declined. In many cases, it’s simply how the system behaves.

This is what we call routine variation, that is, the natural result of many causes interacting within a stable system.

The mistake leaders often make is assuming that every change in the data must mean something.

It doesn’t.

Why Stable Systems Still Vary

When we say a system is stable, it’s easy to misunderstand what that means.

Stable does not mean producing the exact same result every time. It does not mean that the point-to-point variation is small. And it does not mean there are no causes behind the results.

A stable system still produces variation. But that variation is the result of many different cause-and-effect relationships interacting at the same time, where no single cause dominates.

Taken together, those relationships create a kind of equilibrium. The system settles into a pattern of performance that moves up and down within a predictable range.

That’s what makes it stable. And this has an important implication for leaders: In a stable system, every data point has causes, but no single cause explains it. This means trying to explain every increase or decrease as if it has a clear root cause is usually a mistake.

Signal vs. Noise

To make better decisions, leaders need to distinguish between signal and noise.

• Noise is the natural up-and-down movement of a stable system.

• Signal is evidence that something in the system itself has changed.

Most of what we see in school data is noise. That’s not a problem, but rather how systems often behave.

The problem arises when leaders treat noise like a signal.

A small increase becomes proof that a strategy worked.
A small decrease becomes a problem that needs fixing.
Explanations are requested. Adjustments are made.

But if the system hasn’t changed, those actions won’t improve results. They only create more variation.

When leaders can’t distinguish signal from noise, they fall into a predictable pattern.

They react to every movement in the data.

Priorities shift.
New initiatives are introduced.
Teams are asked to explain results that don’t have a single, identifiable cause.

Over time, this creates confusion, frustration, and instability. This is what happens when we try to improve a system by reacting to its outputs.

You Need a Method

The challenge is that you can’t reliably determine signal vs. noise just by looking at a few data points or by glancing at a line graph.

You need a method.

This is where process behavior charts come in.

A process behavior chart helps us understand what a system typically produces and whether a new result falls within that expected range or outside of it.

Instead of guessing, leaders can use the chart to determine whether a change represents routine variation or a meaningful shift in the system.

A Simple Way to Think About It

Imagine a school where attendance has ranged between 90% and 97% for the past 15 data points.

If the latest result is 93%, most leaders would recognize that nothing unusual has happened. That result falls within what the system has been producing over time.

Now suppose the latest result is 88%.

It’s tempting to assume that something must have changed. But without a method, we can’t know that for sure. The system may have always been capable of producing a result like this. Or, it may not have.

That’s the point.

You cannot determine whether a change is a signal just by looking at a single data point or by looking at a run of data casually.

A surprising number is not the same as a meaningful signal; the only way to know for sure is to plot the data on a process behavior chart.

Without that, you’re guessing.

Putting It All Together

Most changes in your data don’t mean what you think they mean, and without a method, it’s difficult to tell which ones matter.

Three ideas can help leaders respond more effectively.

Big Idea 1: Most changes in data are routine. Not every increase or decrease represents improvement or decline.

Big Idea 2: Treating every change as a signal leads to overreaction, wasted effort, and instability.

Big Idea 3: Leaders need a method to distinguish signal from noise before deciding how to respond.

In the next post, we’ll take the next step: what leaders should do when the system isn’t performing at the level they expect.

Whenever you’re ready, here are three ways to continue the work:

1. EMAIL JOHN
Have a question, an improvement idea, or a moment where the numbers changed and you weren’t sure how to respond? I regularly exchange ideas and resources with educators across the country and beyond, and I’d welcome hearing what you’re working through.

2. IMPROVEMENT ADVISING
If you’re looking for a thought partner who understands the realities of leading complex school systems, I work with leaders to strengthen decision-making, interpret data wisely, and build systems that improve over time—without adding noise or unnecessary initiatives.

3. WIN-WIN: THE BOOK
Win-Win is an improvement science text written for education leaders. It equips readers with the concepts and habits of mind from W. Edwards Deming’s System of Profound Knowledge to help them improve systems, not just react to results.

***

John A. Dues serves as the Chief Learning Officer for United Schools, a nonprofit charter management organization that supports four public charter school campuses in Columbus, Ohio. He is also the author of the award-winning book Win-Win: W. Edwards Deming, the System of Profound Knowledge, and the Science of Improving Schools. Send feedback to jdues@unitedschools.org.