BS in CS: Issue #5

You must manage CSMs by their KPIs

This continuing series explores common, but misleading and sometimes dangerous BS in CS. We will apply science to challenge faulty beliefs and suggest better alternatives.

Item #5: You must manage CSMs by their KPIs

Obviously business is all about people. You must evaluate your CSMs on Key Performance Indicators such as account retention, NRR, engagement, Success Plans, CSQLs, etc. And if you coach them to make their numbers, then you’ll get better results.

Except it’s BS. People are essential, but individual differences matter less than people think. Systemic issues primarily determine results. So managers spend too much time focused on the wrong things. They fault the people instead of improving the system.

Where’s the proof?

People vs. Process

As described in a previous post, people are just one part of a business process: methods, equipment, inputs, environment, and information are also present. All factors vary cycle-to-cycle, interacting, adding and subtracting from each other, causing the results to vary, too. If the process is stable, meaning there’s no trending or chaotic dynamics, then we’ll likely see a bell-curve distribution in a histogram of the data:

It’s a common shape because we typically use averages, and the central limit theorem applies. It states that the distribution of sample averages will tend to be normal if the number of samples is large enough. The same thing is true in the multivariate case. If three or more different factors combine to produce an outcome, then that distribution will also be normal.

So whenever we see a bell curve, the elements are inherently mixed, and the results reflect the combination of all of them. It’s impossible to isolate the contributions of any one factor without the aid of regression analysis or classifiers. The specific impact people make, much less any single individual, can’t be determined simply by looking at the data.

Few managers, however, understand this. They frequently rank individuals by results, rewarding those at the top while punishing those at the bottom. In this example, a manager ranks CSMs by Average Expansion $K. Andy is clearly the hero with an average of $153.6K and Joe is the villain with only $107K:

But if we examine the distribution, we see the standard normal. Here the mean (x-bar) is $129K and the sample standard deviation (s) is $14.2K. Applying probability theory, about 95% of the observations will naturally fall within +/- 2 standard deviations of the mean by chance. In this example, $129K - 2*$14.2K = $100.6K, and Joe’s number is within that limit:

So it’s statistically unlikely Joe’s results have anything to do with him. They’re probably caused by other factors, such as the differences in revenue opportunities between accounts. His relative ranking is therefore determined by randomness. Rather than blame him, the manager should simply recognize how the math works.

That’s not to say people are never the cause. If an individual’s results consistently fall outside the norm, then there’s good evidence something’s amiss. If Joe’s results were less than $100K for two consecutive periods, for example, then the likelihood it would have occurred randomly would be 0.05*0.05=0.0025 or 0.25%. In this case, his manager should act, investigating whether it was a fluke or a lack of knowledge, skills, or motivation on Joe’s part. On the other hand, if Joe’s results were consistently more than two standard deviations above the mean, then the manager should learn what makes him special compared to his peers.

Systemic Faults

W. Edwards Deming, regarded as the father of the Total Quality movement, said the vast majority of the issues we face can be attributed to the system, not the people. A real-life example proves his point.

In contact centers, Customer Support Representatives are tasked with resolving customer problems quickly. Managers monitor Quality Assurance (QA) scores, which indicate how well the agent handles inbound calls following procedures, diagnosing and addressing issues, and treating customers with care. Average handle time (AHT) is also important. Time is money, so the faster an agent completes a call, the lower the cost. Consequently, managers typically use both QA and AHT to evaluate agent performance.

Believing better CSR motivation was essential for better results, managers at one call center implemented an incentive program. They ranked individual agents by their combined monthly QA and AHT numbers, paying bonuses to the person on top.

However, the system was fundamentally flawed. After analyzing the rankings of 55 agents across two months, individual differences mattered in the results for only 6% of the agents. And just one of those, Christy, ranked in the upper 25% of her peers in both metrics two months in a row. In fact, she posted results in the top 10%, a situation highly unlikely due to chance. Christy deserved more credit, even though she wasn’t #1 in either month and she didn’t get the bonus. The incentive program, intended to reward agents based on merit, was just a lottery in disguise.

And what was actually causing the variation in monthly results? Inefficient call handling procedures, the wide variety of inbound calls, and varying customer information on hand. Managers were doing the wrong things tinkering with carrots and sticks. They should have been working to improve the system, not the people.

People and Process

The same thing happens in all areas of business, including Customer Success. That’s because most managers suffer a cognitive bias called fundamental attribution error, assuming people, not circumstances, are to blame. They routinely misinterpret data and they take improper actions when it comes to their direct reports.

Indeed, workers are an essential element in most processes and nothing gets done without them. People must have the proper knowledge and skills to do the job. They must inspired to show up every day and give their best, and they must be held accountable for what they do, or fail to do. But individual differences matter very little beyond a certain point. To make substantial and sustainable gains, managers must improve the system itself. Doing so raises the performance of all employees and delivers better results for the business.

So managing CSMs by KPIs doesn’t help that much. And as Deming said, the job of supervisors is twofold: lead people AND improve the system.

It’s time they did both.