Are Third Way’s Focus Groups Valid Research?

Much attention has been lavished on this Molly Ball piece in The Atlantic on the centrist think tank Third Way’s listening tour across America. In it, Ms. Ball subtly fillets the Third Way’s domestic anthropologists in their search for answers that already align with the group’s preconceptions about a fundamentally centrist, moderate America that wants small government, local control and even-tempered politicians in line with the preferences of the group’s corporate donors. Per her story, the focus groups seemed to show one thing, but the conclusions from Third Way showed another.

But the reason I write this is to highlight something more disturbing from the piece that speaks to the Third Way’s methodology in doing the research. Before I do, though, I need to state a few disclaimers.

I considered long and hard about whether to write this because my day job is in qualitative marketing research–basically, I design, moderate and analyze focus groups, interviews, ethnographies, etc.  First, it’s generally bad form to slag on the work of other professionals and to call their expertise or integrity into question, and one wants to avoid doing so unless there is a very good reason. Second, inevitably any discussion of the subject based on one’s own expertise in a forum like this is bound to generate accusations of venal self-promotion–which is certainly not the intent here. So let me make very clear that I’m writing this not out of self-interest but simply because I’m in a professional position to pass judgment on it, and because the results of the research have a profound impact on our nation and its public policy, given that the Third Way is used as a major supplier of advice and research to the national Democratic organizations.

Let’s look at one particular excerpt from the article that should catch our attention:

Hale or her colleague Luke Watson, Third Way’s deputy director of strategy, began each Wisconsin focus group with a variation on the same refrain.

“We are a think tank that deals with what the plurality of Americans are thinking about—in other words, we don’t spend a lot of time on the ideological edges,” one of the two would explain. “It has started seeming like the far left and the far right were the only voice in America, but we know that’s not true. We focus on the 70 percent in the middle, because we think most of us, as Americans, are there.”

Now, let me be clear that I have not been in contact with the Third Way to confirm Ms. Ball’s reporting on this, but I have no reason to doubt her. If this is true, literally nothing any respondent says in the focus group from that point onward can be taken at face value. The groups are poisoned from the very beginning by breaking a number of industry rules designed to minimize bias.

1) Never reveal the sponsor of the research. Third Way’s recruiters apparently told respondents not just during the groups but apparently in the screening process, which means that respondents were at liberty to research the group, its motives and its funding in advance. The potential for problems is enormous: respondents will game their answers to fit the organization’s interests, or conversely come in hostile to the organization. Which is apparently exactly what happened:

It was a thesis that would not go unchallenged, even in flyover country. In rural Wisconsin, it turns out, the natives have Google….“Isn’t this underwritten by the DNC?” a local cafe owner asked Watson after his just-here-to-listen opening spiel. “I read somewhere you’re spending $20 million,” another man said. Another participant asked about corporate donors.

Once a group starts this way, it warps the entire dynamic and limits the honesty of the responses.

Now, sometimes the nature of the material presented makes it obvious who the organization is at least by the middle of the sessions, but that doesn’t seem to be the case here. Occasionally it may be difficult to gain respondents’ trust to attend the session if they don’t know something about the sponsor, but even then it usually suffices to state generalities such as “we’re a policy organization dedicated to finding solutions to America’s problems.” People are generally happy to come state their political opinions, especially if there’s a cash incentive. The key is not

2) Don’t have someone internal to the organization actually collect and analyze the data. Increasingly, organizations are ignoring this for cost-saving reasons, but best practice is to have an independent firm handle data collection and reporting so that it’s clear that the organization’s own internal pressures aren’t biasing the data or the conclusions. From Ms. Ball’s reporting, it seems that Third Way staff designed, executed, collected and analyzed the findings internally. That might be marginally acceptable if you’re quickly trying to figure out which car ad or refrigerator features to go with, but not for a $20 million study to set the direction of one of the world’s two most powerful political parties after a devastating defeat.

3) Do not bias respondents toward your objectives at the beginning of the groups. This is by far the worst of the sins apparently committed by Third Way. It’s so egregious that one is left to wonder whether the malpractice was intentional or simply incompetent. Let’s review how the groups began:

Hale or her colleague Luke Watson, Third Way’s deputy director of strategy, began each Wisconsin focus group with a variation on the same refrain.

“We are a think tank that deals with what the plurality of Americans are thinking about—in other words, we don’t spend a lot of time on the ideological edges,” one of the two would explain. “It has started seeming like the far left and the far right were the only voice in America, but we know that’s not true. We focus on the 70 percent in the middle, because we think most of us, as Americans, are there.”

Even for a non-research professional, the red flags here should be obvious. Moderators have enormous power of persuasion in a focus group room, and the psychological tendency of respondents in a group setting is to adapt their answers to the moderator’s suggestions. Moderators need to be extremely careful in how they phrase even the innocuous questions to avoid pushing respondents toward one direction or another. They certainly should not be telling respondents what the organization’s own beliefs are right out of the gate.

If the entire purpose of Third Way’s listening tour was to discover if Americans could still come together toward centrism or whether they were more energized by partisan persuasion, they ruined their chance at a valid answer before the first respondent even had a chance to speak. Basic human suggestibility would lead any group toward responses more in keeping with Third Way’s own viewpoints, and quash those who harbored more openly ideological notions.

In short, the groups seem to have been invalid before they began, and any results from them would be fruit of a poisoned tree. And even still, if Ms. Ball’s reporting is to be believed, the final analysis veered significantly from the actual data.

If I were the Third Way’s donors and customers, I would be asking some hard questions not just about their conclusions, but also about their methods.