Dr. Michael Patton on AARC's Sucess Rate

[psy]

http://www.cbc.ca/fifth/2008-2009/power ... atton.html

CBC: So AARC, your involvement with them was to supervise a grad student who was doing the study?

PATTON: That’s right.

CBC: Or who was I should say analyzing the data?

PATTON: Analyzing the data, yes. We didn’t do the study, and he didn’t do the study. What he did was analyze the data…

CBC: So how did you end up being involved with them then?... Was it just that they contracted you to do that evaluation?...

PATTON: They knew me from my writings on evaluation—I’ve written textbooks about evaluation—and contacted me. And right at that moment I had this graduate student whose field of study was chemical dependency.

CBC: And they were interested in doing this study, and they just wanted your professional involvement I guess.

PATTON: That’s right. They were looking for a professional evaluator to analyze the data, and chemical dependency evaluation is not my specialty. I’m a generalist evaluator. And I’ve written general textbooks about evaluation. But when they called I had this graduate student for whom this seemed like a perfect project. He was working on his dissertation at the time. And this was a way for him to use his expertise and get some additional support at the end of his doctoral program. And so I agreed to supervise him to do the analysis.

CBC: Right, okay. And did you or your grad student, were you involved in designing the study or helping them design the study? Or is it just the evaluation part?

PATTON: They developed the instrument. I reviewed the instrument with him [his grad student] to be sure that the analysis could be done. And it looked fairly straightforward. He went to Edmonton and spent two or three days up there as part of putting this together. And I actually don’t remember what all he did.

It was clear that our role was not to validate the program or endorse the program in any way. And that’s not what the report does. In fact, you’ll note that the report very carefully describes it as an outcomes study only. So there’s nothing that he was involved in or that I was involved in, in actually looking at the model that they do. We had no involvement with that. There’s no documentation of the model.

There was no direct contact in data gathering in the instrument. So it was to review that the questions were appropriate questions for an outcome study. And then they gathered the data, and he analyzed it with a colleague at Hazelden Foundation.

CBC: Okay. Were you involved in evaluating the data itself?

PATTON: We provided them with the analysis that is the centrepiece of the report, that is how the results came out. I remember adding to the limitations section, which I cited to you yesterday, trying to be careful that the report was not inappropriately used. And so that was my main contribution. But the data analysis that’s presented in that outcome study is right out of the results that they sent to Hazelden.

CBC: Okay. So the study then—who wrote the words that are the bulk of it that we see there? I have just seen the version that I sent to you, and I assume that it looks pretty much like what you have I guess?

PATTON: Yes.

CBC: Is it…

PATTON: Yes.

CBC: … pretty much the same thing?

PATTON: I mean the findings are descriptive findings for the most part—this is what the data said. As I recall there was some back and forth in the final writing about how much was going to be in it and what. So that there are, what, three or four names listed on it, and I presume everybody did some of the writing. I certainly reviewed it, especially adding to the limitations section. But the focus is on what the followup results were, as reported in the questionnaire and interviews, and so it is not more than that.

It’s not a—there are always difficulties in this kind of self-reporting. It is common in evaluation in general and chemical dependency programs specifically to have the problem of relying upon what people tell you after the fact.

So this doesn’t include independent validation of that. I remember that there were some parents contacted where they couldn’t reach the kids. And of course that’s another source of data, but that’s subject to its own problems. So I would treat it as a fairly modest study that is one part of a bigger puzzle, not as a definitive piece of work, as I told you yesterday.

* * *

CBC: It occurred to me that because some of the kids there are court-ordered and would have been on probation and part of their probation condition is that they attend there, now if they are being interviewed by someone from the program, would they have perhaps a bias or an incentive to underreport substance abuse…

PATTON: Sure.

CBC: … because in that case they could have trouble with the court?

PATTON: All the kids could have incentives for underreporting. They want to please the program staff if they have relationships with them. They know what outcome they’re supposed to report. They want to show that they’ve done well. In some cases, people actually believe that they’re doing better than they are. They’re in denial themselves about their use patterns. There’s the problem you mentioned of the court. That’s all the problem with self-reporting data. There are lots of reasons why self-report data are a first level, the very first thing you do to see how it looks on the surface.

There are studies where the self-report data are bad enough that you say, “Well, it’s not worth going to a lot of trouble to validate it because the self-report data are weak.”

And self-report data work best where the data are gathered anonymously by independent people, where there’s no incentive to worry about. And you know, it gets harder with kids to understand what it means when they’re told that they won’t be identified as individuals, that their data will only be aggregated with other kids, that nobody will know what their responses are. And they sign consent forms saying that they understand all of that. But you don’t really know if they trust that.

CBC: In this case was it done anonymously?

PATTON: Well, it can’t be done anonymously because they’re interviews.

CBC: Oh, there were interviews in this case.

PATTON: Yeah.

CBC: It wasn’t anonymous interviews.

PATTON: No.

[Anonymous]

http://www.cbc.ca/fifth/2008-2009/power ... study.html

About AARC's "80% Success Rate"
The Alberta Adolescent Recovery Centre boasts of a success rate unheard of in the addictions field—with more than 80 percent of its grads said to be “clean and sober.”

That claim is based on what AARC’s website calls an “outcome evaluation,” which it says was “completed” by Dr. Michael Patton, a leading U.S. professional evaluator of programs. The study is referred to as “Patton’s research” and “Patton’s study.”

As recently as last year, AARC described the study as an “independent outcomes validation study,” according to an AARC funding submission document sent to the Alberta Alcohol and Drug Abuse Commission, which the fifth estate obtained through the province’s freedom of information legislation.

We obtained a version of the 2003 study and showed it to three psychology professors who specialize in addiction—the University of Calgary’s David Hodgins, the University of Lethbridge’s Robert Williams and Bruce Alexander, professor emeritus at Simon Fraser University.

All three raised questions about the way the study was carried out. While Hodgins described the study as “not a bad program evaluation,” he, Williams and Alexander all listed flaws. Among them:

The success rate doesn’t include people who didn’t finish the program.

The grads were interviewed by people linked to AARC. This could bias what was reported, Alexander said. “Imagine calling up somebody who’s graduated from a program and saying: ‘Hey, are you taking drugs any more?’ And this person has already been put in the program against their will perhaps precisely because they took drugs. And what are they going to say? ‘Oh yes, I’m taking lots of drugs now,’” Alexander said.

As for whether the study is “independent,” Williams characterized it as “semi-independent.” He said in an email, “It is always better to have a totally independent evaluation. However, it is not unusual for ‘in-house’ evaluations to occur.”

The fifth estate also asked the man who AARC says completed the study—Dr. Patton. He told the fifth estate his involvement was largely limited to supervising a graduate student who crunched the data—data gathered by people associated with AARC.

“I did not conduct the study. They conducted the study. I oversaw the analysis,” he said.

Patton said that while the study was a good preliminary “internal evaluation” with positive results, the next step would be to review AARC’s success rate independently. He noted that the study was rejected for publication by two journals.

“It’s expensive of course to commission an external evaluation. But, that would be the next step. I do remember that the internal evaluation results were quite positive. But, the evaluation that was done did not independently examine the process. The graduate student that I supervised did not independently talk to any of the young people or the parents.

He simply analyzed the data that they sent him. And I was the supervisor of him which is how my name ends up on the report,” Patton said.

 Click here for more of Dr. Michael Patton’s interview with the CBC

AAARC’s research has faced criticisms before. In 1994, the Alberta Alcohol and Drug Abuse Commission raised questions about an earlier AARC evaluation of its success rate.

At the time, the commission wanted AARC to have an independent study of its program done by an experienced, credible research group of its program as a condition of a $100,000 grant.

AARC did submit a study. It is even mentioned on AARC’s website, where it is described as “an external review.”

The commission wasn’t so sure. One of its researchers reviewed the study and noted that, in her opinion, it “was not conducted by an independent researcher, but by people associated with AARC,” according to a commission memo obtained through the freedom of information legislation.

That researcher’s conclusion: AARC’s study was not “technically adequate based on widely accepted standards of research and evaluation.”

[Anonymous]

IN case you didn't catch that anon AARC guy... Fifth Estate has revealed this study is BUNK!

It's inside information
Wasn't completed in the manner they claim
Didn't include all program participants

The one thing AARC counts on to sway parents, politicians and the public alike is GARBAGE.

Take a look all the clients who have gone through the doors of AARC in the last year... EVEN all the graduates only and there isn't even CLOSE to an 80% success rate!

Then again... I guess you need to wonder what is considered "success"?

I highly doubt it's the psychological and emotional state of the client!

[Ursus]

The success rate doesn’t include people who didn’t finish the program.

The grads were interviewed by people linked to AARC.

Lols. Typical program self-evaluation SOP. This reminds me of how Hyde School claims a 98% college matriculation rate. [They fail to mention that roughly 40% of any given class never make it to the end of the school year (with Hyde keeping the entire year's tuition). Plus, it is my understanding that one of the current requirements for graduation from Hyde is a college acceptance letter (whether or not a Hyde graduate actually finishes that first year at college is not data that Hyde is interested in). Alumni are further encouraged to contact the school to fill out some sort of a "satisfaction survey," which is handled by Hyde personnel.]

I am so glad CBC/Fifth Estate delved deep into the nit-picky details of AARC's so-called success rate particulars. Nothing but spin-marketing and sheer bunk science! Bunk! Bunk! Bunk!

[Anonymous]

I would love to hear Deans typical response of this person is a liar so they must be an addict.