http://www.cbc.ca/fifth/2008-2009/power ... study.htmlAbout AARC's "80% Success Rate"
The Alberta Adolescent Recovery Centre boasts of a success rate unheard of in the addictions field—with more than 80 percent of its grads said to be “clean and sober.”
That claim is based on what AARC’s website calls an “outcome evaluation,” which it says was “completed” by Dr. Michael Patton, a leading U.S. professional evaluator of programs. The study is referred to as “Patton’s research” and “Patton’s study.”
As recently as last year, AARC described the study as an “independent outcomes validation study,” according to an AARC funding submission document sent to the Alberta Alcohol and Drug Abuse Commission, which the fifth estate obtained through the province’s freedom of information legislation.
We obtained a version of the 2003 study and showed it to three psychology professors who specialize in addiction—the University of Calgary’s David Hodgins, the University of Lethbridge’s Robert Williams and Bruce Alexander, professor emeritus at Simon Fraser University.
All three raised questions about the way the study was carried out. While Hodgins described the study as “not a bad program evaluation,” he, Williams and Alexander all listed flaws. Among them:
The success rate doesn’t include people who didn’t finish the program.
The grads were interviewed by people linked to AARC. This could bias what was reported, Alexander said. “Imagine calling up somebody who’s graduated from a program and saying: ‘Hey, are you taking drugs any more?’ And this person has already been put in the program against their will perhaps precisely because they took drugs. And what are they going to say? ‘Oh yes, I’m taking lots of drugs now,’” Alexander said.
As for whether the study is “independent,” Williams characterized it as “semi-independent.” He said in an email, “It is always better to have a totally independent evaluation. However, it is not unusual for ‘in-house’ evaluations to occur.”
The fifth estate also asked the man who AARC says completed the study—Dr. Patton. He told the fifth estate his involvement was largely limited to supervising a graduate student who crunched the data—data gathered by people associated with AARC.
“I did not conduct the study. They conducted the study. I oversaw the analysis,” he said.
Patton said that while the study was a good preliminary “internal evaluation” with positive results, the next step would be to review AARC’s success rate independently. He noted that the study was rejected for publication by two journals.
“It’s expensive of course to commission an external evaluation. But, that would be the next step. I do remember that the internal evaluation results were quite positive. But, the evaluation that was done did not independently examine the process. The graduate student that I supervised did not independently talk to any of the young people or the parents.
He simply analyzed the data that they sent him. And I was the supervisor of him which is how my name ends up on the report,” Patton said.
Click here for more of Dr. Michael Patton’s interview with the CBC
AAARC’s research has faced criticisms before. In 1994, the Alberta Alcohol and Drug Abuse Commission raised questions about an earlier AARC evaluation of its success rate.
At the time, the commission wanted AARC to have an independent study of its program done by an experienced, credible research group of its program as a condition of a $100,000 grant.
AARC did submit a study. It is even mentioned on AARC’s website, where it is described as “an external review.”
The commission wasn’t so sure. One of its researchers reviewed the study and noted that, in her opinion, it “was not conducted by an independent researcher, but by people associated with AARC,” according to a commission memo obtained through the freedom of information legislation.
That researcher’s conclusion: AARC’s study was not “technically adequate based on widely accepted standards of research and evaluation.”
