A Simplified Method for Routine Outcome Monitoring after Drug Abuse Treatment

EXECUTIVE SUMMARY

The routine collection of drug treatment outcomes to manage quality of care, improve patient satisfaction, and allocate treatment resources is currently hampered by two key difficulties: (1) problems locating clients once they leave treatment; and (2) the prohibitive cost of obtaining meaningful and reliable post-treatment data. This study investigated precise methods for an economical staff-based Routine Outcome Monitoring (ROM) system using an 18-item core measure telephone survey. As implemented at Narconon of Oklahoma, a behavioral and social skills based, residential drug rehabilitation program, the system was determined to be psychometrically adequate for aggregate reporting while providing clinically useful information.

The study analyzed the procedure developed and implemented by Narconon of Oklahoma (“Narconon”) for monitoring the long-term outcomes achieved by the graduates from its drug and alcohol rehabilitation program. This paper describes the precise methods for an economical staff-based Routine Outcome Monitoring (ROM) system using an 18-item core measure telephone survey as used in the Narconon program and the results of the outcome monitoring process.

The goals of routine outcome monitoring are straightforward: to ensure reproducible treatment effectiveness, consistency, and cost-effectiveness; to improve the overall quality of treatment; and to ensure accountability of health service providers to funding sources by monitoring their outcomes and maintaining treatment quality.

The systematic tracking of clients after they have completed a full course of intervention, when they are operating under minimal supervision as a member of their family, workforce and community, is one of the most convincing methods for demonstrating real-world effectiveness of behavioral health programs. Despite stated aims to gather meaningful post-discharge data, most performance-monitoring efforts are still in development stages. Limitations to data collection include: (1) difficulty tracking clients once they leave the treatment setting; (2) using treatment staff to collect follow-up data when their main function is providing chemical dependency services; and (3) relying on personal interviews and other time-consuming protocols that are costly, require substantial staff training, and may result in data drift or loss due to follow-up complexity.

Narconon has developed a streamlined, telephone-based routine outcome monitoring process which is conducted within the context of a continuing care facility. Utilizing a short outcomes survey, it is sufficiently simple to fit within the routines of staff and a smaller facility and has the advantage that it can be implemented by individuals who have not been trained in research methods. The procedure enables the program to obtain rapid feedback and thereby locate and work with clients who have encountered difficulties after leaving the program and, at the same time, enables data to be obtained which is useful for monitoring the overall effectiveness of the rehabilitation program, thereby enabling program managers to make adjustments which may be needed to increase the effectiveness of the program.

The purpose of this study was to evaluate the efficacy of a post-treatment Routine Outcome Monitoring (ROM) system as a tool for measuring, and improving, results from drug rehabilitation services. To achieve this, Narconon International and Psychometric Technologies Incorporated developed a scientifically grounded methodology to obtain useful monitoring data and implemented this in conjunction with enrollment, case management and client follow-up systems which were already in place at Narconon. Initially a 10-item questionnaire was developed but this was expanded during the course of the study based on recommendations of the aftercare and case management staff. Ultimately a survey was finalized which consists of six items directed at self-reported drug and alcohol use in the past 30 days, two items directed at general drug use since leaving treatment, five items directed at quality of life issues in the past 30 days, and several other questions useful in assessing the success of the individual in re-integrating into the community.

For purposes of developing and evaluating this instrument and the ROM methodology this project was limited to those individuals who had completed the full Narconon program (called “graduates”). Narconon staff compiled a list of individuals who had completed the program during the period of 2004 to 2007.

Aftercare specialists then conducted the ROM surveys by telephone. Initially, in addition to telephonic surveys of graduates, surveys were done of close relatives of the graduates in order to assess if the graduate data was reliable. Once it was found that there was a high statistical correlation between the data from the graduates and from their closest relatives, relatives were surveyed to obtain data when, after three attempts, graduates could not be reached directly. Data obtained was de-identified in keeping with Federal and local confidentiality rules and provided to Psychometric Technologies Incorporated for analysis.

Data was obtained from 323 of the 419 subjects who returned to their community; leaving 22.9% with missing data. The first sampling point had an inadequate follow-up rate. By reviewing the successful actions of other researchers in this field1 improvements were made as follows: (1) Staff were trained to use a simple enrollment form to collect multiple phone and email contact information from the client as well as multiple collateral phone and address contact data; (2) All contact information was verified and updated at discharge and, (3) A written checklist was implemented to organize each step of the follow-up process. As a result of these improvements, post-treatment contact rate consistently improved to upwards of 80 percent.

The data from graduates and relatives (collateral sources) was analyzed statistically and found to be highly consistent. From this the validity of the self-report measures used in the ROM approach were established. Out of the total data set, 72.1% of the data used was self-reported by the graduates.

The data was analyzed to determine the degree to which drug problems recurred among Narconon program graduates. The following results were obtained when graduates (or collaterals) were asked about drug use during the 30 days preceding the survey interview:

Of particular note is the ability to monitor 30 day usage rates of drugs and alcohol of former program participants, in order to establish the effectiveness of program procedures. In this case, as shown in Table 5, above, the data showed:

One of the additional purposes for instituting the ROM procedure was to determine its feasibility in monitoring program graduates in order to be able to assist them should they encounter further difficulties with drug or alcohol use. To this end, a question was included to determine if the individual required rehabilitation services since his or her graduation from the program. It was found that only 12% of those completing the program reported having a need for further rehabilitation services.

This report shows the feasibility of a Routine Outcome Monitoring (ROM) system for use in drug abuse treatment facilities. Telephone-based recovery management can address several factors including:

(1) Stabilizing and sustaining recovery congruent with an individualized care philosophy; (2) Any stigma associated with returning to a facility after completing treatment; (3) Reaching clients who live at a distance from the treatment facility; and, (4) Potential staff and financial burden.

Data obtained using the ROM system seems adequate for compliance with grant reporting or accrediting agency requirements. Excepting the first follow-up point, this project consistently produced a representative sampling exceeding 80 percent and required very little staff training. Collateral verification in the early part of the project indicated minimal bias from “grateful testimonials”—where the client might not want to hurt the counselor’s feelings by reporting lack of success—possibly due to using non-treatment staff for follow-up.

This project was undertaken at the request of a program desiring to continuously improve treatment results including those that are not as expected. Engaging treatment program staff in the process of monitoring their own outcomes and incorporating the data obtained into management decisions will likely lead to greater responsibility for improving treatment results. Calculations of treatment effectiveness for purposes such as third-party funding decisions could be strengthened by including a 10% random sample verification by independent telephone-based ROM.

This project focused only on those clients who completed the program. Follow-up rates, and especially outcomes data, can be generalized only to those who complete the full course of care. Future projects to further refine this ROM system should explore feasibility with all discharge categories. For quality assurance and program improvement purposes, it is important to obtain data from people who leave the program prior to completion.

Quality treatment should be effective at changing drug use behavior after the patient leaves treatment. The primary objective of any ROM system is to provide an ongoing trail of long-term outcome data with which to assess any changes in treatment outcomes that might reflect changes in quality or therapy drift.

REFERENCES:

1. Glasner-Edwards S, Rawson R. Evidence-based practices in addiction treatment: review and recommendations for public policy. Health Policy. Oct 2010;97(2–3):93–104.
2. Rosa C, Ghitza U, Tai B. Selection and utilization of assessment instruments in substance abuse treatment trials: the National Drug Abuse Treatment Clinical Trials Network experience. Substance Abuse and Rehabilitation. Jul 17, 2012;3(1):81–9.
3. 111th Congress of the United States of America. GPRA Modernization Act of 2010. Washington, DC; 2010 http://www.gpo.gov/fdsys/pkg/BILLS-111hr2142enr/pdf/BILLS-111hr2142enr.pdf. Accessed Jul 21, 2013.
4. Darby K, Kinnevy SC. GPRA and the development of performance measures. Journal of evidence-based social work. Jan 2010;7(1):5–14.
5. Center for Substance Abuse Treatment (CSAT) Data Collection Tools: Discretionary Services. Available at: https://www.samhsa-gpra.samhsa.gov/ CSAT/System.aspx. Accessed Jul 22, 2013.
6. McCorry F, Garnick DW, Bartlett J, Cotter F, Chalk M. Developing performance measures for alcohol and other drug services in managed care plans. Washington Circle Group. The Joint Commission Journal on Quality Improvement. Nov 2000;26(11):633–43.
7. Harrison PA, Asche SE. Outcomes monitoring in Minnesota: treatment implications, practical limitations. Journal of Substance Abuse Treatment. Dec 2001;21(4):173–83.
8. Soldz S, Panas L, Rodriguez-Howard M. The reliability of the Massachusetts Substance Abuse Management Information System. J Clin Psychol. Sep 2002;58(9):1057–69.
9. Evans E, Hser YI. Pilot-testing a statewide outcome monitoring system: overview of the California Treatment Outcome Project (CALTOP). Journal of Psychoactive Drugs. May 2004;Suppl 2:109–14.
10. Minnesota Department of Human Services. DAANES Web User Manual for Chemical Dependency Treatment Programs. Minnesota Department of Human Services, DAANES Data Processing Unit, Performance Measurement and Quality Improvement Division Saint Paul, Minnesota, 2013. http://www.dhs.state.mn.us/main/groups/business_partners/documents/​pub/dhs16_152493.pdf. Accessed Jul 21, 2013.
11. Company. Minnesota’s Statewide Strategy for Drug and Violent Crime Control. DIANE Publishing Company; 1995.
12. Chi FW, Parthasarathy S, Mertens JR, Weisner CM. Continuing care and long-term substance use outcomes in managed care: early evidence for a primary care-based model. Psychiatric Services. Oct 2011;62(10):1194–200.
13. Wilkerson D, Migas N, Slaven T. Outcome-oriented standards and performance indicators for substance dependency rehabilitation programs. Substance Use and Misuse. Oct–Dec 2000;35(12–14):1679–703.
14. Substance Abuse and Mental Health Services Administration. Alcohol and Drug Services Study (ADSS): The National Substance Abuse Treatment System: Facilities, Clients, Services, and Staffing. Office of Applied Studies. Rockville, MD, 2003. http://www.samhsa.gov/data/ADSS/ADSSOrg.pdf. Accessed Jul 21, 2013.
15. McKay JR, Van Horn DH, Oslin DW, et al. A randomized trial of extended telephone-based continuing care for alcohol dependence: within-treatment substance use outcomes. Journal of Consulting and Clinical Psychology. Dec 2010;78(6):912–23.
16. McKay JR, Lynch KG, Shepard DS, Pettinati HM. The effectiveness of telephone-based continuing care for alcohol and cocaine dependence: 24-month outcomes. Arch Gen Psychiatry. Feb 2005;62(2):199–207.
17. Godley MD, Godley SH, Dennis ML, Funk R, Passetti LL. Preliminary outcomes from the assertive continuing care experiment for adolescents discharged from residential treatment. Journal of Substance Abuse Treatment. Jul 2002;23(1):21–32.
18. Paredes A. The Narconon Drug Rehabilitation Program: A descriptive overview. http://www.narconon.org/Narconon_program_overview_DrParedes.pdf. Accessed Aug 6, 2013.
19. Schnare DW, Denk G, Shields M, Brunton S. Evaluation of a detoxification regimen for fat stored xenobiotics. Medical Hypotheses. Sep 1982;9(3):265–82.
20. Cecchini M, LoPresti V. Drug residues store in the body following cessation of use: impacts on neuroendocrine balance and behavior—use of the Hubbard sauna regimen to remove toxins and restore health. Medical Hypotheses. 2007;68(4):868–79.
21. McLellan AT, Kushner H, Metzger D, et al. The Fifth Edition of the Addiction Severity Index. Journal of Substance Abuse Treatment. 1992;9(3):199–213.
22. Moos RH, King MJ. Participation in community residential treatment and substance abuse patients’ outcomes at discharge. Journal of Substance Abuse Treatment. Jan–Feb 1997;14(1):71–80.
23. Prendergast ML, Podus D, Chang E. Program factors and treatment outcomes in drug dependence treatment: an examination using meta-analysis. Substance Use and Misuse. Oct–Dec 2000;35(12–14):1931–65.
24. Moos RH. Theory-based processes that promote the remission of substance use disorders. Clin Psychol Rev. Jun 2007;27(5):537–51.
25. Weisz JR, Weiss B, Donenberg GR. The lab versus the clinic. Effects of child and adolescent psychotherapy. The American Psychologist. Dec 1992;47(12):1578–85.
26. Center for Substance Abuse Treatment (CSAT). CSAT GPRA Client Outcome Measures for Discretionary Programs. http://www.samhsa.gov/Grants06/downloads/CSAT_GPRA_ClientOutcome2006.pdf. Accessed Aug 6, 2013.
27. McLellan AT, McKay JR, Forman R, Cacciola J, Kemp J. Reconsidering the evaluation of addiction treatment: from retrospective follow-up to concurrent recovery monitoring. Addiction. Apr 2005;100(4):447–58.
28. Laudet AB, White W. What are your priorities right now? Identifying service needs across recovery stages to inform service development. Journal of Substance Abuse Treatment. Jan 2010;38(1):51–9.
29. McLellan AT, Luborsky L, Woody GE, O’Brien CP, Kron R. Are the “addiction-related” problems of substance abusers really related? J Nerv Ment Dis. Apr 1981;169(4):232–9.
30. McLellan AT, Cacciola JC, Alterman AI, Rikoon SH, Carise D. The Addiction Severity Index at 25: origins, contributions and transitions. The American Journal on Addictions/American Academy of Psychiatrists in Alcoholism and Addictions. Mar–Apr 2006;15(2):113–24.
31. Desmond DP, Maddux JF, Johnson TH, Confer BA. Obtaining follow-up interviews for treatment evaluation. Journal of Substance Abuse Treatment. Mar–Apr 1995;12(2):95–102.
32. Carroll ME, Anker JJ, Perry JL. Modeling risk factors for nicotine and other drug abuse in the preclinical laboratory. Drug and Alcohol Dependence. Oct 1, 2009;104 Suppl 1:S70–8.
33. Gerstein DR, Green LW, editors. Preventing Drug Abuse: What Do We Know? Washington, DC: National Academy Press; 1993.
34. Greenfield L, Burgdorf K, Chen X, Porowski A, Roberts T, Herrell J. Effectiveness of long-term residential substance abuse treatment for women: findings from three national studies. The American Journal of Drug and Alcohol Abuse. Aug 2004;30(3):537–50.
35. Oudejans SC, Schippers GM, Merkx MJ, Schramade MH, Koeter MW, van den Brink W. Feasibility and validity of low-budget telephonic follow-up interviews in routine outcome monitoring of substance abuse treatment. Addiction. Jul 2009;104(7):1138–46.
36. Tiet QQ, Byrnes HF, Barnett P, Finney JW. A practical system for monitoring the outcomes of substance use disorder patients. Journal of Substance Abuse Treatment. Jun 2006;30(4):337–347.
37. Stanford M, Banerjee K, Garner R. Chronic care and addictions treatment: a feasibility study on the implementation of posttreatment continuing recovery monitoring. Journal of Psychoactive Drugs. Sep 2010;Suppl 6:295–302.
38. American Society for Addiction Medicine (ASAM). American Society of Addiction Medicine Patient Placement Criteria for the Treatment of Substance-related Disorders, Second Edition Revised. Second Edition, Revised ed. Chevy Chase, MD: Lippincott Williams & Wilkins; 2001.
39. Campbell DT. Reforms as experiments. Amer Psychol. 1969;24:409–29.