Evaluation Review recent issues

Development of a Medicaid Behavioral Health Case-Mix Model

Robst, J. — Tue, 10 Nov 2009 09:50:43 PST

Many Medicaid programs have either fully or partially carved out mental health services. The evaluation of carve-out plans requires a case-mix model that accounts for differing health status across Medicaid managed care plans. This article develops a diagnosis-based case-mix adjustment system specific to Medicaid behavioral health care. Several different model specifications are compared that use untransformed, square root transformed, and log-transformed expenditures.

An Approach for Addressing the Multiple Testing Problem in Social Policy Impact Evaluations

Schochet, P. Z. — Tue, 10 Nov 2009 09:50:43 PST

In social policy evaluations, the multiple testing problem occurs due to the many hypothesis tests that are typically conducted across multiple outcomes and subgroups, which can lead to spurious impact findings. This article discusses a framework for addressing this problem that balances Types I and II errors. The framework involves specifying confirmatory and exploratory analyses in study protocols, delineating confirmatory outcome domains, conducting t tests on composite domain outcomes, and applying multiplicity corrections to composites across domains to obtain summative impact evidence. The article presents statistical background and discusses multiplicity issues for subgroup analyses, designs with multiple treatments, and reporting.

Evaluation of the CSEC Community Intervention Project (CCIP) in Five U.S. Cities

Ferguson, K. M., Soydan, H., Lee, S.-Y., Yamanaka, A., Freer, A. S., Xie, B. — Tue, 10 Nov 2009 09:50:43 PST

In response to the commercial sexual exploitation of children (CSEC) within five U.S. cities, the CSEC Community Intervention Project (CCIP) was created to enhance collaboration among nongovernmental organization (NGO) representatives, law enforcement officials and prosecutors in Chicago, Atlantic City, Denver, Washington, D.C., and San Diego. A total of 211 participants were surveyed during a 3-day CCIP training institute held in each city. Evaluation data suggest that participants were positively influenced in their knowledge, skills, and attitudes regarding CSEC. Our findings inform NGO representatives, law enforcement officials, and prosecutors of the importance of professional training and the benefits of cross-disciplinary collaboration in addressing CSEC.

Evaluating Monitoring Systems in the European Social Fund Context: A Sociotechnical Approach

Iannacci, F., Cornford, T., Cordella, A., Grillo, F. — Thu, 03 Sep 2009 13:34:32 PDT

In contrast to the prevailing image of monitoring systems as technical systems, it is proposed that they should rather be conceived of as social endeavors at exchanging information. Drawing on the monitoring and evaluation framework of Cornford, Doukidis, and Forster, the concept of information agreement is suggested as a way of assessing the quality of monitoring systems in context. Preliminary implications are discussed with regard to the quality of information, the information agreement being conceptualized as a tacit, and/or explicit agreement between and among participating government partners about the quality of information.

The Influence of ''No Child Left Behind'' Legislation on Drug Prevention in U.S. Schools

Cho, H., Dion Hallfors, D., Iritani, B. J., Hartman, S. — Thu, 03 Sep 2009 13:34:32 PDT

This study examines prevention practices and perceptions in U.S. schools since passage of federal No Child Left Behind (NCLB) legislation, using survey data from state education agencies (SEA) and a population-based sample of school districts. Only one third of U.S. public school districts rely on evidence-based prevention curriculum in middle schools. Funding from other sources and large size were positively associated with using evidence-based curricula. States and districts differed on their perceptions of high-priority activities, and neither supported the federal priority on student drug testing. The findings suggest that there is a disconnect between what NCLB says and what is funded.

A Comparison of Web-Based and Paper-Based Survey Methods: Testing Assumptions of Survey Mode and Response Cost

Greenlaw, C., Brown-Welty, S. — Thu, 03 Sep 2009 13:34:32 PDT

Web-based surveys have become more prevalent in areas such as evaluation, research, and marketing research to name a few. The proliferation of these online surveys raises the question, how do response rates compare with traditional surveys and at what cost? This research explored response rates and costs for Web-based surveys, paper surveys, and mixed-mode surveys. The participants included evaluators from the American Evaluation Association (AEA). Results included that mixed-mode, while more expensive, had higher response rates.

Active Consent in Urban Elementary Schools: An Examination of Demographic Differences in Consent Rates

Thu, 03 Sep 2009 13:34:32 PDT

The consent process is an integral piece of research and evaluation studies, especially when conducted within a school setting. The challenge of reaching parents of students to obtain consent is an issue with which those conducting school-based studies grapple. The literature suggests that the success of the consent process can affect the representativeness of the sample. This study describes one consent approach and examines the demographic differences between the eligible population, the consenters and the refusers. Demographic differences were found between consenters and refusers and suggestions are offered for further research and for other researchers who conduct school-based projects.

Evaluating Retailer Behavior in Preventing Youth Access to Harmful Legal Products A Feasibility Test

Courser, M. W., Holder, H. D., Collins, D., Johnson, K., Ogilvie, K. A. — Thu, 03 Sep 2009 13:34:32 PDT

This article reports results from a feasibility study of a community effort to reduce the availability of legal products that youth can use to get high. The study evaluated the potential of youth purchase attempts to detect actual changes in retail availability of harmful legal products. These results were triangulated with self-reports from retailers about their own policies and practices. Before the intervention, less than half of retailers reported using any of six possible strategies identified as ways to reduce youth access to harmful products, and less than 8% of baseline youth attempts to purchase potentially harmful legal products were refused or questioned. After the low-dosage intervention, retailers reported increased use of three strategies and a statistically significant increase in the percentage of purchase attempts that were either questioned or refused by retail clerks. These findings (a) demonstrate the potential feasibility of retailer-focused environmental strategies and (b) support continued use of youth purchase attempts as a measure of actual retailer behavior.

Design and Analysis of the Community Youth Development Study Longitudinal Cohort Sample

Wed, 01 Jul 2009 14:23:16 PDT

Communities That Care (CTC) is a prevention system designed to reduce adolescent substance use and delinquency through the selection of effective preventive interventions tailored to a community's specific profile of risk and protection. A community-randomized trial of CTC, the Community Youth Development Study, is currently being conducted in 24 communities across the United States. This article describes the rationale, multilevel analyses, and baseline comparability for the study's longitudinal cohort design. The cohort sample consists of 4,407 fifth- and sixth-grade students recruited in 2004 and 2005 and surveyed annually through ninth grade. Results of mixed-model ANOVAs indicated that students in CTC and control communities exhibited no significant differences (ps > .05) in baseline levels of student outcomes.

Incorporating Cost in Power Analysis for Three-Level Cluster-Randomized Designs

Konstantopoulos, S. — Wed, 01 Jul 2009 14:23:16 PDT

In experimental designs with nested structures, entire groups (such as schools) are often assigned to treatment conditions. Key aspects of the design in these cluster-randomized experiments involve knowledge of the intraclass correlation structure, the effect size, and the sample sizes necessary to achieve adequate power to detect the treatment effect. However, the units at each level of the hierarchy have a cost associated with them and thus researchers need to decide on sample sizes given a certain budget, when designing their studies. This article provides methods for computing power within an optimal design framework that incorporates costs of units in all three levels for three-level cluster-randomized balanced designs with two levels of nesting at the second and third level. The optimal sample sizes are a function of the variances at each level and the cost of each unit. Overall, larger effect sizes, smaller intraclass correlations at the second and third level, and lower cost of Level 3 and Level 2 units result in higher estimates of power.

Review and Analysis of Laws Related to Strangulation in 50 States

Laughon, K., Glass, N., Worrell, C. — Wed, 01 Jul 2009 14:23:16 PDT

Nonlethal strangulation of intimate partners has substantial direct health effects and is associated with an increased risk of later lethal violence by a partner or ex-intimate partner but can be difficult to prosecute under felony assault statutes. After review of state laws regarding assault, the authors identified 10 states with specific legislation related to strangulation as of March 2009 The authors examined the characteristics of these laws. They recommend that all states develop policies to improve prosecution of strangulation, include strangulation in their criminal codes, and use language that includes all potential victims.

The Impact of Active Consent Procedures on Nonresponse and Nonresponse Error in Youth Survey Data: Evidence From a New Experiment

Courser, M. W., Shamblen, S. R., Lavrakas, P. J., Collins, D., Ditterline, P. — Wed, 01 Jul 2009 14:23:16 PDT

This article reports results from a student survey fielded using an experimental design with 14 Kentucky school districts. Seven of the 14 districts were randomly assigned to implement the survey with active consent procedures; the other seven districts implemented the survey with passive consent procedures. We used our experimental design to investigate the impact of consent procedures on (a) participation rates, (b) demographic characteristic of the survey samples, and (c) estimates of alcohol, tobacco, and other drugs (ATOD) use. We found that the use of active consent procedures resulted in reduced response rates, underrepresentation of male students and older students, and lower lifetime and past 30-day prevalence rates for most drugs and for most antisocial behaviors. Methodological implications of these findings are discussed along with directions for further research.

The Social Construction of ``Evidence-Based'' Drug Prevention Programs: A Reanalysis of Data From the Drug Abuse Resistance Education (DARE) Program

Gorman, D. M., Huber, J. C. — Wed, 01 Jul 2009 14:23:16 PDT

This study explores the possibility that any drug prevention program might be considered ``evidence-based'' given the use of data analysis procedures that optimize the chance of producing statistically significant results by reanalyzing data from a Drug Abuse Resistance Education (DARE) program evaluation. The analysis produced a number of statistically significant differences between the DARE and control conditions on alcohol and marijuana use measures. Many of these differences occurred at cutoff points on the assessment scales for which post hoc meaningful labels were created. Our results are compared to those from evaluations of programs that appear on evidence-based drug prevention lists.

Predicting Drug Use at Electronic Music Dance Events: Self-Reports and Biological Measurement

Johnson, M. B., Voas, R. A., Miller, B. A., Holder, H. D. — Tue, 05 May 2009 12:02:53 PDT

Most information on the prevalence of drug use comes from self-report surveys. The sensitivity of such information is cause for concern about the accuracy of self-report measures. In this study, self-reported drug use in the last 48 hr is compared to results from biological assays of saliva samples from 371 young adults entering clubs. The relationship between self-reports and drug presence in oral fluid was determined for three substances as follows: cocaine, marijuana, and amphetamine. Forty-one percent of the participants with drugs detected in their oral fluids reported no use in the last 48 hr. The significance of these results is discussed.

Institutional Design and Utilization of Evaluation: A Contribution to a Theory of Evaluation Influence Based on Swiss Experience

Balthasar, A. — Tue, 05 May 2009 12:02:53 PDT

Growing interest in the institutionalization of evaluation in the public administration raises the question as to which institutional arrangement offers optimal conditions for the utilization of evaluations. Institutional arrangement denotes the formal organization of processes and competencies, together with procedural rules, that are applicable independently of individual evaluation projects. It reflects the evaluation practice of an institution and defines the distance between evaluators and evaluees. This article outlines the results of a broad-based study of all 300 or so evaluations that the Swiss Federal Administration completed from 1999 to 2002. On this basis, it derives a theory of the influence of institutional factors on the utilization of evaluations.

A Statistical Perspective on the Design of Drug-Court Studies

Merrall, E. L. C., Bird, S. M. — Tue, 05 May 2009 12:02:53 PDT

Recent meta-analyses of drug-court studies recognized the poor methodological quality of the evaluations, with only a few being randomized. This article critiques the design of the randomized studies from a statistical perspective. Learning points are identified for future drug-court studies and are applicable to evaluations both of other specialist courts and of court-based interventions more generally. The specific issues covered are randomization, describing the intervention, and baseline characteristics; study outcomes, and sample size calculations; in-program and postprogram behavior, analysis plan, and presentation of results.

Old Wine in New Skins: The Sensitivity of Established Findings to New Methods

Foster, E. M., Wiley-Exley, E., Bickman, L. — Tue, 05 May 2009 12:02:53 PDT

Findings from an evaluation of a model system for delivering mental health services to youth were reassessed to determine the robustness of key findings to the use of methodologies unavailable to the original analysts. These analyses address a key concern about earlier findings—that the quasi-experimental design involved the comparison of two noncomparable groups. The authors employed propensity score methodology to reconsider between-group baseline differences in observed characteristics of participating families. The authors also considered the possible effect of unobserved between-group differences. The data support previous studies that show few differences in outcomes, but the findings are sensitive to unobserved heterogeneity.