Evaluation Review recent issues

Using Tax Parcels to Select a Location-Based Sample: An Illustration That Examines Residents' Awareness of Sex Offenders in Neighborhoods

Craun, S. W., Freisthler, B. — 2008-06-30

Social science research is increasingly considering place when examining social programs and policies with a spatial component. A specific research challenge involving spatial policies is how to select a sample of individuals based on their geographic locations. This article illustrates the use of geographic information systems, tax parcels, and mail surveys to target residents in varied geographic areas. A provided example demonstrates how researchers obtained a sample of respondents living within one tenth of a mile of multiple registered sex offenders. The challenges of using tax parcels to obtain addresses for apartments and mobile home parks are also explored.

Active Parental Consent in School-Based Research: How Much Is Enough and How Do We Get It?

Esbensen, F.-A., Melde, C., Taylor, T. J., Peterson, D. — 2008-06-30

Active parental consent policies have been blamed for low participation rates and selection bias (i.e., loss of "high-risk" youths) in school-based studies. In this article, the authors describe active consent procedures that produced an overall active consent rate of 79% in a sample of more than 4,500 middle school students attending 29 schools in seven cities across the United States. Consent rates, however, varied considerably both within and between schools. To better understand factors associated with active parental consent rates, the authors examined district-level, school-level, and teacher-specific effects on consent rates.

What Respondents Really Expect From Researchers

Kolar, T., Kolar, I. — 2008-06-30

This article addresses the issue of falling response rates in telephone surveys. To better understand and maintain respondent goodwill, concepts of psychological contract and respondent expectations are introduced and explored. Results of the qualitative study show that respondent expectations are not only socially contingent but also ego-expressive, utilitarian, pleasurable, and epistemic by nature. Although results are reassuring in terms of commercialization of the psychological contract, they indicate some radical changes that are needed for the respondents to accept its continuation. The article discusses several practical and theoretical implications of such changes and suggests a series of corresponding propositions aimed at facilitating and inspiring future developments in this field.

Weighting Regressions by Propensity Scores

Freedman, D. A., Berk, R. A. — 2008-06-30

Regressions can be weighted by propensity scores in order to reduce bias. However, weighting is likely to increase random error in the estimates, and to bias the estimated standard errors downward, even when selection mechanisms are well understood. Moreover, in some cases, weighting will increase the bias in estimated causal parameters. If investigators have a good causal model, it seems better just to fit the model without weights. If the causal model is improperly specified, there can be significant problems in retrieving the situation by weighting, although weighting may help under some circumstances.

Erratum

2008-06-30

Mental Health and Firearms in Community-Based Surveys: Implications for Suicide Prevention

Sorenson, S. B., Vittes, K. A. — 2008-05-02

Suicide rates are higher among those who own or live in a household with a hand gun. This article examines the association between hand gun ownership and mental health, another risk factor for suicide. Data from the General Social Survey, a series of surveys of U.S. adults, are analyzed to compare general emotional and mental health, sadness and depression, functional mental health, and mental health help seeking among gun owners, persons who do not own but live in a household with a gun, and those who do not own a gun. After taking into account a few basic demographic characteristics associated with both variables, there appears to be no association between mental health and gun ownership. Nor is there any association between mental health and living in a household with a firearm. Findings suggest that the high risk of suicide among those who own or live in a household with a gun is not related to poor mental health. Implications for prevention are discussed.

Rates of Missing Responses in Personal Digital Assistant (PDA) Versus Paper Assessments

2008-05-02

This article describes rates of missing item responses in personal digital assistant (PDA) assessments as compared to paper assessments. Data come from the evaluation of a classroom-based leisure, life skills, and sexuality education program delivered to high school students in Cape Town, South Africa. Analyses show that the paper assessments had much higher rates of missing-ness than PDA assessments. This association is moderated by item order. Certain analyses also suggest that paper assessments have higher rates of missingness for items pertaining to participants' sexual behavior. Implications of these results for evaluation research will be discussed.

Tightening Income Documentation in a Means-Tested Program: Who Stays Away?

Gleason, P., Burghardt, J., Strasberg, P., Hulsey, L. — 2008-05-02

Programs using means tests to identify low-income households face a trade-off between promoting access and ensuring program integrity. The authors use a comparison-district design to estimate the effects of a pilot program to improve the accuracy of the process of certifying students for free or reduced-price meals in the National School Lunch Program. This pilot program required households to provide income documentation with their applications for these benefits. Requiring income documentation did not reduce the proportion of ineligible households getting free or reduced-price meals. Furthermore, this requirement did reduce access to the program among eligible households.

The Impact of Child Obesity on Active Parental Consent in School-Based Survey Research on Healthy Eating and Physical Activity

Mellor, J. M., Rapoport, R. B., Maliniak, D. — 2008-05-02

Previous studies have shown that active consent procedures result in sampling bias in surveys dealing with adolescent risk behaviors such as cigarette smoking and illicit drug use. To examine sampling bias from active consent procedures when the survey topic pertains to childhood obesity and associated health behaviors, the authors pair data obtained from both active and passive consent procedures. The authors find that parents of children who are overweight or at risk for being overweight are significantly less likely to give active consent. In addition, parents of children enrolled in lower grades are more reluctant to consent to participate.

Using an Empirical Binomial Hierarchical Bayesian Model as an Alternative to Analyzing Data From Multisite Studies

Hardin, J. M., Anderson, B. S., Woodby, L. L., Crawford, M. A., Russell, T. V. — 2008-03-04

This article explores the statistical methodologies used in demonstration and effectiveness studies when the treatments are applied across multiple settings. The importance of evaluating and how to evaluate these types of studies are discussed. As an alternative to standard methodology, the authors of this article offer an empirical binomial hierarchical Bayesian model as a way to effectively evaluate multisite studies. An application of using the Bayesian model in a real-world multisite study is given.

The Truncation-by-Death Problem: What To Do in an Experimental Evaluation When the Outcome Is Not Always Defined

McConnell, S., Stuart, E. A., Devaney, B. — 2008-03-04

Although experiments are viewed as the gold standard for evaluation, some of their benefits may be lost when, as is common, outcomes are not defined for some sample members. In evaluations of marriage interventions, for example, a key outcome—relationship quality—is undefined when a couple splits up. This article shows how treatment-control differences in mean outcomes can be misleading when outcomes are not defined for everyone and discusses ways to identify the seriousness of the problem. Potential solutions to the problem are described, including approaches that rely on simple treatment-control differences-in-means as well as more complex modeling approaches.

Mixed Results in a Transitional Planning Program for Alternative School Students

Wolf, E. M., Wolf, D. A. — 2008-03-04

Disciplinary alternative schools have a reputation as gateways to the juvenile and criminal justice systems. The authors conducted an evaluation of an intervention (Strategies for Success) designed to divert seventh-, eighth-, and ninth-grade alternative school students from this gateway. They used propensity score matching and a multivariate random effects model to estimate program impacts and found that the program not only increased attendance rates, at least in the short term, but also increased the likelihood of reassignment to alternative schools. The discussion focuses on possible reasons and solutions for high rates of return to alternative school and for the erosion of program effects.

Designing a Household Survey to Address Seasonality in Child Care Arrangements

Schmidt, S. R., Wang, K. H., Sonenstein, F. L. — 2008-03-04

In household telephone surveys, a long field period may be required to maximize the response rate and achieve adequate sample sizes. However, long field periods can be problematic when measures of seasonally affected behavior are sought. Surveys of child care use are one example because child care arrangements vary by season. Options include varying the questions posed about school-year and summer arrangements or posing retrospective questions about child care use for the school year only. This article evaluates the bias associated with the use of retrospective questions about school-year child care arrangements in the 1999 National Survey of America's Families. The authors find little evidence of bias and hence recommend that future surveys use the retrospective approach.

Longitudinal Research That Can Inform Dynamic Models for the Treatment of Addiction as a Disease

Hilton, T. F., Chandler, R. K., Compton, W. M. — 2008-01-14

The Interaction of Co-Occurring Mental Disorders and Recovery Management Checkups on Substance Abuse Treatment Participation and Recovery

Rush, B. R., Dennis, M. L., Scott, C. K., Castel, S., Funk, R. R. — 2008-01-14

This article examines the effectiveness of quarterly Recovery Management Checkups (RMCs) for people with substance disorders by level of co-occurring mental disorders (34% none, 27% internalizing disorders, and 39% internalizing and externalizing) across two randomized experiments with 92% to 97% follow-up. The 865 participants are 82% African American, 53% female, and age 37 on average. RMC involves identification of those in need of treatment, motivational interviews, and treatment linkage assistance. It is effective in linking participants in need to treatment, with equal or better outcomes among those with more mental disorders. The data support the utility of monitoring and re-intervention for clients with co-occurring disorders.

Nine-Year Psychiatric Trajectories and Substance Use Outcomes: An Application of the Group-Based Modeling Approach

Chi, F. W., Weisner, C. M. — 2008-01-14

This study identifies longitudinal psychiatric trajectories of 934 adult individuals entering chemical dependency treatment in a private, managed care health plan and examines the relationship of these trajectories with substance use (SU) outcomes. The authors apply a group-based modeling approach to identify trajectory groups based on repeated measures of psychiatric severity for 9 years and identify four distinct groups. Results of multivariate logistic generalized estimating equation models find an association between psychiatric trajectories and long-term SU. Older cohorts and life course measures of marital status and employment status as individuals changed over time are related to drug and some alcohol outcomes.

Patterns of Crime and Drug Use Trajectories in Relation to Treatment Initiation and 5-Year Outcomes: An Application of Growth Mixture Modeling Across Three Data Sets

Prendergast, M., Huang, D., Hser, Y.-I. — 2008-01-28

Drug abusers vary considerably in their drug use and criminal behavior over time, and these trajectories are likely to influence drug treatment participation and treatment outcomes. Drawing on longitudinal natural history data from three samples of adult male drug users, we identify four groups with distinctive drug use and crime trajectories during the 5 years prior to their first treatment episode. The groups' characteristics of initial treatment are compared. The trajectory groups are then included in Poisson growth curve models to predict drug use, incarceration, and employment during the 5 years following first treatment. Findings indicate that posttreatment drug use decreased and posttreatment employment increased. There was little change in posttreatment incarceration. Posttreatment trajectories for drug use, incarceration, and employment were significantly different across the four trajectory groups.

Longitudinal HIV Risk Behavior Among the Drug Abuse Treatment Outcome Studies (DATOS) Adult Sample

Murphy, D. A., Brecht, M.-L., Herbeck, D., Evans, E., Huang, D., Hser, Y.-I. — 2008-01-14

Longitudinal trajectories for HIV risk were examined over 5 years following treatment among 1,393 patients who participated in the nationwide Drug Abuse Treatment Outcome Studies. Both injection drug use and sexual risk behavior declined over time, with most of the decline occurring between intake and the first-year follow-up. However, results of the application of growth mixture models for both sets of trajectories indicated that a subgroup of individuals reverted to a high-risk behavior over time, with a higher level of risk at the 5-year follow-up than their original risk level at intake. Of clients who were engaged in regular injection drug use at intake, 76% continued to inject drug at a moderate—stable or increased rate during the 5-year follow-up.

Gender Similarities and Differences in the Treatment, Relapse, and Recovery Cycle

Grella, C. E., Scott, C. K., Foss, M. A., Dennis, M. L. — 2008-01-14

This study explores the influence of gender on changes in recovery status among participants in a longitudinal study. The study sample (N = 1,202; 60% female) is recruited on referral to treatment, and annual interviews are conducted from Years 2 to 6 following intake. At each annual observation, participants are classified into one of four statuses (recovery, treatment, incarcerated, and using), and the transitional probabilities and correlates of transitioning from one status to another are estimated. About 80% of the participants changed status at least once over the follow-up period. Women are one third less likely to transition from recovery to using; the predictors of transitioning to different statuses vary by gender. The implications of gender as a moderator of the recovery process are discussed.