The people behind the Poverty Insights website have been following the New York Department of Homeless Services study of the city’s homeless services programs. The Department is under fire for performing a randomized controlled trial with a sampling of the families, all seekers of the DHS services, in two groups of 200 families each. The first group experienced no change in services, but the second grouping was simply given a list of nonprofits that engage in similar social services. The second group was then told they could receive no assistance directly from DHS for the duration of the study. The study is to last for two years.
Now, measuring effectiveness is vital, as it is something that is all too often neglected in the social arena. Still, this denial of services is a troubling thing. David Henderson, a blogger for Change.org and an editor for Poverty Insights, noted the following in his post on the subject from October 11:
DHS had the means to provide their service to all who demanded it, but chose not to for the sake of their study. While this style of research is typical in the medical world, where one group of patients is administered an experimental drug then compared to another group that did not receive treatment, it’s not clear that such an approach is appropriate in social sector evaluation.
This is an important issue for the sector to consider. Where do we draw the line in evaluation?
Where indeed? Michael Gechter, co-founder of Idealistics, Inc., comments on the Homebase program being evaluated in the study in his Poverty Insights post from October 19:
I’m willing to doubt that HomeBase’s mediation, budgeting, job training, and advocacy services have an effect on preventing families from falling into homelessness. Mediation may not prevent eviction, budgeting and job training may come too late, and benefits advocacy might be too little to stop a family from becoming homeless. But there is no doubt that emergency rental assistance prevents homelessness, at least temporarily, if it stops a family from being evicted.
New York City’s Mayor’s Management Report [PDF] suggests that the temporary respite offered by HomeBase may have a permanent effect: 94.6% of the families receiving preventative services in fiscal year 2010 did not enter the shelter system. This is not a rigorous evaluation by any means: we don’t know how many of those families would have entered the shelter system if they didn’t have help from HomeBase.
Once again, we come to the concept of a hand up rather than a hand out. While the data is, as stated, far from a solid study, it still points in an interesting direction — a direction that supports our own stance on preventative and rehabilitative programs. Gechter then goes on to raise a very important question about the study:
In the language of randomized control trials, there is still some doubt as to whether the treatment works. But it certainly raises the question: do we need an evaluation? Are we willing to risk the lives of 200 families to evaluate a service that, at worst, temporarily prevents homelessness?
Mary McLaughlin, Ph.D., a president of Emotional Education Services, LLC, then followed up with a brilliantly detailed examination of the ethics of the study, in which she raised additional questions. A few of them include:
- Given that the rights of children have historically been vigorously protected by IRB’s, was the study protocol reviewed by an IRB at the City University of New York, the institution charged with oversight of this investigation?
- Why is this $530,000 investigation being conducted at all given that the HomeBase program is already known to be highly effective as reported in the Mayor’s 2010 Management Report?
- How does the city reconcile its denial of services to two hundred families for two years with its record of repeated losses and settlements in lawsuits against the city and state of New York that insure the right to shelter for homeless men, women, children and families in New York City and state beginning in 1981, pursuant to Callahan v Carey, and culminating as recently as 2008 with a court judgment in Boston v The City of New York which reaffirmed the right to shelter for homeless families with children?
- Is the City prepared to pay damages awards including the possibility of punitive damages to study participants who subsequently sue?
There is much, much more. You can read it all in Dr. McLaughlin’s post on Poverty Insights. I would highly advise it even for those outside of the NYC area as it is a great way to get an idea of how complex the underpinnings of the situation are. The way in which data is collected and interpreted needs to be neutral and transparent in order to maintain its veracity. Embracing data-collection practices that do potential harm to those studied is not the proper approach to take.
My hat is off to the team at Poverty Insights: This is an excellent series of posts that illuminate the murky underbelly of our system. Please visit their site and leave them a comment. Let them know we all appreciate their efforts on this vital issue.
Source: “New York Department of Homeless Services Study Violates Research Ethics Principles,” Poverty Insights, 10/28/10
Source: “New York Department of Homeless Services Denies Two-Hundred Families Assistance in Name of Research,” Poverty Insights, 10/11/10
Source: “When Testing Hurts: Why the New York City Department of Homeless Services is Wrong,” Poverty Insights, 10/19/10
Image by aprilzosia, used under its Creative Commons license.
Visit Us on Facebook: Humane Exposures Publishing, downTownUSA, Maggots in My Sweet Potatoes, It’s More Expensive To Do Nothing.
