Meeting Basic Needs
Center and Colleagues Stop New York City from Unlawfully Preventing Access to Benefits
New York City Admits Monitoring Insufficient to Establish Proper Processing of Applications for Food Stamps, Medicaid, and Cash Assistance: Court Bars Opening of More Job Centers
From September, 1999 Welfare News
Introduction. In another victory for plaintiffs who have so far succeeded in their claims that New York City Job Centers illegally discourage people from applying for Food Stamps, Medicaid, and cash assistance, a federal court judge has recently denied New York City permission to continue to convert welfare centers to Job Centers and refused to set aside its January 1999 sweeping order requiring the City to comply with federal and state law. The City’s request to the court came after it had implemented far-reaching policy changes in how it processes applications for Food Stamps, Medicaid, and cash assistance and after it was forced to admit that its monitoring was not sufficient to establish that the new policies actually resulted in better processing of Food Stamps, Medicaid, and cash assistance applications.
The court’s July 26, 1999 denial of the City’s request in the case, Reynolds v. Giuliani, was significant for two reasons. First, the City will now be forced to develop monitoring that will provide a truly accurate portrayal of its performance. Second, the court’s bar on further conversions of welfare centers to Job Centers remains intact. Plaintiffs’ representatives have observed that welfare centers do not appear to engage in deterrence, lawful or otherwise, and, therefore, needy individuals and families will still have a number of traditional welfare centers at which to apply for benefits.
This article discusses many of the evidentiary issues raised by the fight over the quality of the City’s monitoring efforts. In addition, it discusses the role of the plaintiffs’ and the city’s expert witnesses in addressing questions concerning the sufficiency of those monitoring efforts.
Background. In December 1998, the Welfare Law Center, the New York Legal Aid Society and two other local legal services providers filed Reynolds v. Giuliani, a federal class action challenge to the City’s failure to comply with federal and state law requirements governing the Food Stamps, Medicaid, and cash assistance application process. Plaintiffs alleged that the City’s new Job Centers, which replace welfare centers, illegally discourage and deter needy people from applying. The Court agreed with the plaintiffs and on January 25, 1999, entered a sweeping preliminary injunction which barred the City from converting any more welfare offices to Job Centers, mandated compliance with applicable federal and state statutory and regulatory provisions governing applications, and ordered the City to develop a corrective action plan.
In May 1999, the court approved the City’s corrective action plan along with revised notices, signs, training materials, and instructional memoranda. While the Court found that the City’s corrective action plan represented a significant effort to change how it processes applications, it remained concerned that the City had not demonstrated that the plan was being implemented in a manner that would result in changes in practice. In addition, the Court agreed that plaintiffs had raised serious questions going to the validity of the City’s monitoring efforts. It scheduled a further hearing to resolve questions concerning the sufficiency of the monitoring and granted the parties the opportunity to develop evi-
dence on this issue. For more extensive background, see the December 1998, February and April 1999 issues of Welfare News.
The City’s Monitoring Plan and Plaintiffs’ Initial Objections. Last spring the City defendants indicated to the court that they would use auditing reports to show that their corrective action plan was working. These auditing reports purportedly track the City’s ability to process applications for expedited Food Stamps and emergency cash assistance grants and to process separately applications for Medicaid and Food Stamps when the cash assistance application is either withdrawn or denied. Plaintiffs objected for several reasons. First, plaintiffs’ counsel contended that the audits are flawed and do not accurately measure the City’s performance. Second, they argued that the audits do not monitor key aspects of the process that led to the filing of the lawsuit. For example, the audits do not measure at all whether applicants are unlawfully discouraged from filing applications for cash assistance, Food Stamps, and/or Medicaid. Finally, plaintiffs’ counsel urged that even if the auditing were sufficient to measure the City defendants’ performance, the level at which the City was performing warranted both maintaining the stay and continuing the mandatory injunctive relief.
City’s Auditing - Too little/too flawed. City defendants believed that they could demonstrate that they were in compliance with the Court’s order as well as federal Food Stamp and Medicaid application processing requirements through the use of "random" audits. The City’s plan was to analyze two cases per worker per center each month and to assess how well the workers 1) processed the application for expedited Food Stamps and emergency cash assistance, and 2) made separate determinations for Food Stamps and Medicaid when the application for cash is either withdrawn or denied. The City began collecting data in February 1999 for January 1999 applications. The City’s reports revealed an improvement in their performance with many more cases receiving expedited Food Stamps and emergency cash assistance and more cases receiving separate determinations of entitlement to Food Stamps and Medicaid.
However, as discussed more fully below, the plaintiffs believed that City’s auditing was not sufficient to provide a true picture of the City’s performance for several reasons, including the following: 1) the sample sizes are far too small; 2) the sampling methodology was not designed to produce an unbiased sample; and 3) the auditors did not follow even the flawed auditing guidelines developed by the City. As a result, plaintiffs urged that while the auditing results suggest an improvement, it is impossible to assess fully the extent of the improvement.
The Battle of the Experts. The City defendants determined to proceed to a fact-finding hearing on the sufficiency of their auditing efforts. They believed that the Court would determine that their auditing techniques were likely to demonstrate considerable improvement. To that end, they retained the services of well-credentialed expert, Dr. June O’Neill, a former Director of the Congressional Budget Office for four years. Dr. O’Neill teaches economics at the City University of New York and is frequently quoted in conservative publications on social welfare issues. Plaintiffs’ counsel secured the services of Richard Faust as their expert. Mr. Faust has had considerable experience analyzing monitoring efforts engaged in by various New York City agencies and has helped to develop monitoring tools in two prior welfare class actions. Wendell Primus of the Center on Budget and Policy Priorities also provided extensive advise to plaintiffs’ counsel.
Consistent with the schedule imposed by the Court, the City defendant’s expert first submitted her report, in which she contended that the City’s monitoring reports indicate improvement in performance. Plaintiffs were then entitled to depose her (that is, to question her in a proceeding called a deposition) and their expert then drafted and submitted his report.
Most public interest counsel do not have much experience working with monitoring experts, and a very experienced expert will assist counsel in making the best possible use of his or her (the expert’s) skills. In Reynolds, the schedule set by the court which had the City defendants’ expert presenting her report first was important because, inevitably, the plaintiffs did not have access to the full range of information relied on by the defendants’ expert. In addition, the plaintiffs’ ability to depose the city’s expert before the plaintiffs’ expert submitted his report allowed plaintiffs to expose the flaws in the City expert’s evaluation of the sufficiency of the auditing.
The deposition of the City expert revealed that her area of focus had been on the statistical validity of the City’s auditing and not on the adequacy of the data collection methodology. This difference is important because the data must be collected pursuant to unbiased means and must be collected in a sufficiently large sample to provide useful information about the defendants’ performance. The City’s expert uncritically accepted the City’s representations that the data was collected pursuant to an acceptable methodology. It was only with probing during the depositions of the City officials responsible for conducting the audits that plaintiffs were able to expose the flaws in the City’s monitoring.
City Monitoring Was Flawed in Three Major Ways. At the outset, it was obvious that the City was using too small a sample to assess accurately its performance. Specifically, the City was studying approximately 25 cases per Job Center and approximately 350 per month across all Job Centers. While this number may have been sufficient to create a large enough sample over the entire population of Job Centers, it would not be statistically significant to reliably reveal information as to any Job Center.
There is, of course, no magic number that defines the size of the sample. For example, surveys of potential voters in a presidential race are often conducted of samples as small as 1,400. Yet, a sample of only 1,400 will reveal reliable information about the voting preferences of the country, assuming the questions are framed well and the answers are honest. The issue is that the smaller the sample, the less reliable the information. Because auditing can be expensive and burdensome, there may be pressure to use smaller samples.
In this case, the two experts disagreed as to the smallest sample that would yield usable reliable data. However, both agreed that a sample of approximately 350 cases taken from the total population of applicants would be considerably more likely to yield reliable information than the 25 per Job Center. Advocates elsewhere should keep in mind that to judges who are not familiar with statistics, it will appear logical that larger samples are considerably more desirable than statistical sampling rules would necessitate. However, at some point, the degree of statistical reliability that results from increasing the sample size dramatically is not significant. For example, doubling a sample from 5,000 to 10,000 will not yield a doubling in the degree of reliability. Because of the number of variables that can be at play, the appropriate size of any sample in other cases should be discussed with an expert in statistics and sampling methodology.
Second, the City’s expert acknowledged that she did not study the case selection methodology design. This is important because the selection was not "random" in the statistical sense of the word. Instead, the City attempted to design a non-biased sampling methodology. However, they failed. The most important failure was the City’s emphasis on selecting two cases per worker. As a consequence, workers who see many more clients because they fail to take the time to process applications fully are treated just the same as a worker who sees fewer clients because she exhaustively reviews each application. In addition, the City failed to take into account the fact that different Job Centers serve differently sized populations. Some Job centers are "small," while one center in Queens has traditionally had a caseload larger than the number of recipients in many other states.
In this case, a better sampling methodology might have been to determine the appropriate sample size ("S") and to divide the total population of applicants ("P") by the sample size. This would result in an "N"th number or the number of applications to be counted to derive the sample size. For example, if the population of applicants of 6,000 per month and the desired sample is 500, then the "N"th number is 12. (6,000/500 = 12). Thus, the City defendants could count every 12th application towards the unbiased sample of 500. In Reynolds, the count would need to be performed manually from the daily logs of all applicants.
Third, depositions conducted of City officials and other evidence gathered revealed that the auditors did not even follow the flawed auditing procedures. Instead, plaintiffs’ counsel’s review of the more than 2,600 separate cases selected for auditing by City defendants revealed double-counting, skipped cases, and other significant problems in the case selection methodology. Errors in the case selection methodology increase the unreliability of the data and any conclusions being drawn from the audits. The City’s expert was not aware of the auditors’ failure to fully follow the auditing protocols. Plaintiffs’ expert was instrumental in guiding the plaintiffs in developing the tools needed to best analyze and interpret the City’s raw data.
Plaintiffs never lost sight of the fact that the audits, even if flawless, were too limited in scope. The City’s audits were not designed to assess whether persons were being inappropriately deterred from filing an application at the outset. If hundreds or thousands of applicants are unable to access the systems because they are unlawfully discouraged or denied the right to file an application, the Reynolds issues would not be resolved simply by a showing that expedited Food Stamps are processed timely. The City’s expert agreed that she had not studied the City’s monitoring for the purpose of forming an opinion as to whether the City could ascertain the extent of illegal discouragement and denials. However, plaintiffs’ expert did study summaries of City data as well as the underlying data collection instruments. He determined that not only could the City not determine the extent to which it was following the law, but that the methods for collecting and recording information differed greatly from center to center as well as over time.
Thus, because plaintiffs were not prepared to take the City expert’s testimony at face value, they determined that the scope of her testimony was very narrow and that she had not been provided sufficient information by the City to accurately assess the City’s compliance with the preliminary injunction and federal mandates. Evidently the City agreed because it told the court that its monitoring was not sufficient and is currently designing new monitoring protocols.
Advocates interested in further information or assistance in addressing similar issues in their own community should contact Marc Cohan and Rebecca Scharf of the Welfare Law Center, co-counsel for the Reynolds plaintiffs .
