In probability and statistics, the fact that sometimes a gross calculation may result in a misleading result is a very well known phenomenon called Simpson's Paradox 1. One of the best practical and real life examples of Simpson’s Paradox occurred in a sex bias case against the University of California, Berkeley in 1973. The gross analysis of graduate admission data to UC Berkeley showed that the graduate admission rate for men was about 44%, while the admission rate for women was only 35%. However, when the admission rates were adjusted by the number of men and women applying to each department and also by the number of students who were being admitted to each department, the results showed that most departments actually admitted women at a higher rate and overall there was no difference between admission rates of men and women 2.
Is it possible that the Simpson Paradox might show up in an OFCCP Desk Audit? During Desk Audit, OFCCP may ask to receive a copy of all applications that the contractor has received during a specific time period and for a specific job group. For example, OFCCP asks to receive all of the applications that the contractor has received for the “service workers” job groups. If applicants’ demographic information (gender and ethnicity information) is not kept with the applications, OFCCP asks to receive the demographic information separately. In addition to the applications, OFCCP also asks for the contractor’s hire log for the same job group. Once OFCCP has all of these data elements, it can calculate the hiring impact ratios for all various protected groups. If any of the impact ratios turn out to be statistically significant, then the OFCCP concludes that the statistically significant impact ratio is an evidence of the contractor’s discriminatory hiring practice.
Let’s use a simple hypothetical example to show how Simpson’s Paradox might occur in an OFCCP Desk Audit. Suppose OFCCP, during a Desk Audit, asks a contractor in the hospitality industry to turn over all of the applications the contractor has received for the “Service Workers” job group. Suppose further that the contractor turns over 600 applications, 300 from female applicants and 300 from male applicants. Since most of the jobs in this job group are entry level jobs and usually do not require special skills or certifications, OFCCP might assume that all 600 applicants are equally qualified. Therefore, since 300 out of 600 applicants or 50% are women, then OFCCP will expect to see 50% of the hires to go to female applicants.
Assume that the contractor’s hire logs show 42 women and 98 men were hired into service workers positions. This means that 42 women out of 300 female applicants were hired; therefore, the hire rate for women was 14%. Similarly, hiring of 98 men out of 300 male applicants means that the hire rate for men was 33%. The difference of almost 19% between hiring rates of women and men is a statistically significant difference and the impact ratio of hiring is 43% (14% / 33%). 3
OFCCP was expecting to see 50% of hires to go to women since half of applicants were women, but only 42 out of 140 hires or 30% went to women. Given this difference between hiring rates of men and women, OFCCP will determine that such a large difference indicates that the hiring practice of the contractor is biased against women. Chart -1 below summarizes the applicant and hire data for Service Workers job group.
Suppose further investigation reveals that within service workers job group there are actually two different job categories. The first job category is Hosts and Hostesses 4 and the second job category is Dishwasher 5. Investigation also shows that majority of the women who filed an application with the contractor were actually interested in Hosts and Hostesses job category and the majority of men who filed an application were actually interested in Dishwasher job category.
A quick look at the census data reveals that nationwide about 260,000 employees are employed as Hosts and Hostesses, and 212,000 of those Hosts and Hostesses are women. This means almost 82% of all Hosts and Hostesses are women. There are also about 270,000 employees who are employed as Dishwashers, and 50,000 of those Dishwashers are women. This means almost 19% of Dishwashers are women 6.
In our hypothetical example, let’s assume each of the job categories within the Service Workers job group received 300 applications. Among 300 who applied to the Hosts and Hostesses job category, 240 or 80% were female and 60 applicants were male. Hire logs indicate that 17 women and 3 men were hired into the Hosts and Hostesses job category. Therefore the hiring rate into the Hostesses job for women was actually 7% (or 17/240) and the hiring rate for men was 5% (or 3/60). Women were hired at a higher rate than men into the Hostesses job category.
For the Dishwashers job category, let’s assume that among 300 who applied to the Dishwashers job category, 60 or 20% were female and 240 applicants were male. Hire logs indicate that 25 women and 95 men were hired into the Dishwashers job category. Therefore, the hiring rate into the Dishwashers job category for women was actually 42% (or 25/60) and the hiring rate for men was 40% (or 95/240). Again, women were hired at a higher rate than men into the Dishwashers job category.
It appears that women were hired into both job categories at higher rates than men, so the question is why did OFCCP’s original gross analysis find a significantly lower hiring rate for women? It turns out that the assumption of 50% female availability for both jobs across Service Workers job group is not correct. To aggregate the results across both job categories properly, one has to take into account the fact that female availability was different across two job categories (80% for Hosts and Hostesses vs. 20% for Dishwashers). Also, the fact that one job had more hires than the other (only 20 hires into Hosts and Hostesses vs. 120 hires into Dishwashers) is an important factor and it needs to be taken into account when we compare hiring rates across different jobs. In our example, if female availability is properly calculated, then the aggregate hiring rates across two jobs exhibits a slightly higher hiring rate for women.
This example illustrates that calculation of gross hiring rates and impact ratio might produce results that are misleading. The example also indicates that in order to detect and explain the problems with the gross calculation, the contractor will need to use detailed information on the hiring process. For example, it is very important that the contractor tracks and preserves the expression of interest of all applicants for a specific position. It is also important to note that the hiring process is a dynamic process. Some applicants might express an interest in some specific jobs at one point, but at a later date they might change their mind and remove themselves from further consideration. The contractor will need to preserve the records of these voluntary decisions. In short, the contractor has to keep a dynamic database which reflects all of the information, and also all of the changes to that information, that hiring authorities use during the application evaluation and subsequently in all of the stages of the selection process.
1. [Named after Edward Simpson who described this phenomenon in a technical statistical article in 1951, please see: Simpson, Edward H. (1951). "The Interpretation of Interaction in Contingency Tables". Journal of the Royal Statistical Society, Ser. B 13: 238–241] ↩
2. [For a more complete explanation please see: Bickel, E.A. Hammel and J.W. O'Connell (1975). "Sex Bias in Graduate Admissions: Data From Berkeley". Science 187 (4175): 398–404.] ↩
3. [Differences between ratios and percentages here and elsewhere are due to rounding.] ↩
4. [Census code for “Hosts and Hostesses” job category is 415 and Standard Occupational Code (SOC) for this job is 35-9031.] ↩
5. [code for “Dishwasher” job category is 414 and Standard Occupational Code (SOC) for this job is 35-9021.] ↩
6. [Bureau of Labor Statistics, Current Population Survey (CPS), Table: Household Annual Averages; Employed persons by detailed occupation, sex, race, and Hispanic or Latino ethnicity, 2012.] ↩