Methods for Setting Cut Scores

Methods for Setting Cut Scores

Module Chapter 7 p655 wk3

C H A P T E R 7

Utility

In Ieveryday language, we use the term utility to refer to the usefulness of some thing or some process. In the language of psychometrics, utility (also referred to as test utility) means much the same thing; it refers to how useful a test is. More specifically, it refers to the practical value of using a test to aid in decision making. An overview of some frequently raised utility-related questions would include the following:

· How useful is this test in terms of cost efficiency?

· How useful is this test in terms of savings in time?

· What is the comparative utility of this test? That is, how useful is this test as compared to another test?

· What is the clinical utility of this test? That is, how useful is it for purposes of diagnostic assessment or treatment?

· What is the diagnostic utility of this neurological test? That is, how useful is it for classification purposes?

· How useful is this medical school admissions test used in assigning a limited number of openings to an overwhelming number of applicants?

· How useful is the addition of another test to the test battery already in use for screening purposes?

· How useful is this personnel test as a tool for the selection of new employees?

· Is this particular personnel test used for promoting middle-management employees more useful than using no test at all?

· Is the time and money it takes to administer, score, and interpret this personnel promotion test battery worth it as compared to simply asking the employee’s supervisor for a recommendation as to whether the employee should be promoted?

· How useful is the training program in place for new recruits?

· How effective is this particular clinical technique?

· Should this new intervention be used in place of an existing intervention?

Page 201

What Is Utility?

We may define utility in the context of testing and assessment as the usefulness or practical value of testing to improve efficiency. Note that in this definition, “testing” refers to anything from a single test to a large-scale testing program that employs a battery of tests. For simplicity and convenience, in this chapter we often refer to the utility of one individual test. Keep in mind, however, that such discussion is applicable and generalizable to the utility of large-scale testing programs that may employ many tests or test batteries. Utility is also used to refer to the usefulness or practical value of a training program or intervention. We may speak, for example, of the utility of adding a particular component to an existing corporate training program or clinical intervention. Throughout this chapter, however, our discussion and illustrations will focus primarily on utility as it relates to testing.

JUST THINK . . .

Based on everything that you have read about tests and testing so far in this book, how do you think you would go about making a judgment regarding the utility of a test?

If your response to our Just Think question about judging a test’s utility made reference to the reliability of a test or the validity of a test, then you are correct—well, partly. Judgments concerning the utility of a test are made on the basis of test reliability and validity data as well as on other data.

Factors That Affect a Test’s Utility

A number of considerations are involved in making a judgment about the utility of a test. Here we will review how a test’s psychometric soundness, costs, and benefits can all affect a judgment concerning a test’s utility.

Psychometric soundness

By psychometric soundness, we refer—as you probably know by now—to the reliability and validity of a test. A test is said to be psychometrically sound for a particular purpose if reliability and validity coefficients are acceptably high. How can an index of utility be distinguished from an index of reliability or validity? The short answer to that question is as follows: An index of reliability can tell us something about how consistently a test measures what it measures; and an index of validity can tell us something about whether a test measures what it purports to measure. But an index of utility can tell us something about the practical value of the information derived from scores on the test. Test scores are said to have utility if their use in a particular situation helps us to make better decisions—better, that is, in the sense of being more cost-effective (see, for example, Brettschneider et al., 2015; or Winser et al., 2015).

In previous chapters on reliability and validity, it was noted that reliability sets a ceiling on validity. It is tempting to draw the conclusion that a comparable relationship exists between validity and utility and conclude that “validity sets a ceiling on utility.” In many instances, such a conclusion would certainly be defensible. After all, a test must be valid to be useful. Of what practical value or usefulness is a test for a specific purpose if the test is not valid for that purpose?

Unfortunately, few things about utility theory and its application are simple and uncomplicated. Generally speaking, the higher the criterion-related validity of test scores for making a particular decision, the higher the utility of the test is likely to be. However, there are exceptions to this general rule. This is so because many factors may enter into an estimate of a test’s utility, and there are great variations in the ways in which the utility of a test is determined. In a study of the utility of a test used for personnel selection, for example, the selection ratio may be very high. We’ll review the concept of a selection ratio (introduced in the previous chapter) in greater detail later in this chapter. For now, let’s simply note that if the selection ratio is very high, most people who apply for the job are being hired. Under such circumstances, the validity of the test may have little to do with the test’s utility.Page 202

What about the other side of the coin? Would it be accurate to conclude that “a valid test is a useful test”? At first blush this statement may also seem perfectly logical and true. But once again—we’re talking about utility theory here, and this can be very complicated stuff—the answer is no; it is not the case that “a valid test is a useful test.” People often refer to a particular test as “valid” if scores on the test have been shown to be good indicators of how the person will score on the criterion.

An example from the published literature may help to further illustrate how a valid tool of assessment may have questionable utility. One way of monitoring the drug use of cocaine users being treated on an outpatient basis is through regular urine tests. As an alternative to that monitoring method, researchers developed a patch which, if worn day and night, could detect cocaine use through sweat. In a study designed to explore the utility of the sweat patch with 63 opiate-dependent volunteers who were seeking treatment, investigators found a 92% level of agreement between a positive urine test for cocaine and a positive test on the sweat patch for cocaine. On the face of it, these results would seem to be encouraging for the developers of the patch. However, this high rate of agreement occurred only when the patch had been untampered with and properly applied by research participants—which, as it turned out, wasn’t all that often. Overall, the researchers felt compelled to conclude that the sweat patch had limited utility as a means of monitoring drug use in outpatient treatment facilities (Chawarski et al., 2007). This study illustrates that even though a test may be psychometrically sound, it may have little utility—particularly if the targeted testtakers demonstrate a tendency to “bend, fold, spindle, mutilate, destroy, tamper with,” or otherwise fail to scrupulously follow the test’s directions.

Another utility-related factor does not necessarily have anything to do with the behavior of targeted testtakers. In fact, it typically has more to do with the behavior of the test’s targeted users.

Costs

Mention the word costs and what comes to mind? Usually words like money or dollars. In considerations of test utility, factors variously referred to as economic, financial, or budget-related in nature must certainly be taken into account. In fact, one of the most basic elements in any utility analysis is the financial cost of the selection device (or training program or clinical intervention) under study. However, the meaning of “cost” as applied to test utility can extend far beyond dollars and cents (see Figure 7–1). Briefly, cost in the context of test utility refers to disadvantages, losses, or expenses in both economic and noneconomic terms.

Figure 7–1 Rethinking the “Costs” of Testing—and of Not Testing The cost of this X-ray might be $100 or so . . . but what is the cost of not having this diagnostic procedure done? Depending on the particular case, the cost of not testing might be unnecessary pain and suffering, lifelong disability, or worse. In sum, the decision to test or not must be made with thoughtful consideration of all possible pros and cons, financial and otherwise.© Martin Barraud/age fotostock RF

As used with respect to test utility decisions, the term costs can be interpreted in the traditional, economic sense; that is, relating to expenditures associated with testing or not testing. If testing is to be conducted, then it may be necessary to allocate funds to purchase (1) a particular test, (2) a supply of blank test protocols, and (3) computerized test processing, scoring, and interpretation from the test publisher or some independent service. AssociatedPage 203 costs of testing may come in the form of (1) payment to professional personnel and staff associated with test administration, scoring, and interpretation, (2) facility rental, mortgage, and/or other charges related to the usage of the test facility, and (3) insurance, legal, accounting, licensing, and other routine costs of doing business. In some settings, such as private clinics, these costs may be offset by revenue, such as fees paid by testtakers. In other settings, such as research organizations, these costs will be paid from the test user’s funds, which may in turn derive from sources such as private donations or government grants.

The economic costs listed here are the easy ones to calculate. Not so easy to calculate are other economic costs, particularly those associated with not testing or testing with an instrument that turns out to be ineffective. As an admittedly far-fetched example, what if skyrocketing fuel costs prompted a commercial airline to institute cost-cutting methods?1 What if one of the cost-cutting methods the airline instituted was the cessation of its personnel assessment program? Now, all personnel—-including pilots and equipment repair personnel—would be hired and trained with little or no evaluation. Alternatively, what if the airline simply converted its current hiring and training program to a much less expensive program with much less rigorous (and perhaps ineffective) testing for all personnel? What economic (and noneconomic) consequences do you envision might result from such action? Would cost-cutting actions such as those described previously be prudent from a business perspective?

One need not hold an M.B.A. or an advanced degree in consumer psychology to understand that such actions on the part of the airline would probably not be effective. The resulting cost savings from elimination of such assessment programs would pale in comparison to the probable losses in customer revenue once word got out about the airline’s strategy for cost cutting; loss of public confidence in the safety of the airline would almost certainly translate into a loss of ticket sales. Additionally, such revenue losses would be irrevocably compounded by any safety-related incidents (with their attendant lawsuits) that occurred as a consequence of such imprudent cost cutting.

In this example, mention of the variable of “loss of confidence” brings us to another meaning of “costs” in terms of utility analyses; that is, costs in terms of loss. Noneconomic costs of drastic cost cutting by the airline might come in the form of harm or injury to airline passengers and crew as a result of incompetent pilots flying the plane and incompetent ground crews servicing the planes. Although people (and most notably insurance companies) do place dollar amounts on the loss of life and limb, for our purposes we can still categorize such tragic losses as noneconomic in nature.

JUST THINK . . .

How would you describe the non-economic cost of a nation’s armed forces using ineffective screening mechanisms to screen military recruits?

Other noneconomic costs of testing can be far more subtle. Consider, for example, a published study that examined the utility of taking four X-ray pictures as compared to two X-ray pictures in routine screening for fractured ribs among potential child abuse victims. Hansen et al. (2008) found that a four-view series of X-rays differed significantly from the more traditional, two-view series in terms of the number of fractures identified. These researchers recommended the addition of two more views in the routine X-ray protocols for possible physical abuse. Stated another way, these authors found diagnostic utility in adding two X-ray views to the more traditional protocol. The financial cost of using the two additional X-rays was seen as worth it, given the consequences and potential costs of failing to diagnose the injuries. Here, the (non-economic) cost concerns the risk of letting a potential child abuser continue to abuse a child without detection. In other medical research, such as that described by our featured assessment professional, the utility of various other tests and procedures are routinely evaluatedPage 204 (see this chapter’s Meet an Assessment Professional ).

MEET AN ASSESSMENT PROFESSIONAL

Meet Dr. Delphine Courvoisier

My name is Delphine Courvoisier. I hold a Ph.D. in psychometrics from the University of Geneva, Switzerland, and Master’s degrees in statistics from the University of Geneva, in epidemiology from Harvard School of Public Health, and in human resources from the University of Geneva. I currently work as a biostatistician in the Department of Rheumatology, at the University Hospitals of Geneva, Switzerland. A typical work day for me entails consulting with clinicians about their research projects. Assistance from me may be sought at any stage in a research project. So, for example, I might help out one team of researchers in conceptualizing initial hypotheses. Another research team might require assistance in selecting the most appropriate outcome measures, given the population of subjects with whom they are working. Yet another team might request assistance with data analysis or interpretation. In addition to all of that, a work day typically includes providing a colleague with some technical or social support—this to counter the concern or discouragement that may have been engendered by some methodological or statistical complexity inherent in a project that they are working on.

Rheumatoid arthritis is a chronic disease. Patients with this disease frequently suffer pain and may have limited functioning. Among other variables, research team members may focus their attention on quality-of-life issues for members of this population. Quality-of-life research may be conducted at different points in time through the course of the disease. In conducting the research, various tools of assessment, including psychological tests and structured interviews, may be used.

The focus of my own research team has been on several overlapping variables, including health-related quality of life, degree of functional disability, and disease activity and progression. We measure health-related quality of life using the Short-Form 36 Health Survey (SF36). We measure functional disability by means of the Health Assessment Questionnaire (HAQ). We assess disease activity and progression by means of a structured interview conducted by a health-care professional. The interview yields a proprietary disease activity score (DAS). All these data are then employed to evaluate the effectiveness of various treatment regimens, and adjust, where necessary, patient treatment plans.

Delphine Courvoisier, Ph.D., Psychometrician and biostatistician at the Department of Rheumatology at the University Hospitals of Geneva, Switzerland. © Delphine Courvoisier

Since so much of our work involves evaluation by means of tests or other assessment procedures, it is important to examine the utility of the methods we use. For example, when a research project demands that subjects respond to a series of telephone calls, it would be instructive to understand how compliance (or, answering the phone and responding to the experimenter’s questions) versus non-compliance (or, not answering the phone) affects the other variables under study. It may be, for example, that people who are more compliant are simply more conscientious. If that was indeed the case, all the data collected from people who answered the phone might be more causally related to a personality variable (such as conscientiousness) than anything else. Thus, prior to analyzing content of phone interviews, it would be useful to test—and reject—the hypothesis that only patients high on the personality trait of conscientiousness will answer the phone.

We conducted a study that entailed the administration of a personality test (the NEO Personality Inventory-Revised), as well as ecological momentary assessment (EMA) in the form of a series of phone interviews with subjects (Courvoisier et al., 2012). EMA is a tool of assessment that researchersPage 205 can use to examine behaviors and subjective states in the settings in which they naturally occur, and at a frequency that can capture their variability. Through the use of EMA we learned, among other things, that subject compliance was not attributable to personality factors (see Courvoisier et al., 2012 for full details).

Being a psychometrician can be most fulfilling, especially when one’s measurement-related knowledge and expertise brings added value to a research project that has exciting prospects for bettering the quality of life for members of a specific population. Psychologists who raise compelling research questions understand that the road to satisfactory answers is paved with psychometric essentials such as a sound research design, the use of appropriate measures, and accurate analysis and interpretation of findings. Psychometricians lend their expertise in these areas to help make research meaningful, replicable, generalizable, and actionable. From my own experience, one day I might be meeting with a researcher to discuss why a particular test is (or is not) more appropriate as an outcome measure, given the unique design and objectives of the study. Another day might find me cautioning experimenters against the use of a spontaneously created, “home-made” questionnaire for the purpose of screening subjects. In such scenarios, a strong knowledge of psychometrics combined with a certain savoir faire in diplomacy would seem to be useful prerequisites to success.

I would advise any student who is considering or contemplating a career as a psychometrician to learn everything they can about measurement theory and practice. In addition, the student would do well to cultivate the interpersonal skills that will most certainly be needed to interact professionally and effectively with fellow producers and consumers of psychological research. Contrary to what many may hold as an intuitive truth, success in the world of psychometrics cannot be measured by numbers alone.

Used with permission of Delphine Courvoisier.

Benefits

Judgments regarding the utility of a test may take into account whether the benefits of testing justify the costs of administering, scoring, and interpreting the test. So, when evaluating the utility of a particular test, an evaluation is made of the costs incurred by testing as compared to the benefits accrued from testing. Here, benefit refers to profits, gains, or advantages. As we did in discussing costs associated with testing (and not testing), we can view benefits in both economic and noneconomic terms.

From an economic perspective, the cost of administering tests can be minuscule when compared to the economic benefits—or financial returns in dollars and cents—that a successful testing program can yield. For example, if a new personnel testing program results in the selection of employees who produce significantly more than other employees, then the program will have been responsible for greater productivity on the part of the new employees. This greater productivity may lead to greater overall company profits. If a new method of quality control in a food-processing plant results in higher quality products and less product being trashed as waste, the net result will be greater profits for the company.

There are also many potential noneconomic benefits to be derived from thoughtfully designed and well-run testing programs. In industrial settings, a partial list of such noneconomic benefits—many carrying with them economic benefits as well—would include:

· an increase in the quality of workers’ performance;

· an increase in the quantity of workers’ performance;

· a decrease in the time needed to train workers;

· a reduction in the number of accidents;

· a reduction in worker turnover.

The cost of administering tests can be well worth it if the result is certain noneconomic benefits, such as a good work environment. As an example, consider the admissions program in place at most universities. Educational institutions that pride themselves on their graduates are often on the lookout for ways to improve the way that they select applicants for theirPage 206 programs. Why? Because it is to the credit of a university that their graduates succeed at their chosen careers. A large portion of happy, successful graduates enhances the university’s reputation and sends the message that the university is doing something right. Related benefits to a university that has students who are successfully going through its programs may include high morale and a good learning environment for students, high morale of and a good work environment for the faculty, and reduced load on counselors and on disciplinary personnel and boards. With fewer students leaving the school before graduation for academic reasons, there might actually be less of a load on admissions personnel as well; the admissions office will not be constantly working to select students to replace those who have left before completing their degree programs. A good work environment and a good learning environment are not necessarily things that money can buy. Such outcomes can, however, result from a well-administered admissions program that consistently selects qualified students who will keep up with the work and “fit in” to the environment of a particular university.

JUST THINK . . .

Provide an example of another situation in which the stakes involving the utility of a tool of psychological assessment are high.

One of the economic benefits of a diagnostic test used to make decisions about involuntary hospitalization of psychiatric patients is a benefit to society at large. Persons are frequently confined involuntarily for psychiatric reasons if they are harmful to themselves or others. Tools of psychological assessment such as tests, case history data, and interviews may be used to make a decision regarding involuntary psychiatric hospitalization. The more useful such tools of assessment are, the safer society will be from individuals intent on inflicting harm or injury. Clearly, the potential noneconomic benefit derived from the use of such diagnostic tools is great. It is also true, however, that the potential economic costs are great when errors are made. Errors in clinical determination made in cases of involuntary hospitalization may cause people who are not threats to themselves or others to be denied their freedom. The stakes involving the utility of tests can indeed be quite high.

How do professionals in the field of testing and assessment balance variables such as psychometric soundness, benefits, and costs? How do they come to a judgment regarding the utility of a specific test? How do they decide that the benefits (however defined) outweigh the costs (however defined) and that a test or intervention indeed has utility? There are formulas that can be used with values that can be filled in, and there are tables that can be used with values to be looked up. We will introduce you to such methods in this chapter. But let’s preface our discussion of utility analysis by emphasizing that other, less definable elements—such as prudence, vision, and, for lack of a better (or more technical) term, common sense—must be ever-present in the process. A psychometrically sound test of practical value is worth paying for, even when the dollar cost is high, if the potential benefits of its use are also high or if the potential costs of not using it are high. We have discussed “costs” and “benefits” at length in order to underscore that such matters cannot be considered solely in monetary terms.

Utility Analysis

What Is a Utility Analysis?

A utility analysis may be broadly defined as a family of techniques that entail a cost–benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment. Note that in this definition, we used the phrase “family of techniques.” This is so because a utility analysis is not one specific technique used for one specific objective. Rather, utility analysis is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding various kinds of output. Some utility analyses are quite sophisticated, employing high-level mathematical models and detailed strategies forPage 207 weighting the different variables under consideration (Roth et al., 2001). Other utility analyses are far more straightforward and can be readily understood in terms of answers to relatively uncomplicated questions, such as: “Which test gives us more bang for the buck?”

In a most general sense, a utility analysis may be undertaken for the purpose of evaluating whether the benefits of using a test (or training program or intervention) outweigh the costs. If undertaken to evaluate a test, the utility analysis will help make decisions regarding whether:

· one test is preferable to another test for use for a specific purpose;

· one tool of assessment (such as a test) is preferable to another tool of assessment (such as behavioral observation) for a specific purpose;

· the addition of one or more tests (or other tools of assessment) to one or more tests (or other tools of assessment) that are already in use is preferable for a specific purpose;

· no testing or assessment is preferable to any testing or assessment.

If undertaken for the purpose of evaluating a training program or intervention, the utility analysis will help make decisions regarding whether:

· one training program is preferable to another training program;

· one method of intervention is preferable to another method of intervention;

· the addition or subtraction of elements to an existing training program improves the overall training program by making it more effective and efficient;

· the addition or subtraction of elements to an existing method of intervention improves the overall intervention by making it more effective and efficient;

· no training program is preferable to a given training program;

· no intervention is preferable to a given intervention.

The endpoint of a utility analysis is typically an educated decision about which of many possible courses of action is optimal. For example, in a now-classic utility analysis, Cascio and Ramos (1986) found that the use of a particular approach to assessment in selecting managers could save a telephone company more than $13 million over four years (see also Cascio, 1994, 2000).

Whether reading about utility analysis in this chapter or in other sources, a solid foundation in the language of this endeavor—both written and graphic—is essential. Toward that end, we hope you find the detailed case illustration presented in our Close-Up helpful.

Page 208

CLOSE-UP

Utility Analysis: An Illustration

Like factor analysis, discriminant analysis, psychoanalysis, and other specific approaches to analysis and evaluation, utility analysis has its own vocabulary. It even has its own images in terms of graphic representations of various phenomena. As a point of departure for learning about the words and images associated with utility analysis, we present a hypothetical scenario involving utility-related issues that arise in a corporate personnel office. The company is a South American package delivery company called Federale (pronounced fed-a-rally) Express (FE). The question at hand concerns the cost-effectiveness of adding a new test to the process of hiring delivery drivers. Consider the following details.

Dr. Wanda Carlos, the personnel director of Federale Express, has been charged with the task of evaluating the utility of adding a new test to the procedures currently in place for hiring delivery drivers. Current FE policy states that drivers must possess a valid driver’s license and have no criminal record. Once hired, the delivery driver is placed on probation for three months, during which time on-the-job supervisory ratings (OTJSRs) are collected on random work days. If scores on the OTJSRs are satisfactory at the end of the probationary period, then the new delivery driver is deemed “qualified.” Only qualified drivers attain permanent employee status and benefits at Federale Express.

The new evaluation procedure to be considered from a cost-benefit perspective is the Federale Express Road Test (FERT). The FERT is a procedure that takes less than one hour and entails the applicant driving an FE truck in actual traffic to a given destination, parallel parking, and then driving back to the start point. Does the FERT evidence criterion-related validity? If so, what cut score instituted to designate passing and failing scores would provide the greatest utility? These are preliminary questions that Dr. Carlos seeks to answer “on the road” to tackling issues of utility. They will be addressed in a study exploring the predictive validity of the FERT.

Dr. Carlos conducts a study in which a new group of drivers is hired based on FE’s existing requirements: possession of a valid driver’s license and no criminal record. However, to shed light on the question of the value of adding a new test to the process, these new hires must also take the FERT. So, subsequent to their hiring and after taking the FERT, these new employees are all placed on probation for the usual period of three months. During this probationary period, the usual on-the-job supervisory ratings (OTJSRs) are collected on randomly selected work days. The total scores the new employees achieve on the OTJSRs will be used to address not only the question of whether the new hire is qualified but also questions concerning the added value of the FERT in the hiring process.

The three-month probationary period for the new hires is now over, and Dr. Carlos has accumulated quite a bit of data including scores on the predictor measure (the FERT) and scores on the criterion measure (the OTJSR). Looking at these data, Dr. Carlos wonders aloud about setting a cut score for the FERT . . . but does she even need to set a cut score? What if FE hired as many new permanent drivers as they need by a process of top-down selection with regard to OTJSRs? Top-down selection is a process of awarding available positions to applicants whereby the highest scorer is awarded the first position, the next highest scorer the next position, and so forth until all positions are filled. Dr. Carlos decides against a top-down hiring policy based on her awareness of its possible adverse impact. Top-down selection practices may carry with them unintended discriminatory effects (Cascio et al., 1995; De Corte & Lievens, 2005; McKinney & Collins, 1991; Zedeck et al., 1996).

For assistance in setting a cut score for hiring and in answering questions related to the utility of the FERT, Dr. Carlos purchases a (hypothetical) computer program entitled Utility Analysis Made Easy. This program contains definitions for a wealth of utility-related terms and also provides the tools for automatically creating computer-generated, utility-related tables and graphs. In what follows we learn, along with Dr. Carlos, how utility analysis can be “made easy” (or, at the very least, somewhat less complicated). After entering all of the data from this study, she enters the command set cut score, and what pops up is a table (Table 1) and this prompt:

There is no single, all-around best way to determine the cut score to use on the FERT. The cut score chosen will reflect the goal of the selection process. In this case, consider which of the following four options best reflects the company’s hiring policy and objectives. For some companies, the best cut score may be no cut score (Option 1).

(1) Limit the cost of selection by not using the FERT.

This goal could be appropriate (a) if Federale Express just needs “bodies” to fill positions in order to continue operations, (b) if the consequences of hiring unqualified personnel are not a major consideration; and/or (c) if the size of the applicant pool is equal to or smaller than the number of openings.

Page 209

Table 1 Hits and Misses

Term

General Definition

What It Means in This Study

Implication

Hit

1. A correct classification
2. A passing score on the FERT is associated with satisfactory performance on the OTJSR, and a failing score on the FERT is associated with unsatisfactory performance on the OTJSR.
3. The predictor test has successfully predicted performance on the criterion; it has successfully predicted on-the-job outcome. A qualified driver is hired; an unqualified driver is not hired.

Miss

1. An incorrect classification; a mistake
2. A passing score on the FERT is associated with unsatisfactory performance on the OTJSR, and a failing score on the FERT is associated with satisfactory performance on the OTJSR.
3. The predictor test has not predicted performance on the criterion; it has failed to predict the on-the-job outcome. A qualified driver is not hired; an unqualified driver is hired.

Hit rate

1. The proportion of people that an assessment tool accurately identifies as possessing or exhibiting a particular trait, ability, behavior, or attribute
2. The proportion of FE drivers with a passing FERT score who perform satisfactorily after three months based on OTJSRs. Also, the proportion of FE drivers with a failing FERT score who do not perform satisfactorily after three months based on OTJSRs.
3. The proportion of qualified drivers with a passing FERT score who actually gain permanent employee status after three months on the job. Also, the proportion of unqualified drivers with a failing FERT score who are let go after three months.

Miss rate

1. The proportion of people that an assessment tool inaccurately identifies as possessing or exhibiting a particular trait, ability, behavior, or attribute
2. The proportion of FE drivers with a passing FERT score who perform unsatisfactorily after three months based on OTJSRs. Also, the proportion of FE drivers with a failing FERT score who perform satisfactorily after three months based on OTJSRs.
3. The proportion of drivers whom the FERT inaccurately predicted to be qualified. Also, the proportion of drivers whom the FERT inaccurately predicted to be unqualified

False positive

1. A specific type of miss whereby an assessment tool falsely indicates that the testtaker possesses or exhibits a particular trait, ability, behavior, or attribute
2. The FERT indicates that the new hire will perform successfully on the job but, in fact, the new driver does not.
3. A driver who is hired is not qualified

False negative

1. A specific type of miss whereby an assessment tool falsely indicates that the testtaker does not possess or exhibit a particular trait, ability, behavior, or attribute
2. The FERT indicates that the new hire will not perform successfully on the job but, in fact, the new driver would have performed successfully.
3. FERT says to not hire but driver would have been rated as qualified.

a. Boudreau (1988).

Page 210

(2) Ensure that qualified candidates are not rejected.

To accomplish this goal, set a FERT cut score that ensures that no one who is rejected by the cut would have been deemed qualified at the end of the probationary period. Stated another way, set a cut score that yields the lowest false negative rate. The emphasis in such a scenario is on weeding out the “worst” applicants; that is, those applicants who will definitely be deemed unqualified at the end of the probationary period.

(3) Ensure that all candidates selected will prove to be qualified.

To accomplish this goal, set a FERT cut score that ensures that everyone who “makes the cut” on the FERT is rated as qualified at the end of the probationary period; no one who “makes the cut” is rated as unqualified at the end of the probationary period. Stated another way, set a cut score that yields the lowest false positive rate. The emphasis in such a scenario is on selecting only the best applicants; that is, those applicants who will definitely be deemed qualified at the end of the probationary period.

(4) Ensure, to the extent possible, that qualified candidates will be selected and unqualified candidates will be rejected.

This objective can be met by setting a cut score on the FERT that is helpful in (a) selecting for permanent positions those drivers who performed satisfactorily on the OTJSR, (b) eliminating from consideration those drivers who performed unsatisfactorily on the OTJSR, and (c) reducing the miss rate as much as possible. This approach to setting a cut score will yield the highest hit rate while allowing for FERT-related “misses” that may be either of the false-positive or false-negative variety. Here, false positives are seen as no better or worse than false negatives and vice versa.

It is seldom possible to “have it all ways.” In other words, it is seldom possible to have the lowest false positive rate, the lowest false negative rate, the highest hit rate, and not incur any costs of testing. Which of the four listed objectives represents the best “fit” with your policies and the company’s hiring objectives? Before responding, it may be helpful to review Table 1 .

After reviewing Table 1 and all of the material on terms including hit, miss, false positive, and false negative, Dr. Carlos elects to continue and is presented with the following four options from which to choose.

1. Select applicants without using the FERT.
2. Use the FERT to select with the lowest false negative rate.
3. Use the FERT to select with the lowest false positive rate.
4. Use the FERT to yield the highest hit rate and lowest miss rate.

Curious about the outcome associated with each of these four options, Dr. Carlos wishes to explore all of them. She begins by selecting Option 1: Select applicants without using the FERT. Immediately, a graph (Close-Up Figure 1) and this prompt pop up:

Figure 1 Base Rate Data for Federale Express Before the use of the FERT, any applicant with a valid driver’s license and no criminal record was hired for a permanent position as an FE driver. Drivers could be classified into two groups based on their on-the-job supervisory ratings (OTJSRs): those whose driving was considered to be satisfactory (located above the dashed horizontal line) and those whose driving was considered to be unsatisfactory (below the dashed line). Without use of the FERT, then, all applicants were hired and the selection ratio was 1.0; 60 drivers were hired out of the 60 applicants. However, the base rate of successful performance shown in Figure 1 was only .50. This means that only half of the drivers hired (30 of 60) were considered “qualified” drivers by their supervisor. This also shows a miss rate of .50, because half of the drivers turned out to perform below the minimally accepted level.   Yet because scores on the FERT and the OTJSRs are positively correlated, the FERT can be used to help select the individuals who are likely to be rated as qualified drivers. Thus, using the FERT is a good idea, but how should it be used? One method would entail top-down selection. That is, a permanent position could be offered first to the individual with the highest score on the FERT (top, rightmost case in Figure 1 ), followed by the individual with the next highest FERT score, and so on until all available positions are filled. As you can see in the figure, if permanent positions are offered only to individuals with the top 20 FERT scores, then OTJSR ratings of the permanent hires will mostly be in the satisfactory performer range. However, as previously noted, such a top-down selection policy can be discriminatory.

Generally speaking, base rate is defined as the proportion of people in the population that possess a particular trait, behavior, characteristic, or attribute. In this study, base rate refers to the proportion of new hire drivers who would go on to perform satisfactorily on the criterion measure (the OTJSRs) and be deemed “qualified” regardless of whether or not a test such as the FERT existed (and regardless of their score on the FERT if it were administered). The base rate is represented in Figure 1 (and in all subsequent graphs) by the number of drivers whose OTJSRs fall above the dashed horizontal line (a line that refers to minimally acceptable performance on the OTJSR) as compared to the total number of scores. In other words, the base rate is equal to the ratio of qualified applicants to the total number of applicants.

Without the use of the FERT, it is estimated that about one-half of all new hires would exhibit satisfactory performance; that is, the base rate would be .50. Without use of the FERT, the miss rate would also be .50—this because half of all drivers hired would be deemed unqualified based on the OTJSRs at the end of the probationary period.

Page 211

Dr. Carlos considers the consequences of a 50% miss rate. She thinks about the possibility of an increase in customer complaints regarding the level of service. She envisions an increase in at-fault accidents and costly lawsuits. Dr. Carlos is pleasantly distracted from these potential nightmares when she inadvertently leans on her keyboard and it furiously begins to beep. Having rejected Option 1, she “presses on” and next explores what outcomes would be associated with Option 2: Use the FERT to select with the lowest false negative rate. Now, another graph (Close-Up Figure 2) appears along with this text:

This graph, as well as all others incorporating FERT cut-score data, have FERT (predictor) scores on the horizontal axis (which increase from left to right), and OTJSR (criterion) scores on the vertical axis (with scores increasing from the bottom toward the top). The selection ratio provides an indication of the competitiveness of the position; it is directly affected by the cut score used in selection. As the cut score is set farther to the right, the selection ratio goes down. The practical implication of the decreasing selection ratio is that hiring becomes more selective; this means that there is more competition for a position and that the proportion of people actually hired (from all of those who applied) will be less.2 As the cut score is set farther to the left, the selection ratio goes up; hiring becomes less selective, and chances are that more people will be hired.3

Using a cut score of 18 on the FERT, as compared to not using the FERT at all, reduces the miss rate from 50% to 45% (see Figure 2). The major advantage of setting the cut score this low is that the false negative rate falls to zero; no potentially qualified drivers will be rejected based on the FERT. Use of this FERT cut score also increases the base rate of successful performance from .50 to .526. This means that the percentage of hires who will be rated as “qualified” has increased from 50% without use of the FERT to 52.6% with the FERT. The selection ratio associated with using 18 as the cut score is .95, which means that 95% of drivers who apply are selected.

Figure 2 Selection with Low Cut Score and High Selection Ratio As we saw in Figure 1 , without the use of the FERT, only half of all the probationary hires would be rated as satisfactory drivers by their supervisors. Now we will consider how to improve selection by using the FERT. For ease of reference, each of the quadrants in Figure 2 (as well as the remaining Close-Up graphs) have been labeled, A, B, C, or D. The selection ratio in this and the following graphs may be defined as being equal to the ratio of the number of people who are hired on a permanent basis (qualified applicants as determined by FERT score) compared to the total number of people who apply.   The total number of applicants for permanent positions was 60, as evidenced by all of the dots in all of the quadrants. In quadrants A and B, just to the right of the vertical Cut score line (set at 18), are the 57 FE drivers who were offered permanent employment. We can also see that the false positive rate is zero because no scores fall in quadrant D; thus, no potentially qualified drivers will be rejected based on use of the FERT with a cut score of 18. The selection ratio in this scenario is 57/60, or .95. We can therefore conclude that 57 applicants (95% of the 60 who originally applied) would have been hired on the basis of their FERT scores with a cut score set at 18 (resulting in a “high” selection ratio of 95%); only three applicants would not be hired based on their FERT scores. These three applicants would also be rated as unqualified by their supervisors at the end of the probationary period. We can also see that, by removing the lowest-scoring applicants, the base rate of successful performance improves slightly as compared to not using the FERT at all. Instead of having a successful performance base rate of only .50 (as was the case when all applicants were hired), now the base rate of successful performance is .526. This is so because 30 drivers are still rated as qualified based on OTJSRs while the number of drivers hired has been reduced from 60 to 57.

Dr. Carlos appreciates that the false negative rate is zero and thus no potentially qualified drivers are turned away based on FERT score. She also believes that a 5% reduction in the miss Page 212rate is better than no reduction at all. She wonders, however, whether this reduction in the miss rate is statistically significant. She would have to formally analyze these data to be certain but, after simply “eyeballing” these findings, a decrease in the miss rate from 50% to 45% does not seem significant. Similarly, an increase in the number of qualified drivers of only 2.6% through the use of a test for selection purposes does not, on its face, seem significant. It simply does not seem prudent to institute a new personnel selection test at real cost and expense to the company if the only benefit of the test is to reject the lowest-scoring 3 of 60 applicants—when, in reality, 30 of the 60 applicants will be rated as “unqualified.”

Dr. Carlos pauses to envision a situation in which reducing the false negative rate to zero might be prudent; it might be ideal if she were testing drivers for drug use, because she would definitely not want a test to indicate a driver is drug-free if that driver had been using drugs. Of course, a test with a false negative rate of zero would likely also have a high false positive rate. But then she could retest any candidate who received a positive result with a second, more expensive, more accurate test—this to ensure that the initial positive result was correct and not a testing error. As Dr. Carlos mulls over these issues, a colleague startles her with a friendly query: “How’s that FERT researching coming?”

Dr. Carlos says, “Fine,” and smoothly reaches for her keyboard to select Option 3: Use the FERT to select with the lowest false positive rate. Now, another graph (Close-Up Figure 3) and another message pop up:

Using a cut score of 80 on the FERT, as compared to not using the FERT at all, results in a reduction of the miss rate from 50% to 40% (see Figure 3) but also reduces the false positive rate to zero. Use of this FERT cut score also increases the base rate of successful performance from .50 to 1.00. This means that the percentage of drivers selected who are rated as “qualified” increases from 50% without use of the FERT to 100% when the FERT is used with a cut score of 80. The selection ratio associated with using 80 as the cut score is .10, which means that 10% of applicants are selected.

Figure 3 Selection with High Cut Score and Low Selection Ratio As before, the total number of applicants for permanent positions was 60, as evidenced by all of the dots in all of the quadrants. In quadrants A and B, just to the right of the vertical Cut score line (set at a FERT score of 80), are the 6 FE drivers who were offered permanent employment. The selection ratio in this scenario is 6/60, or .10. We can therefore conclude that 6 applicants (10% of the 60 who originally applied) would have been hired on the basis of their FERT scores with the cut score set at 80 (and with a “low” selection ratio of 10%). Note also that the base rate improves dramatically, from .50 without use of the FERT to 1.00 with a FERT cut score set at 80. This means that all drivers selected when this cut score is in place will be qualified. Although only 10% of the drivers will be offered permanent positions, all who are offered permanent positions will be rated qualified drivers on the OTJSR. Note, however, that even though the false positive rate drops to zero, the overall miss rate only drops to .40. This is so because a substantial number (24) of qualified applicants would be denied permanent positions because their FERT scores were below 80.

Dr. Carlos likes the idea of the “100% solution” entailed by a false positive rate of zero. It means that 100% of the applicants selected by their FERT scores will turn out to be qualified drivers. At first blush, this solution seems optimal. However, there is, as they say, a fly in the ointment. Although the high cut score (80) results in the selection of only qualified candidates, the selection ratio is so stringent that only 10% of those candidates would actually be hired. Dr. Carlos envisions the consequences of this low selection ratio. She sees herself as having to recruit and test at least 100 applicants for every 10 drivers she actually hires. To meet her company goal of hiring 60 drivers, for example, she would have to recruit about 600 applicants for testing. Attracting that many applicants to the company is a venture that has some obvious (as well as some less obvious) costs. Dr. Carlos sees her recruiting budget dwindle as she repeatedly writes checks for classified advertising in newspapers. She sees herself purchasing airline tickets and making hotel reservations in order to attend various job fairs, far and wide. Fantasizing about the applicants she will attract at one of those job fairs, she is abruptly brought Page 213back to the here-and-now by the friendly voice of a fellow staff member asking her if she wants to go to lunch. Still half-steeped in thought about a potential budget crisis, Dr. Carlos responds, “Yes, just give me ten dollars . . . I mean, ten minutes.”

As Dr. Carlos takes the menu of a local hamburger haunt from her desk to review, she still can’t get the “100% solution” out of her mind. Although clearly attractive, she has reservations (about the solution, not for the restaurant). Offering permanent positions to only the top-performing applicants could easily backfire. Competing companies could be expected to also offer these applicants positions, perhaps with more attractive benefit packages. How many of the top drivers hired would actually stay at Federale Express? Hard to say. What is not hard to say, however, is that the use of the “100% solution” has essentially brought Dr. Carlos full circle back to the top-down hiring policy that she sought to avoid in the first place. Also, scrutinizing Figure 3, Dr. Carlos sees that—even though the base rate with this cut score is 100%—the percentage of misclassifications (as compared to not using any selection test) is reduced only by a measly 10%. Further, there would be many qualified drivers who would also be cut by this cut score. In this instance, then, a cut score that scrupulously seeks to avoid the hiring of unqualified drivers also leads to rejecting a number of qualified applicants. Perhaps in the hiring of “super responsible” positions—say, nuclear power plant supervisors—such a rigorous selection policy could be justified. But is such rigor really required in the selection of Federale Express drivers?

Hoping for a more reasonable solution to her cut-score dilemma and beginning to feel hungry, Dr. Carlos leafs through the burger menu while choosing Option 4 on her computer screen: Use the FERT to yield the highest hit rate and lowest miss rate. In response to this selection, another graph (Close-Up Figure 4) along with the following message is presented:

Using a cut score of 48 on the FERT results in a reduction of the miss rate from 50% to 15% as compared to not using the FERT (see Figure 4). False positive and false negative rates are both fairly low at .167 and .133, respectively. Use of this cut score also increases the base rate from .50 (without use of the FERT) to .839. This means that the percentage of hired drivers who are rated as “qualified” at the end of the probationary period has increased from 50% (without use of the FERT) to 83.9%. The selection ratio associated with using 48 as the cut score is .517, which means that 51.7% of applicants will be hired.

Figure 4 Selection with Moderate Cut Score and Moderate Selection Ratio Again, the total number of applicants was 60. In quadrants A and B, just to the right of the vertical Cut Score line (set at 48), are the 31 FE drivers who were offered permanent employment at the end of the probationary period. The selection ratio in this scenario is therefore equal to 31/60, or about .517. This means that slightly more than half of all applicants will be hired based on the use of 48 as the FERT cut score. The selection ratio of .517 is a moderate one. It is not as stringent as is the .10 selection ratio that results from a cut score of 80, nor is it as lenient as the .95 selection ratio that results from a cut score of 18. Note also that the cut score set at 48 effectively weeds out many of the applicants who won’t receive acceptable performance ratings. Further, it does this while retaining many of the applicants who will receive acceptable performance ratings. With a FERT cut score of 48, the base rate increases quite a bit: from .50 (as was the case without using the FERT) to .839. This means that about 84% (83.9%, to be exact) of the hired drivers will be rated as qualified when the FERT cut score is set to 48 for driver selection.

Although a formal analysis would have to be run, Dr. Carlos again “eyeballs” the findings and, based on her extensive experience, strongly suspects that these results are statistically significant. Moreover, these findings would seem to be of practical significance. As compared to not using the FERT, use of the FERT with a cut score of 48 could reduce misclassifications from 50% to 15%. Such a reduction in misclassifications would almost certainly have positive cost–benefit implications for FE. Also, the percentage of drivers who are deemed qualified at the end of the probationary period would rise from 50% (without use of the FERT) to 83.9% (using the FERT with a cut score of 48). The implications of such improved selection are many and include better service to customers (leading to an increase in business volume), less costly accidents, and fewer costs involved in hiring and training new personnel.

Yet another benefit of using the FERT with a cut score of 48 concerns recruiting costs. Using a cut score of 48, FE would need to recruit only 39 or so qualified applicants for every 20 permanent positions it needed to fill. Now, anticipating real savings in her annual budget, Dr. Carlos returns the hamburger menu to her desk drawer and removes instead the menu from her favorite (pricey) steakhouse.

Page 214

Dr. Carlos decides that the moderate cut-score solution is optimal for FE. She acknowledges that this solution doesn’t reduce any of the error rates to zero. However, it produces relatively low error rates overall. It also yields a relatively high hit rate; about 84% of the drivers hired will be qualified at the end of the probationary period. Dr. Carlos believes that the costs associated with recruitment and testing using this FERT cut score will be more than compensated by the evolution of a work force that evidences satisfactory performance and has fewer accidents. As she peruses the steakhouse menu and mentally debates the pros and cons of sautéed onions, she also wonders about the dollars-and-cents utility of using the FERT. Are all of the costs associated with instituting the FERT as part of FE hiring procedures worth the benefits?

Dr. Carlos puts down the menu and begins to calculate the company’s return on investment (the ratio of benefits to costs). She estimates the cost of each FERT to be about $200, including the costs associated with truck usage, gas, and supervisory personnel time. She further estimates that FE will test 120 applicants per year in order to select approximately 60 new hires based on a moderate FERT cut score. Given the cost of each test ($200) administered individually to 120 applicants, the total to be spent on testing annually will be about $24,000. So, is it worth it? Considering all of the possible benefits previously listed that could result from a significant reduction of the misclassification rate, Dr. Carlos’s guess is, “Yes, it would be worth it.” Of course, decisions like that aren’t made with guesses. So continue reading—later in this chapter, a formula will be applied that will prove Dr. Carlos right. In fact, the moderate cut score shown in Figure 4 would produce a return on investment of 12.5 to 1. And once Dr. Carlos gets wind of these projections, you can bet it will be surf-and-turf-tortilla time at Federale Express.

How Is a Utility Analysis Conducted?

The specific objective of a utility analysis will dictate what sort of information will be required as well as the specific methods to be used. Here we will briefly discuss two general approaches to utility analysis. The first is an approach that employs data that should actually be quite familiar.

Expectancy data

Some utility analyses will require little more than converting a scatterplot of test data to an expectancy table (much like the process described in the previous chapter). An expectancy table can provide an indication of the likelihood that a testtaker will score within some interval of scores on a criterion measure—an interval that may be categorized as “passing,” “acceptable,” or “failing.” For example, with regard to the utility of a new and experimental personnel test in a corporate setting, an expectancy table can provide vital information to decision-makers. An expectancy table might indicate, for example, that the higher a worker’s score is on this new test, the greater the probability that the worker will be judged successful. In other words, the test is working as it should and, by instituting this new test on a permanent basis, the company could reasonably expect to improve its productivity.

Tables that could be used as an aid for personnel directors in their decision-making chores were published by H. C. Taylor and J. T. Russell in the Journal of Applied Psychology in 1939. Referred to by the names of their authors, the Taylor-Russell tables provide an estimate of the extent to which inclusion of a particular test in the selection system will improve selection. More specifically, the tables provide an estimate of the percentage of employees hired by the use of a particular test who will be successful at their jobs, given different combinations of three variables: the test’s validity, the selection ratio used, and the base rate.

The value assigned for the test’s validity is the computed validity coefficient. The selection ratio is a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired. For instance, if there are 50 positions and 100 applicants, then the selection ratio is 50/100, or .50. As used here, base rate refers to the percentage of people hired under the existing system for a particular position. If, for example, a firm employs 25 computer programmers and 20 are considered successful, the base rate would be .80. With knowledge of the validity coefficient of a particular test along with the selection ratio, reference to the Taylor-Russell tables provides the personnel officer with an estimate of how much using the test would improve selection over existing methods.

A sample Taylor-Russell table is presented in Table 7–1. This table is for the base rate of .60, meaning that 60% of those hired under the existing system are successful in their work. Down the left-hand side are validity coefficients for a test that could be used to help select employees. Across the top are the various selection ratios. They reflect the proportion of the people applying for the jobs who will be hired. If a new test is introduced to help select employees in a situation with a selection ratio of .20 and if the new test has a predictive validity coefficient of .55, then the table shows that the base rate will increase to .88. This means that, rather than 60% of the hired employees being expected to perform successfully, a full 88% can be expected to do so. When selection ratios are low, as when only 5% of the applicants will be hired, even tests with low validity coefficients, such as .15, can result in improved base rates.

Selection Ratio

Validity (ρxy)

.05

.10

.20

.30

.40

.50

.60

.70

.80

.90

.95

.00

.60

.60

.60

.60

.60

.60

.60

.60

.60

.60

.60

.05

.64

.63

.63

.62

.62

.62

.61

.61

.61

.60

.60

.10

.68

.67

.65

.64

.64

.63

.63

.62

.61

.61

.60

.15

.71

.70

.68

.67

.66

.65

.64

.63

.62

.61

.61

.20

.75

.73

.71

.69

.67

.66

.65

.64

.63

.62

.61

.25

.78

.76

.7

The post Methods for Setting Cut Scores appeared first on Lion Essays.

 

“Looking for a Similar Assignment? Get Expert Help at an Amazing Discount!”

---

Methods for Setting Cut Scores was first posted on May 2, 2019 at 9:42 pm.
©2019 "Lion Essays". Use of this feed is for personal non-commercial use only. If you are not reading this article in your feed reader, then the site is guilty of copyright infringement. Please contact me at support@Lion Essays.com