Looking for Patterns in Performance Evaluations
Most employers expect their managers to complete regular performance evaluations for the staff they supervise, which can be used to track employee progress and provide at least a minimal employment history. An observer could reasonably ask, however, whether companies derive as much benefit from this process as they assume. A January 9, 2015 online article titled Pattern Discovery in Students’ Evaluations of Professors: A Statistical Data Mining Approach, which was submitted to the Journal of Applied Statistics by researchers Necla Gündüz at Gazi University in Ankara, Turkey and Ernest Fokoué at the Rochester Institute of Technology, offers some insights for employers examining their evaluation process and reviewing managerial performance.
The researchers here examined 5,820 evaluations by students of their professors in 13 courses taught at a Turkish university. Each evaluation consisted of 28 statements (such as “the course aims and objectives were clearly stated,” “the course was relevant and beneficial to my professional development,” and “the instructor came prepared for class”), and asked students whether they strongly disagreed, disagreed, had a neutral response, agreed, or strongly agreed with each statement. These survey responses were then examined using a variety of statistical techniques.
Slightly over half of all respondents gave what the authors termed a “zero variation response”, meaning that they gave a uniform response (“agree” or “strongly disagree”, for example) to all 28 statements. The authors’ data also reveals that, at least for some classes, 45% of the students giving a zero-variation response had poor attendance, and 40% chose (candidly or not) to rate the course as “too easy.” (There was very high overlap between those who didn’t come to class and those who claimed it was too easy; the study did not, however, compare these responses to the actual grades students received in the class. In contrast, students who thought the course had a normal level of difficulty tended to take the time to provide varied answers to these survey questions.) The researchers correctly counsel that zero-variation respondents simply didn’t think very carefully about their responses, and therefore their answers – half of the data set – may be useless to those trying to improve the course, except (arguably) to assess overall satisfaction.
Further analysis revealed that, out of all 28 queries administered as part of these course surveys, one query turned out to consistently predict students’ overall impressions: their response to the statement, “My initial expectations about the course were met at the end of the period or year.” Students’ responses to this statement, in turn, were almost perfectly predicted by their response to a statement about the contribution of tests and assignments to their learning the material. To the researchers’ surprise, no variable relating to the professors’ performance seemed to be important to overall student satisfaction; they worry, in effect, that students may not care about the professor as long as they’re satisfied with their grade. The researchers conclude with reflections that a 28-question survey is apparently too taxing on participants, that the length of the questionnaire may contribute to the high rate of zero-variation responses, and that asking only two questions instead (“What is your overall rating for this instructor,” and “Would you recommend this course to any other student”) might be more useful, as these two questions together captured 85% of all variation in the survey responses.
Many employers require their managers to complete comparable questionnaires about their employees each year. Offhand, I can’t think of any I’ve seen that are shorter than two pages, and some are a dozen pages or more in length, and usually combine categorical ratings (corresponding to exceeds, meets, or fails to meet expectations) with narrative sections. I tend to review these documents only in the context of actual or threatened litigation, but if my own experience is a guide, most managers seem to complete these reviews conscientiously for most employees: like the committed students in Gündüz and Fokoué’s study, their responses vary from question to question, suggesting careful consideration of the questions being posed. But there are occasional instances in these performance evaluations where the manager has given what appears to be a zero-variation response that may actually reveal more about the relationships in the workplace than about the employee’s actual performance. I’ve also seen a handful of instances where the supervisor appears to have copied their narrative verbatim from one year’s evaluation to the next, apparently without recognizing that doing so may imply neglect or failure somewhere in the manager-employee relationship.
Perhaps because managers usually have a greater opportunity to get to know and spend time with their employees than students do their professors, Gündüz and Fokoué’s study cannot capture a further nuance of the zero-variation response that is all too familiar to most human resources professionals. In difficult termination decisions, a client may report that an employee has a long history of exasperating behavior and poor performance, but have to admit that little or none of this is reflected in the performance evaluations completed by the manager. Why this is so is almost never satisfactorily explained – sometimes, I think, it’s because the supervisor dislikes confrontation or delivering criticism, or believes that encouragement will motivate the employee more than consistently negative feedback, or (most disastrously) the supervisor seems to think that they’ll achieve the desired improvement in any employee’s performance by combining a positive written performance evaluation that omits any mention of the employee’s technical and behavioral shortcomings with spoken remarks that purport to address these flaws. I’ve discussed previously some of the reasons why false praise and other disingenuous feedback can be counterproductive, in that they discourage the employee’s engagement in their work; we can all imagine circumstances where an employee would be utterly demoralized if they dedicated a year to diligent self-improvement only to receive a performance evaluation copied verbatim from the prior year. I believe the study by Gündüz and Fokoué in some ways highlights the other side of the phenomenon: that a supervisor who provides zero-variation performance evaluations may not be supervising effectively. Zero-variation performance evaluations may occur for various reasons, ranging from reluctance to assert authority to burnout, but the result is the same: that the employee would arguably have some justification to disregard a performance evaluation that appeared to bear no relationship to their efforts, behavior, or performance. Such a result implicitly sanctions insubordination, however, and so should be intolerable to management.
In these managers’ defense, it’s a rare employer that actually teaches its managers how to evaluate performance, and I’m not aware of any employers that systematically review completed performance evaluations as part of their evaluation of their managers, even on a random or selective basis. I’d also guess that the quality of their completed performance evaluations is rarely, if ever, a factor in determining a manager’s year-end bonus or salary increase. So I could easily imagine a manager weighing the discomfort of an anticipated face-to-face interaction with a mediocre employee (perhaps who responds to criticism with rage or tears) against the employer’s seeming indifference to the content of these evaluations in most cases (no one ever plans for litigation), and choosing to postpone their confrontation with the employee over performance and behavior for another year. (Ideally, of course, the manager would be having such discussions when the behavior occurred, rather than waiting for year’s end, so that the performance evaluation wouldn’t be a surprise.) This procrastination, in turn, entrenches an unhealthy and counterproductive dynamic: the employee is taught that their behavior is acceptable, because the manager hasn’t said otherwise; the manager gives the employer the impression that a sincere performance evaluation was conducted, when the work product may actually be useless; an employee who took their evaluation seriously may be discouraged and demoralized by a cookie-cutter response; and careless performance evaluations may dramatically increase the cost and expense of litigation.
Employers should give careful thought to what they want their performance evaluations to accomplish. Although some managers seem to complete them reasonably well, even without training, the quality of the evaluation process seems to be largely unmonitored within most corporate hierarchies, and tends not to receive any scrutiny until employment litigation is already underway. Gündüz and Fokoué’s study therefore suggests some insights into the evaluation process that employers may wish to consider. First, employers should consider a short, concise format for performance evaluations that’s tailored to their intended purposes; for academic course evaluations, Gündüz and Fokoué thought two questions might be adequate. Employers would probably need more than two questions, but still have reason to keep their forms relatively concise. Second, and not suggested by these authors, managers should be coached about how to complete these forms, so they understand what they’re expected to accomplish and how the evaluations may be used to improve productivity and defend the company in litigation. Third, as inspired by the study, employers may wish to evaluate and hold their managers to account for the effectiveness of their performance evaluations, even if only by means of random audits that check for such details as change from previous years, the relevance of the narrative, and the reliability of zero-variation responses. A zero-variation performance review may reflect a lack of training, managerial burnout, a workplace dynamic that needs to be addressed, or otherwise signal that the manager may not be performing effectively in the role. Although such practices are not obligatory – and currently seem to be performed rarely, if at all – employers may find that experimenting with them may have a marked effect on a productivity and morale, and an indirect effect on litigation risk.
