The issue of cheating on tests is a rather frequent topic in the news. Unfortunately, these are often superficial accounts and don’t provide much detail. In contrast, USA Today had a recent investigative piece that looked at cheating on statewide school-based tests and the methods used to detect it.

There appear to be two basic detection approaches, and both look for test taking behavior that deviates significantly from the norm. One is simply based upon changes in scores from year to year. Specifically, when the scores for an entire grade go up very substantially from one year to the next concerns are raised. (Here, the typical criterion for a “very substantial increase” is when scores go up by 3 or more standard deviations, which is a huge difference). This is not to say that something untoward has necessarily occurred (it is certainly possible that there are some very good teachers in that grade) but it does raise concerns. This is even more the case if the grades drop substantially the following year.

Another metric is the average number of erasures. (Recall that most school-based testing  is still administered with paper and pencil.) Certainly, mistakes happen when kids put an answer in the wrong row, and all the following responses are out of sequence and need to be “fixed” and realigned. As a result, the focus is on whether entire groups of students have more erasures than others, especially if most of them result in correct answers.

This is certainly not to say that cheating is widespread on school-based testing. However, it would only seem to be human nature that there may be times when teachers do things that unfairly increase student test scores, especially when these assessments are used, in my opinion often unfairly, as a primary measure of an educator’s effectiveness. Regardless, this report provides some interesting insights into how cheating  behavior (which has been documented for over a millennium) is detected is school settings.

Reid Klion

Registration is now open for the 2011 Conference of the International Personnel Assessment Council. To be held July 17-20 in Washington, DC, the meeting promises to be of great value to anyone working in the area of selection and assessment. Guest speakers include Elaine Pulakos (PDRI), Wayne Camara (College Board) and Eric Dunleavy, Mike Aamodt, and David Cohen (all from the Center for Corporate Equality). There are also a number of pre-conference workshops on topics including how to design an assessment strategy, best practices in questionnaire design, strategies for scoring job simulations, and job task analysis. pan is again proud to be a sponsor of the conference and that Jocelyn Courtney-Hays will be speaking.

Reid Klion

There have been seemingly countless discussions about the potential risks associated with unproctored assessment. Facilitated by the development of internet-based testing, the benefits of having job candidates complete assessments without supervision are rather clear and include increased convenience as well as cost and time savings compared with proctored assessment. As might be expected, risks associated with non-proctored assessment center around concerns about whether candidates are receiving unauthorized help in completing the test (either by consulting with others or using various tools and resources available on the Internet or elsewhere) as well as the fact that candidates are completing assessments non-standardized in settings of their choice. Unfortunately, these discussions have been almost exclusively theoretical in nature as there has been little empirical evidence to serve as a guide.

A recent study may shed some light on this question. The paper looked at the very question of how the validity of non-cognitive assessments (personality and biodata-based tools) may change when they are moved from proctored to non-proctored modes of administration. In general, it was found that these assessments largely retained their ability to predict job performance when candidates completed them without the benefit of proctoring. Of course, this is a single study (albeit a large one with one analysis involving over 1200 candidates and the other about 3300) that only involved a handful of tests. It also should be noted that personality and biodata assessments are not objectively scored and makes “cheating” on them more difficult than knowledge-based assessments. However, these findings do support the notion that, at least in some settings, test validity may not be significantly compromised when assessments are delivered in a non-proctored fashion.

Reid Klion

Charles Handler, an I-O psychologist, assessment industry commentator, and friend of pan, recently wrote an interesting piece on “A Pre-Employment Assessment Candidate Bill of Rights” where he outlines a number of “rights” to which he believes test takers are entitled. For the sake of discussion, I will leave aside debate as to whether what are described are actually “rights” as might thought of in the same terms as the Bill of Rights or more accurately “best practices” or “good ideas.” That being said, he raises a number of provocative points, and several are worthy of comment.

Candidates have the right to an assessment experience that is of a reasonable length.

True but this is often a balancing act between collecting enough information to make a reasonable decision about an applicant’s viability as a potential employee and not overtaxing a candidate’s time Here, there can be a tension between recruiters who favor a briefer experience and I-O psychologists, tending to be a bit obsessive by nature, who will almost always opt for collecting more, rather than less, information when given a choice.

Candidates have the right to know where the assessment fits within the overall hiring process, and what they can expect next.

I couldn’t agree more that we should be transparent about the assessment process and “next steps.” This also seems important for recruitment and PR purposes since job candidates may be clients of the organizations where they are seeking work,

Candidates have the right to a good user experience.
Candidates have the right to technical support, no matter when they are applying.

Here, too, I would completely concur that assessments should employ good user interface design, be easy-to-use, and as foolproof as possible. Given the advances in web-based technologies over the past 10 years, there is little reason to not accomplish this nor to provide competent technical support if problems arise.

Candidates have the right to assessment content that appears job related.

Well, I am not necessarily convinced of this in all cases. Part of the issue is that what doesn’t appear to be job-related to candidates may very well be critical to job performance to those who understand the job. For example, someone may think that a test of cognitive ability may not be relevant to a position when it is well-known that cognitive assessments are often the best predictors of future job performance. Also, biodata assessments are often quite predictive of job performance even though the items may not necessarily appear to be job-related to an external observer.

Candidates have the right to an enjoyable assessment experience.

Another “right” I am not terribly convinced of. Our business is assessment, not entertainment. The primary goal of a testing program is to provide a fair determination of each candidate’s ability to succeed on a job. While we certainly may want to design as assessment process that is engaging, our primary goal is to provide a valid assessment of the candidate’s ability, not necessarily to provide a fun experience.

As always, Charles is a provocative writer. This article is certainly not an exception and definitely worth a look.

Reid Klion

Last week, my wife and I celebrated our one year anniversary by taking a trip to San Francisco. We spent a couple of days in the Napa/Sonoma Valley area tasting and learning about wine. Being wine novices, we were pretty fascinated by all the factors that go into the taste and quality of each of wine. It turns out there’s more to making wine than just fermenting some grapes. Not only are there the grapes themselves to consider, but the soil they’re grown in, the amount of moisture where they are grown, and even the elevation. Another major factor we learned was the importance of the type of barrel used. It was interesting to learn how a barrel made from an oak tree in France leads to a different taste than a barrel made from an oak tree in Missouri. This multitude of factors made what seemed to be a fairly straightforward process into something very complex and interesting.

Being the I/Oish person that I am, I found the complexity of factors that go into making a good wine to be not so different from the complexity of factors that go into finding a good job candidate. Now of course I’m not talking about storing candidates in barrels to see how well they age, but often times searching for a quality job candidate is mistaken as a fairly straightforward and obvious process. The truth is that most of the time there are a number of factors that go into what it takes to be successful on a job. As a result, it is important to use appropriate selection tools available to ensure that we’re accounting for these critical factors. This can include assessments, behavioral interviews, job simulations, and other kinds of job related tools. The key is to take the due diligence to appreciate the complexity of truly finding quality job candidates. Without accounting for this complexity we can be left with sour grapes.

Nate Studebaker

I take the same route to and from work every day. I’m sure others have had the same experience, but there are days when I get to work and don’t remember much of the drive. I don’t pay very much attention to my surroundings because the buildings I pass and the streets I drive on don’t change.

One afternoon at work, one of my colleagues (and fellow blogger), Reid, asked me if I had noticed the squirrel statue in the roundabout by our building. (If you aren’t familiar with a roundabout, it’s basically a four-way intersection, but it has a circular roadway around a central island). Even though Reid is a psychologist, I thought he had lost his mind. Surely if there was something as strange as a statue of a squirrel in the roundabout that I go through twice a day I would have noticed it. But sure enough, as soon as he pointed it out to me from our office window, I saw the squirrel positioned right in the middle of the roundabout. I couldn’t believe it and wondered how I could have missed something so obvious. Now I wonder what else I’ve missed not only in my surroundings but in my day-to-day activities as well.

It is easy for work processes to become like my commute – so routine that we switch to autopilot and perform our tasks without even noticing them. When was the last time you systematically evaluated your pre-employment hiring process? If you always go through the process in the same way without really looking at what you are doing, how will you make improvements? If you look for candidates without really thinking about what makes a person successful on the job, what are you missing? Perhaps we should take some time to step back and evaluate our processes and determine what it is that we are really trying to accomplish. We might find something important that needs to be altered (or at least something we hadn’t noticed before) that will help us be more successful. Feel free to call or email us so that pan can help you find your squirrel statue.

Jenni Ginsburg