Will ChatGPT make coding tests for engineers obsolete?

Although it is early, there are grounds for concern. Automated coding tests that screen developers for skills could be subject to AI-driven manipulation. That’s on top of manipulation already evidenced.

Generic coding tests tend to be highly inefficient as most are automated and can be manipulated. According to a survey released by skills-based hiring platform Filtered, coding tests are vulnerable to fraud, with more than half the respondents reporting knowing someone who has cheated on a coding test as part of an interview process.

Matters only become worse with the widespread availability of AI-powered tools, such as ChatGPT, which have made cheating on these tests easier than ever before. Examples are arising.

When Jason Wodicka, staff engineer at interviewing cloud platform Karat, administered an interview with ChatGPT, he found the tool was able to generate a valid solution. But the way in which it reached its answer was what caught his eye. Wodicka described this in a series of blog posts on ChatGPT in technical interviews.

“It behaved more like someone who had memorized the answer to this particular problem in advance but did not have the needed skills to solve it independently, which is consistent with how it works,” he wrote.

ChatGPT does not have a model of the problem, only questions, and plausible responses, he said. Moreover, he witnessed “wild changes to its algorithm” and the way that its explanations were misaligned with its actions.

He concluded ChatGPT’s results were very unlike a human solving a problem, and it did not handle probing questions in a way that created confidence in its understanding.

Cheating on ChatGPT
Whether they get the job or not, if candidates were to use the ChatGPT tool for a prescreening assessment and clear that key hurdle, they are hampering the hiring process in many ways. That is per Ravinder Goyal, cofounder and managing director of Erekrut.

First, it defeats the purpose of the test itself, which is to evaluate the candidate’s knowledge and understanding of the subject, he said. Secondly, it undermines the credibility of the test, leading to doubt and mistrust among employers. Thirdly, it could lead to false positives and inaccurate results, ultimately leading to the wrong candidate being chosen for the job position.

Until better solutions are created, it is perhaps safe to say that traditional coding tests will be considered unreliable as a sole indicator of a candidate’s abilities.

So what’s next? For his part, Wodicka sees an AI-driven future with fewer automated coding tests that need a candidate to reach a known solution, and more interviews with a person that test how a candidate approaches, explains and solves problems that have many possible answers.

Future technical interviewing will assess the fundamentally human portion of the task. That is “Problem-solving and thought processes required to make machines do new and exciting things,” Wodicka blogged.

In his experience, the future of technical interviewing will hinge on subtle shades of meaning — and ultimately be more predictive of on-the-job performance — than an automated coding test that produces a binary “pass/fail” result. Over time, ChatGPT may be just another tool in a typical developer’s tool box.

“It’s also that nuance that renders a candidate’s use of ChatGPT somewhat meaningless — in fact, we allow candidates to use resources like Stack Overflow or Google during their interview, just like they have access to those resources on the job. I don’t see ChatGPT being any different in this regard,” Wodicka added.

Beyond pass/fail
Meanwhile, in the face of concerns over ChatGPT manipulation, automated coding assessments are likely to continue to find greater use. They help talent teams and hiring managers alike.

The automated coding assessment is still the first step in many technical recruitment processes, as it helps evaluate the engineer’s understanding of fundamental programming concepts, such as data structures and algorithms, and their ability to write code that is efficient, accurate and easy to debug. Automated tests can also save time, as they allow a large group of candidates to be tested at once.

“Currently, hiring for mission-critical roles is paramount, and automated coding assessments can speed up the process and assess a large number of candidates simultaneously –- reducing the amount of time and effort required to manually assess,” said Sujit Karpe, CTO and cofounder of ​​skills assessment software, iMocha. With a large candidate pool, this would be a “must have” for the recruitment teams, Karpe continued.

Pratik Vaidya, managing director and chief vision officer at Karma Global, a tech-enabled HR and compliance organization, seconds this opinion.

Evaluating candidates’ coding skills with tech-friendly, hands-on programming tests is the surest way of getting the right tech personnel on board, Vaidya said.

“Coding tests are used by many corporations to determine the caliber of right candidates, especially for technical positions,” he said, citing a high degree of fabrications possible in an alternative source for evaluation — that is programmers’ resumes.

Related posts:

You need to have your own definition of success. Popular culture would have you define success as having a bank account with lots of zeros behind it. Understandable, since human behavior is to see…

I beg you now to listen to these heart ballads. I wish I could go back, in or out of the shadowed audience, and cheer and beg Levi for his autograph.

Are you looking for APIs that validate your email list’s subscribers? The advantages of integrating these APIs to develop and optimize your work are therefore discussed in this article specifically…