ChatGPT really isn’t great at answering programming questions yet
Generative AI isn’t all that good at helping developers for now
When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.
Despite becoming an overnight hit when it launchedChatGPTis still struggling to excel in some areas - particularly assisting coding, new research has claimed.
Positioned as an ideal solution to programming issues, some developers have been making use of an array of generative AI tools such as GitHub’s Copilot to speed up workflow, freeing up more time to focus on productive work.
However a newstudyfrom researchers at Purdue University found more than half (52%) of the responses that ChatGPT produced are incorrect.
ChatGPT helping with coding
The researchers analyzed 517 questions from Stack Overflow and compared ChatGPT’s answers to human responses, finding that the AI’s errors were widespread. In all, more than half (54%) were conceptual misunderstandings, around one in three (36%) were factual inaccuracies, a similar number (28%) were logical mistakes in code, and 12% were terminology errors.
In the paper, ChatGPT was also criticized for producing unnecessarily long and complex responses containing more detail than what was needed, leading to potential confusion and distractions. However, the ultra-small-scale poll of 12 programmers saw one-third preferring ChatGPT’s articulate and textbook-like answers, highlighting the ease with which coders can be misled.
The implications of these findings are pretty significant, because errors in coding can ultimately lead to bigger problems downstream, affecting multiple departments or organizations.
The writers summarize: “Since ChatGPT produces a large number of incorrect answers, our results emphasize the necessity of caution and awareness regarding the usage of ChatGPT answers in programming tasks.”
Are you a pro? Subscribe to our newsletter
Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!
Besides exercising caution, the researchers also call for further research into identifying and mitigating such errors, as well as greater transparency and communication around potential inaccuracies.
More from TechRadar Pro
With several years’ experience freelancing in tech and automotive circles, Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars and the decarbonisation of personal transportation. As an avid bargain-hunter, you can be sure that any deal Craig finds is top value!
LG Electronics sets ambitious B2B revenue goal to offset declining consumer demand
New fanless cooling technology enhances energy efficiency for AI workloads by achieving a 90% reduction in cooling power consumption
Phishing attacks surge in 2024 as cybercriminals adopt AI tools and multi-channel tactics
