ChatGPT's Accuracy in Code Questions: Worse Than a Coin Flip

A study conducted by Purdue University found that OpenAI's chatbot, ChatGPT, provides incorrect answers to software programming questions 52% of the time and is verbose in 77% of its responses. However, despite the inaccuracies, ChatGPT's well-articulated language style and comprehensiveness still make it preferred by users 39.34% of the time. The study also revealed that even when errors were obvious, participants still marked ChatGPT's responses as preferred due to its pleasant and authoritative style. The authors suggest that Stack Overflow should improve the discoverability of their answers and incorporate methods to detect toxicity and negative sentiments. Stack Overflow's usage has reportedly declined, with some attributing it to the popularity of ChatGPT.