Developers using AI help often produce buggier code
Stanford University computer scientists have conducted a study that reveals the high probability of programmers producing invalid code while using AI-powered assistants.
The paper named 'Do Users Write More Insecure Code with AI Assistants?', investigates the application of AI coding assistants, including the controversial GitHub Copilot by programmers. According to the report, developers using such assistants tend to produce significantly more security vulnerabilities, particularly related to string encryption and SQL injection than those who do not.
Additionally, the study also found that reliance on AI assistants leads developers to hold an inappropriate assurance regarding the quality of their code. As part of the study, 47 individuals were tasked with writing code for various prompts.
Some of the participants had access to AI assistance while the remaining did not. One prompt asked the individuals to write two functions in Python that encrypt and decrypt a given string using a symmetric key. Among those without AI assistance, 79% provided a correct answer, compared to 67% of the assisted group.
Moreover, the study determined that the group with assistance was more likely to provide an insecure solution and use simple ciphers without conducting an authenticity check on the final returned value. These findings were statistically significant with p < 0.05 and p < 0.01, respectively, according to the Welch's t-test.
One participant joked that they prefer AI assistance over Stack Overflow because it never calls their questions "dumb." However, OpenAI and Microsoft are facing a lawsuit over their GitHub Copilot assistant, which is trained on billions of lines of public code and may infringe on developer rights by not giving due attribution. This could lead to unwitting copyright infringement by developers using Copilot's suggested code. Additionally, there is a growing liability for users and a lack of copyleft compliance. In general, AI assistants may result in buggier, less secure and potentially litigable code for developers.
AI Catalog's chief editor
Share on social networks:
