Copilot finished my function

GitHub opened a technical preview of Copilot last week. It’s a plugin for VS Code that suggests code as you type, from single lines to whole functions, based on the file you’re in and comments you’ve written. It runs on Codex, a model from OpenAI descended from GPT-3 and trained on public code from GitHub.

I got off the waitlist faster than I expected and spent a couple of evenings with it on a side project, a small Python script for organizing photos by the date and location in their metadata.

The first surprise: it’s good at the boring parts, and most code is the boring parts. I wrote a comment saying “parse the EXIF date from a JPEG and return a datetime,” and it wrote the function, including the fallback for when the field is missing, using the library I had already imported. I wrote the signature of a function to group photos by day and it filled in the body correctly. For code that looks like code that has been written thousands of times, it’s almost always right, and fast.

The second: it’s confidently wrong in exactly the ways GPT-3 was. It called a method that doesn’t exist on the object, with a very plausible name. It used an older version of an API. It wrote a date calculation that was off by one around midnight. If I hadn’t been reading every line, I’d have shipped bugs. It doesn’t know anything about my code beyond the open file, so as soon as a function depends on something defined elsewhere, it guesses.

The third is harder to describe. The way I write code changed within an hour. I started writing more comments first, as instructions, because the comment was now an input. I started naming things more clearly, because good names led to better suggestions. Programming became something closer to a conversation, where I describe intent and review proposals, than typing out every character.

I’ve managed engineers for years, and it felt a lot like reviewing a junior developer’s pull request, only one that arrives in half a second. That’s where I think the danger and the opportunity both are. Reviewing code is harder than writing it, and people get lazy about it. A tool that writes a lot of plausible code quickly moves more of the work into review, and if people treat suggestions as correct, quality will drop.

But the direction seems clear. Last summer I said coding would change first because code is text with strong structure. I’d now guess that within a few years most new code is at least partly suggested by a model, and that the skills that matter more for engineers are specifying intent clearly and reviewing well.