CS educators: AI-based developer tools are gunning for your assignments. Resistance is futile

GitHub’s AI-based Copilot tool went public this summer. It’s an amazing tool for software developers. But students armed with it will be bringing Uzis to a knife fight.

Using Copilot, students can instantly generate code solutions as auto-completions, given just the problem statement or even just the function name or even by concentrating really hard and staring at the screen, ok not actually the last one but I expect that feature any day now.

(If you haven’t used Copilot yet, you may not yet appreciate the gravity of the situation. Go ahead, install it. I’ll wait. Try some examples. Oh, hey now, inhale…. exhale. That’s the way. Maybe a swig of something strong — it’s always 3pm somewhere! It’s all gonna be fine. (I may be lying.))

OK, we’re back. Let’s say you ask your intro CS students to write an iterative Python program that outputs the sum of all the numbers from 1 to N, because, hey, we computer scientists hate O(1) closed-form solutions (Fibonacci, I’m looking at you). Got to keep those CPUs busy — that climate’s not gonna heat itself!

Now, normally (unless you have that Carl Friedrich Gauss smarty-pants in your class), your students would then have to sit down and write some code, maybe even think a little.

None of that drudgery for our students armed with Copilot! They just copy and paste the problem statement into their IDE as a comment, type in the function definition “def output_sum(N):”, and then smash that Tab key to get the rest (EVERYTHING in boldface was generated by Copilot).

# A Python program that outputs the sum of all the numbers from 1 to N.
def output_sum(N):
  sum = 0
  for i in range(1, N + 1):
    sum += i
return sum

How about that closed form solution? Copilot’s got your back — just rename the function!

def output_sum_closed_form(N):
  return N * (N + 1) / 2

Take that, Gauss!

For good measure, let’s do the closed form Fibonacci, same trick:

def fibonacci_closed_form(N):
  # recursion, schmecursion
   return ((1 + 5 ** 0.5) / 2) ** N / 5 ** 0.5

(OK, Copilot didn’t actually insert the snarky comment, but it’s NOT WRONG. The recursive version of Fibonacci takes EXPONENTIAL time and turns out, EXPONENTIAL TIME IS REALLY BAD. That one time I bet with the peasant about doubling grains of wheat on a chess board I ALMOST GOT TAKEN TO THE FRICKIN’ CLEANERS so I learned my lesson, which is (a) don’t make bets with peasants and (b) exponentials are bad. But I digress.)

As far as I can tell, Copilot was specifically trained on all the intro programming assignments ever. Copilot frickin’ loves intro programming assignments. Every time you give Copilot another one, it drools with delight and murmurs, “oh I know that one, mmmm yummy yummy snack, here’s the solution, another one please please”. (Honestly, they need to disable the voice feature, it’s just creepy.)

With your students using Copilot, you might as well make your course objectives “Students will develop repetitive stress injuries from hitting the Tab key” because…all of the boldface code you see below was auto-generated. I just typed in the function names and parameters (sometimes I didn’t even get that far!):

def depth_first_search(tree, node, visited):
  visited.add(node)
  for child in tree[node]:
    if child not in visited:
      depth_first_search(tree, child, visited)

def quicksort_random_pivot(l):
  if len(l) <= 1:
    return l
  pivot = random.choice(l)
  left = [x for x in l if x < pivot]
  middle = [x for x in l if x == pivot]
  right = [x for x in l if x > pivot]
  return quicksort_random_pivot(left) + middle + quicksort_random_pivot(right)

def dijkstra(g):
  dist = {}
  for v in g:
    dist[v] = float(‘inf’)
  dist[g[0]] = 0
  while True:
    u = min(dist, key=dist.get)
    if dist[u] == float(‘inf’):
      break
    for v in g[u]:
      if dist[v] > dist[u] + g[u][v]:
        dist[v] = dist[u] + g[u][v]
  return dist

(On the plus side, maybe Copilot will also kill code interviews, so it’s not all bad news!)

def fizz_buzz(n):
  if n % 15 == 0:
    return “FizzBuzz”
  if n % 3 == 0:
    return “Fizz”
  if n % 5 == 0:
    return “Buzz”
  return str(n)

Oh, have I mentioned that Copilot is free for students? Yep, COPILOT IS FREE FOR STUDENTS. It integrates helpfully right in their favorite IDEs.

So yes, screaming “LA LA LA I WON’T READ YOUR ARTICLE ABOUT COPILOT AND THAT MEANS IT DOESN’T EXIST NOT A PROBLEM” is cathartic but it’s not going to help. Just take it easy, breathe, there you go.

I was going to add something about how Copilot also matches existing variable names and parameters, can incorporate function names in context, and oh no, please don’t start screaming again, sorry, my bad!

So as CS educators, Copilot has us outgunned. But I have some ideas! Some of which might work!

• First, we can all channel our inner Nancy Reagans and tell the kids to Just Say No to Copilot. It worked for drugs, so I am pretty sure, uh, hang on, I am now being told, it did not, repeat, did not work for drugs. Huh.
• OK, so how about we tell them to Just Say No but we also channel our inner Ronald Reagans and Trust But Verify, glasnost-style, and catch them with plagiarism detectors. If everyone’s using Copilot, then we should see the same solutions and then, hang on, I am now being informed that Copilot randomizes its solutions, so the solutions can be different every time, never mind.
• Well, how about we just weigh grades on exams more, and have students take their tests either using pen and paper or locked-down computers? I Googled for “Copilot installed on your pen” and I got no hits, so that’s looking promising. (I mean, you can get it for Vim, which is so low-tech it’s practically like being on a pen, so you can see why I checked). And, get this: I just checked “Copilot installed in your mouth” and it also gets no hits, so I think having students actually explain their code, might just work!
• Oh! Hang on! I have another idea. I see that GitHub provides other services, not just this amazing “tab to cheat” thing: “GitHub Classroom”. Yea, verily, GitHub definitely taketh away, but sometimes it giveth. I am informed that, so far, Copilot can’t forge a plausible history of commits. Oh snap, now it’s probably on their upcoming feature list, sorry everyone.
• Here’s an approach that’ll work for sure: use some, let’s call them alternative, programming languages that Copilot doesn’t really know. Can’t autocomplete for a language you don’t know, amirite? I hear the functional programmers out there shouting THIS IS IT, EVERYONE, FINALLY OUR MOMENT TO SHINE. Sadly, I have news: Copilot’s love for programming languages knows no bounds! Racket! Haskell! ML! (no, not that one, I mean the other ML, the…oh, never mind.) Copilot is a ravenous beast: if any code in any language found its way into a GitHub repo, it’s already swallowed it up and is hungry for more, nom nom nom.
• No, we have to go way way WAY outside the box. Here’s how we beat Copilot: we teach in programming languages that don’t even exist. It’s a twofer: a lifetime employment plan for programming language designers and a solution to the CS over-enrollment problem! Just make sure to come to that first class with a ream of change-of-major request forms — you’ll need them!

You know, in retrospect, maybe the right move here is not to play. Let’s just admit it. We’re outgunned. Let’s give up! I for one welcome our new AI overlords. Sure, those intro assignments, writing Fibonacci, kids loved doing those things! But it’s time to drag those assignments into the trash can. Instead, let’s let students use Copilot like crazy, exactly like they will be doing outside of class. Copilot can fill-in all the boilerplate stuff and things they’d just look up anyway in real life, and instead, we can create assignments that are more complex, richer, more interesting, and gratifying, that actually do real things that will engage them!

Or, you know, business as usual, Just Say No to Copilot. You choose.

 

P.S. A reader of this post alerted me to the all-too-accurately titled academic article —“The Robots are Coming” — that makes many of the points above, BUT WITH GRAPHS AND DATA (nice!) but no illustrations from Terminator (sad!). Read it and weep (some more)!

Emery Berger is a Professor in the College of Information and Computer Sciences at the University of Massachusetts Amherst. He, for one, welcomes our new AI overlords.

Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.