CS2680 Modern AI Systems: Agents and System Optimizations
Assignments

Assignments

Two hands-on assignments in Part I, before the systems material begins. Both are individual.

Assignment 1 — Build something with an agent (10%)

Out Sep 15 · due Sep 29, 11:59pm

Use an existing agent framework to build something that does real work for you. The point is not the artifact — it is the write-up: where the agent succeeded, where it failed, and which of those failures were the model's fault versus the surrounding system's. Keep the transcripts; they are the evidence.

Assignment 2 — Build an agent (10%)

Out Sep 17 · due Oct 8, 11:59pm

Now write the loop yourself rather than importing one: tool calling, context management, retries and error recovery, and a stopping condition. A small agent you fully understand is worth more here than a capable one you assembled from parts. Report where your implementation costs the most tokens and the most wall-clock time — you will need that instinct for Part II.

Assignment 3 — Make your agent 3× cheaper (10%)

Out Oct 8 · due Oct 20, 11:59pm

Take the agent you built in Assignment 2 and make it measurably cheaper to run. Three steps, in this order — and the order is part of the grade:

  1. Profile. Instrument your agent and account for where every token and every second goes across a fixed set of tasks: model calls, tool time, retries, and context you re-send that has not changed. Nothing unattributed.
  2. Predict, then lock it in. Before you change a line of code, submit a short written prediction: which optimization you will apply, why you expect it to be the biggest win, and roughly how much you expect it to save. This is timestamped and cannot be revised afterwards.
  3. Optimize and measure. Apply the change, re-run the same task set, and report the difference. Same tasks, same model, same measurement method — otherwise the comparison means nothing.

Techniques worth considering: parallelise or batch tool calls, restructure prompts so a stable prefix can be reused, stop re-sending unchanged context, route easy steps to a smaller model, cache tool results, or simply stop earlier. Which one wins depends on your profile — that is the point of doing step 1 first.

3× is the title, not the bar. A carefully measured 1.4× that you can fully explain scores better than a hand-waved 3×. Being wrong in your prediction costs almost nothing; being wrong and not noticing costs a great deal.

How the 10% breaks down: a profile accounting for essentially all of the cost (3%); the quality of your locked prediction — the reasoning, not whether it turned out right (2%); a measured, honestly reported improvement (3%); and the write-up, including what you tried that did not work (2%).