CS2680 Modern AI Systems: Agents and System Optimizations
Course Project

Project Overview

The project is 40% of your grade and the part of the course you are most likely to remember. Build or measure something real.

Work in teams of two to three. Solo projects are allowed with instructor approval, but the scope expectation does not drop much, so teams are strongly preferred. Teams of four are not permitted.

A good project for this course does one of the following:

  • Builds a system or a substantial component — a scheduler, a cache policy, a kernel, a serving optimization — and shows it works.
  • Measures something the community assumes but has not carefully checked, and reports what you actually find.
  • Reproduces a recent paper's central result and stress-tests where it holds and where it breaks.

A negative result, honestly measured and clearly explained, is a perfectly good project. A project that reports a 3× speedup that turns out to be a measurement artifact is not. We care much more about whether your numbers are trustworthy than about whether they are impressive.

Project Ideas

These are starting points, not an assignment list. The best projects usually come from something you are already curious about. For six of these worked out in full — question, baselines, metrics, and the pitfall that would invalidate each one — see the worked examples below.

  • Characterize KV-cache reuse in a realistic multi-turn workload; quantify what prefix caching actually buys and where it fails.
  • Implement and evaluate an admission or eviction policy for a KV cache under memory pressure.
  • Measure the real cost of disaggregating prefill and decode at small scale — where is the crossover?
  • Profile a training job end to end and account for every lost GPU-second: data loading, collectives, recomputation, stragglers, checkpointing.
  • Write a fused kernel in Triton for an operator that currently is not fused, and quantify the gain against a roofline prediction.
  • Evaluate speculative decoding across draft-model choices and workloads; find where the acceptance rate makes it a loss.
  • Build a checkpointing scheme and measure recovery time against failure rate, rather than reporting only checkpoint overhead.
  • Compare quantization methods on the axis papers usually skip: not accuracy, but throughput, latency tails, and memory under real serving load.
  • Study how batching policy interacts with latency SLOs when request lengths are heavy-tailed.
  • Anything from the reading list that made you think "I do not believe that number."

Example Projects

How to read these

Six worked examples, at roughly the scope a two-to-three person team can finish in a term.

These are not a menu. They exist to show what a well-shaped project looks like: a question narrow enough to answer, a baseline worth beating, and a metric that would embarrass you if it were wrong. The best projects in a course like this usually come from something you were already curious about — these are calibration, not assignment options. Picking one as written is allowed, and you will be held to the same standard as anyone who invented their own.

Notice what every example has in common: each one names the number that would change your mind. That is the part students most often leave out, and it is the part the evaluation rubric weighs most heavily.

1. What does an agent loop actually cost?

Part I & agent serving — instrumentation over invention

The question

  • For a real agent doing a real task, where do the tokens and the wall-clock seconds go? Model forward passes, tool calls, retries, or re-sending context that has not changed?

What you would build

  • Instrument the agent from Assignment 2 (or a framework agent) to log per-step token counts, latency, cache hits, and tool time.
  • Run it over a workload of 50–100 tasks with varying difficulty and trajectory length.
  • Attribute every second and every token to a component. Nothing unaccounted for.

Baselines and metrics

  • Cost per completed task, not per request — a cheap step that fails and retries is expensive.
  • Fraction of prompt tokens that are re-sent context versus genuinely new.
  • Tail latency across trajectories, not just the mean; agents are heavy-tailed.

What a good result looks like

A breakdown that surprises someone who builds agents — for instance, showing that most tokens are re-sent context rather than reasoning, and quantifying what prefix caching would recover.
Pitfall: measuring against a hosted API where you cannot separate queueing from compute. Either run the model yourself or be explicit that your latency numbers include someone else's scheduler.

Starting points

2. What does prefix caching actually buy?

Prefix cache — a characterization project

The question

  • Reported prefix-cache hit rates come from workloads chosen by the systems' authors. On a workload you construct honestly, what is the hit rate — and where does it collapse?

What you would build

  • A workload generator with controllable sharing structure: shared system prompts, multi-turn conversations, few-shot prefixes, and adversarially unshared traffic.
  • Measurement harness against a real engine with prefix caching on and off.

Baselines and metrics

  • Hit rate, time-to-first-token, and throughput — caching helps TTFT far more than throughput, and reporting only one hides that.
  • Sensitivity to cache capacity: the interesting regime is when the cache does not fit.

What a good result looks like

A curve of benefit against sharing rate with the crossover identified, plus a clear statement of the workload assumption under which published numbers hold.
Pitfall: building a workload that shares prefixes by construction, then reporting the resulting high hit rate as a finding. Your generator's sharing structure is the result.

Starting points

3. A fused kernel against its roofline prediction

GPU programming and kernels — build and justify

The question

  • Pick an operator sequence that is not currently fused. Predict the speedup from a roofline model first, then write the kernel and see whether the prediction holds.

What you would build

  • A Triton kernel fusing the sequence, correctness-checked against the unfused reference.
  • A written arithmetic-intensity analysis committed before you benchmark.

Baselines and metrics

  • The unfused PyTorch path and, where one exists, the vendor library implementation.
  • Achieved bandwidth and FLOP/s against the hardware's roofline, not just wall-clock speedup.

What a good result looks like

Prediction and measurement side by side. If they disagree, explaining why is a better project than if they had agreed — that gap is where the real behavior lives.
Pitfall: benchmarking without warmup, without CUDA synchronization, or at a single shape. Report a shape sweep; fusion wins are extremely shape-dependent.

Starting points

4. Where does the GPU time go in a training step?

Training and data pipelines — accounting for every lost second

The question

  • On a multi-GPU training job, what fraction of GPU-seconds is spent on useful compute — and where does the rest go: data loading, collectives, recomputation, stragglers, checkpointing?

What you would build

  • A profiled training run with a per-phase breakdown, then targeted interventions (prefetch depth, checkpoint interval, recomputation policy) with before-and-after numbers.

Baselines and metrics

  • Model FLOP utilization, not "GPU utilization" — a GPU spinning on a memory-bound kernel reports high occupancy while doing very little useful work.
  • Step-time distribution across ranks, which is where stragglers hide.

What a good result looks like

A budget that sums to 100% of wall-clock, with the largest recoverable slice identified and at least one intervention measured end to end.
Pitfall: conflating occupancy with utility. Say precisely which metric you mean and how you obtained it.

Starting points

5. Where does speculative decoding stop paying?

Speculative decoding — find the crossover, not the win

The question

  • Speculative decoding is reported as a straightforward win. Under what batch size, acceptance rate, and draft-model cost does it become a loss?

What you would build

  • A harness sweeping draft-model size, speculation depth, batch size, and workload domain.
  • A simple analytic model of expected speedup, checked against measurement.

Baselines and metrics

  • Standard autoregressive decoding at the same batch size — the comparison is often made at batch size 1, where speculation looks best.
  • Acceptance rate per domain, throughput, and latency together.

What a good result looks like

A phase diagram over batch size and acceptance rate showing where the technique wins, ties, and loses — useful to anyone deciding whether to deploy it.
Pitfall: reporting speedup at batch size 1 only. Under server-side batching the arithmetic changes completely, and that is the regime production runs in.

Starting points

6. Quantization under serving load, not accuracy benchmarks

Pruning and quantization — measure the axis papers skip

The question

  • Quantization papers report accuracy retention and sometimes a microbenchmark. Under realistic serving load, what do the methods actually deliver in throughput, tail latency, and memory headroom?

What you would build

  • An end-to-end serving comparison of two or three quantization methods at matched accuracy, under a request workload with realistic length distribution.

Baselines and metrics

  • Full-precision serving at the same batch policy, plus at least one competing method — not just quantized-versus-not.
  • P50 and P99 latency, sustained throughput, and how much larger a batch the memory saving buys.

What a good result looks like

Evidence about whether the memory saving converts into serving capacity, which is the claim practitioners actually care about and the one most often left unmeasured.
Pitfall: comparing methods at different accuracy levels. Match accuracy first, then compare systems metrics, or the comparison means nothing.

Starting points

What these have in common

A question, not a topic. "KV-cache optimization" is a topic. "At what sharing rate does prefix caching stop paying for its memory?" is a question, and you can tell when you have answered it.

A baseline someone would defend. The weakest part of most project reports is a baseline chosen because it was easy to beat. Pick the one a skeptical reader would demand.

A metric tied to a decision. Every example above measures something that would change what an engineer does on Monday.

A stated failure mode. Each has a pitfall that would invalidate the result. Knowing yours in advance is most of the work — see the paper reading strategy for the same habit applied to other people's papers.

Related

Grading Criteria

Criterion Excellent (90-100%) Good (75-89%) Adequate (60-74%) Needs Improvement (<60%) Weight
Problem & Motivation Sharp, well-scoped question that matters; clearly situated against existing work Reasonable question with adequate context Vague question or thin engagement with related work No clear question; unaware of obvious prior work 15%
Technical Execution Substantial implementation or measurement effort; sound engineering; reproducible Solid work with minor gaps in rigor or completeness Limited scope, or implementation that only partly works Little working code or measurement 30%
Evaluation Quality Right baselines, right metrics, error bars, ablations; anticipates and rules out artifacts Reasonable methodology with some missing controls Weak baselines or single-run numbers presented as fact Unsupported or unreproducible claims 30%
Report & Presentation Clear writing, effective figures, honest about limitations Understandable with minor clarity or figure issues Hard to follow; figures do not support the claims Incoherent or missing key sections 25%
On honesty: reporting a limitation costs you nothing here. Hiding one that we find costs you a great deal. Every strong systems paper has a "where this does not work" paragraph; yours should too.

Project Timeline

Milestone Due Weight What to submit
Team formation Oct 6, 2026 — Team members and a one-line topic direction
Proposal Oct 22, 2026, 11:59pm 5% One-page PDF: problem, why it matters, plan, what you will measure, what could go wrong
Checkpoint Nov 19, 2026, 11:59pm 5% Two-page PDF: progress, preliminary numbers, revised plan
Presentation Dec 1 & Dec 3, 2026 10% 12-minute talk plus 3 minutes of questions
Final report Dec 9, 2026, 11:59pm 20% 6-8 page PDF plus a code repository

The checkpoint exists so that a project heading in a bad direction can be redirected while there is still time. Use it honestly — a checkpoint reporting "this did not work and here is why" gets full credit and a useful conversation.

Report Formatting

  • Use the ACM sigconf template (LaTeX or Word), two-column, 10pt.
  • Final report: 6-8 pages excluding references. Proposal: 1 page. Checkpoint: 2 pages.
  • Figures must be legible in print and in grayscale. Label your axes, including units.
  • Report the hardware, software versions, and workload for every number you present.
  • Include an AI-use disclosure statement, per the AI use policy.

Deliverables

  • Report — PDF, submitted to Canvas by one team member.
  • Code — a repository with a README that explains how to reproduce your main result. Someone else should be able to run it.
  • Data and scripts — the plotting scripts and raw measurements behind your figures.
  • Slides — PDF, uploaded before your presentation slot.

How to submit

All submissions go through Canvas. Assignment links will be posted here and announced on the home page as each milestone opens.

Compute Resources

Access instructions will be posted here once allocations are confirmed. If your project needs hardware beyond what is listed, talk to the instructor early — sometimes it can be arranged, but not in the last two weeks.

Harvard FASRC (Cannon)

The FAS Research Computing Cannon cluster has GPU partitions available to Harvard courses. Plan for queue time: a job that takes an hour to run may wait considerably longer than that to start, which matters a great deal in the last week of the term.

CloudLab

CloudLab provides bare-metal machines for research and teaching, free of charge, with full control over the software stack. This is the right choice when your project needs to touch the kernel, the network, or the storage layer, or when you need reproducible isolation that a shared cluster cannot give you.

Other options

  • Your research group's cluster, if you have one and your advisor agrees.
  • Cloud credits — check with the instructor before spending your own money on this course.
  • A single consumer GPU is enough for many good projects. Scope to what you have.

AI Resources

Useful starting points for project work:

  • vLLM and SGLang — production-quality serving engines, and readable enough to modify.
  • Triton — write GPU kernels without writing CUDA.
  • Nsight Systems and the PyTorch profiler — find out where the time actually goes before you optimize.
  • The Ultra-Scale Playbook — a practical guide to distributed training that pairs well with the lectures.
  • MLPerf — for how the field defines and reports performance, and for baselines worth arguing with.