The AI assistant
The assistant doesn't just chat about your plugin — it builds the graph, tunes parameters, writes your UI source, scaffolds DSP blocks, and checks its own work. Here's what it can touch and how to steer it.
For plugin makers who want to get the most out of AI-assisted building.
It has hands, not just opinions
The assistant in the chat panel operates directly on your open project. Ask for "a warm tape delay with a one-knob UI" and it doesn't describe one — it places the blocks, wires the cables, sets the parameters, and writes the interface, while you listen to the patch take shape live.
What it can actually do
| Capability | What that looks like |
|---|---|
| Build and rewire the graph | Add, move, and remove blocks; connect and re-route cables; validate the result |
| Shape the sound | Set and expose parameters, wire modulation (LFOs, envelopes), change voice count |
| Add musical content | Load samples into players, generate or load wavetables, write melodies into the MIDI player, drive the arp and step sequencer for auditioning |
| Create custom DSP blocks | Scaffold a new block under blocks/<name>/ with descriptor, Cmajor DSP source, and optional custom node visuals — compile-checked before it lands |
| Edit your UI | Read and rewrite ui/src/Plugin.svelte, eject built-in components into your project, verify every control still binds to a real parameter |
| Check its own work | Read compile status and console logs, screenshot the UI canvas and graph nodes, check layout bounds, inspect live telemetry |
| Use your shell | Run commands in your project folder — git, npm, file inspection, anything you approve |
Everything operates on plain files in your project, so anything the AI builds you can inspect, edit, and version-control afterwards.
How it sees your project
Every turn, the assistant gets a live snapshot: the full node graph with parameter values, the compile status, your plugin's canvas size, and which helpers are running. For anything deeper it reads your actual project files — the graph, your UI source, your custom blocks — and it pulls in focused expertise (Cmajor pitfalls, modulation routing, layout, polyphony) on demand rather than guessing.
It also looks at its own output. After writing a UI it can screenshot the canvas and check for overflowing elements; after a rewire it can read the generated DSP source to confirm the connection really landed.
Teach it your conventions
Every project ships three markdown files the assistant treats as the contract for what it's building:
CLAUDE.md— the project's notebook: plugin facts, working rules, and a notes section the AI updates as the project evolvesbrief.md— the vision: what it should sound like, how it should feel to useparameter-spec.md— the parameter table: ids, ranges, defaults
These are yours to edit. Write "all filters are ladder filters" or "UI stays monochrome, no gradients" into CLAUDE.md and the assistant follows it from then on. Vague briefs produce vague plugins — sharpening these files is the highest-leverage steering you have.
Tip
After a session goes well, ask the assistant to record what worked in CLAUDE.md. Future sessions start from that knowledge instead of rediscovering it.
Choosing a model
The model picker offers a range from fast-and-cheap (the default) up to frontier reasoning models, each labeled with its relative cost. Cheap models handle routine graph edits and parameter tweaks fine; reach for a stronger model when you're designing a complex custom block or a full UI from scratch. If you attach an image to a model that can't see, Patchwerk automatically routes that message to a vision-capable one.
Credits and your account
AI usage runs through your Patchwerk account — no API keys to manage. Each request costs credits based on the model's token pricing (the per-model rates are shown right in the picker). Billing is idempotent per request, so a flaky connection and a retry never double-charges you.
You stay in control of the shell
Graph and file edits are scoped to your open project. Shell commands are gated: each command the assistant wants to run is shown to you for approval first, with an opt-in per-chat bypass when you trust the session. Nothing executes behind your back.
Power move: bring your own agent
Because a Patchwerk project is just files, you can drive it from an external coding agent — Claude Code, Cursor, or anything else — instead of (or alongside) the built-in chat.
Two routes, both shipped inside every project:
- File-based. Patchwerk writes an
AGENTS.mdguide plus a.patchwerk/context bundle (block library manifest, graph schema, deep-dive skill docs) into your project on every open. An external agent editsdsp/graph.jsonorui/src/Plugin.svelte, the running app hot-reloads and recompiles, and the agent reads the compile result back from.patchwerk/status.json. - Live MCP tools. Patchwerk exposes a local MCP server so an external agent can call the same graph tools the built-in assistant uses — adding nodes, wiring modulation, setting params, creating blocks, checking compiles — with live validation instead of raw file edits.
Note The MCP route requires the Patchwerk app to be running; the file-based route only needs the project folder. Both can be mixed freely with the in-app assistant — it all converges on the same project files.