This page describes a system I run for managing AI context across my work — a local-first context layer that any AI tool can read from, that ingests automatically from the places I already work, and that lets me switch between Claude, ChatGPT, or any other model without losing accumulated history.
It started as a fix for a specific frustration: heavy AI users end up locked in not to the software but to the memory. Your project history lives inside whichever tool you've been using, and switching tools means starting from zero. Tool choice gets driven by "which one has my context" rather than "which is best for this task." The system below solves that, and a few other problems along the way.
What follows is a high-level walkthrough — what it does, how it's structured, what using it day-to-day looks like, and a rough roadmap if you want to build one for yourself or your team.
The Problem
If you use AI tools heavily, you've probably noticed a pattern: you spend a surprising amount of time re-loading context. Re-uploading the same PDFs. Re-explaining the same projects. Pasting the same reference material into a new chat because the last one got too long, or because you switched tools, or because the model "forgot."
The deeper issue is structural. The intelligence in any AI workflow isn't the model — models are increasingly interchangeable. The intelligence is the context: the documents, the prior decisions, the project history, the institutional knowledge. And right now, that context lives in three bad places:
- Inside vendor memory features — opaque, non-portable, you can't see what's stored or move it elsewhere.
- In your head — meaning you re-type it into every new chat.
- Scattered across files — in Drive, in Notes, in transcripts, in emails — that you manually drag into chats one at a time.
The result is real cost: wasted tokens, wasted time, and a creeping sense that you're working for the AI tools instead of the other way around. When teams scale this up, the inefficiency compounds. One person's context can't easily be handed to another. The same client folder gets re-uploaded by three different people in three different sessions in the same week.
The premise of this system is simple: the context layer should be something you own, structured so that any AI tool can navigate it efficiently, maintained automatically so it doesn't become a chore.
What It Does
Seven things, in roughly the order they matter:
Cuts token usage dramatically
Context is stored in cascading layers of detail (more on this below). When an AI tool needs context, it loads the lightest layer first — one-line descriptions of every relevant file — and only digs deeper into the heavy material when the conversation actually needs it. Source material that used to get pasted in full now gets referenced by summary, with the original available on demand.
Controls what context bleeds into what
Context is tagged by project and bucket. When you want cross-project reasoning ("draft something similar to what we did for Project A"), you can pull across boundaries explicitly. When you don't ("answer this using only Project B material"), you can exclude everything else just as easily. No more accidental context contamination from a chat that drifted across topics.
Lets multiple people share the same context
Because the context lives on a shared server rather than inside one person's chat history, anyone with access to a project can load its context into their own AI tool. Whoever picks up the work next gets the same accumulated history the previous person had — without copy-paste, without screenshots, without "let me find that thread for you."
Stays local and under your control
Everything sensitive lives on infrastructure you own — typically a small on-prem server. Backups go to encrypted storage you control. Cloud AI tools only see what you explicitly send them in a given session, and that's drawn from your local files. No SaaS context platform sitting between you and your own data.
Ingests automatically from the tools you already use
Dropping a PDF in a watched folder gets it processed and filed. Meeting transcripts arrive from your meeting tool, get summarized, and join the right project's context. Email threads, notes, code changes — all picked up on a schedule. The manual "save this somewhere AI can find it later" step disappears.
Stays AI-vendor agnostic
The context is just files. Markdown, mostly. Any AI tool can read them. Switching from Claude to ChatGPT to a local model takes seconds — no migration, no rebuilding, no loss. You can also use different tools for different strengths in the same project without fragmenting your history.
Persists work across sessions
At the end of a working session, a script summarizes what happened and folds the new information back into the context layer. The next session — whether it's you the next morning or a colleague next week, in the same AI tool or a different one — starts with that summary already loaded. Sessions become continuous instead of starting from zero each time.
The High-Level Picture
At the broadest view, the system has three parts: sources that feed into it, a context server that processes and stores everything, and AI tools that read from it.
The Four-Tier Cascade
This is the core idea that makes the token math work. Every piece of context gets stored in four versions of decreasing size. AI tools navigate from the smallest tier down, only loading heavier material when the conversation actually requires it.
In practice, a one-hour meeting transcript might be 30,000 words raw. The Cleaned version strips the speaker tags and timestamps. The Summarized version is a one- or two-page recap of who said what and what was decided. The Gloss version is a single sentence: "Strategy call with [client] on June 12 covering Q3 launch timing and budget."
When you ask the AI a question, it can scan thousands of Gloss lines for almost nothing — and pull only the four or five Summarized files that actually matter.
The Index and the Agents
The Index
A central index file (or lightweight database) maps the entire context library, organized by bucket and project. When an AI tool needs context for a query, it consults the index first to figure out what's relevant — rather than loading everything.
A typical query path: "Help me draft an update for Project X" → Index → Project X folder → Read Gloss files (a few hundred tokens) → Identify the relevant Summarized files → Load those into context → If a specific document needs quoting, dig into Cleaned. The result: the AI knows the project deeply, having loaded maybe 1–3% of the total material.
The Agents
The system maintains itself through a small number of scheduled agents running on the server:
- Hourly ingest: scans configured data sources for new content — meeting transcripts that finished processing, files dropped in watched folders, new emails matching filters — and stages them.
- Nightly processor: runs staged files through the four-tier pipeline. Cleans, summarizes, generates a gloss line, updates the index, and commits everything to a private repo for version control.
- On-demand processor: a manual trigger for one-off work — "I need this PDF processed right now."
The agents call out to an AI API for the summarization and gloss-generation steps. This is the only ongoing cost, and it's small — you're paying to summarize each piece of content exactly once, not every time you reference it.
Using It Day-to-Day
Loading Context Into a Chat
The most common interaction. Open whichever AI tool you're working in — Claude, ChatGPT, whatever — and trigger the load. There are a few ways: a custom skill or instruction, a snippet you keep handy, or a small browser extension. The flow looks like this:
What gets loaded depends on what you're working on. A casual question pulls just the Gloss layer for the relevant project. A drafting task pulls the Summarized layer too. If you ask the AI to quote or analyze something specific, it can request that file's Cleaned version on the fly. You can also tell it explicitly — "load Project X, exclude everything else," or "compare Project X and Project Y precedents" — and the loader handles the scoping.
Saving a Session
When you finish a working session, you trigger /close-session. The AI summarizes what was discussed and decided, writes it as a new context entry, drops it into the right project folder, and updates the index. The next time anyone loads that project's context, the new summary is part of it.
This is what makes the system continuous across tools and people. A conversation you had in Claude on Monday becomes summarized context that's loaded into ChatGPT on Wednesday by a colleague — without anyone manually moving anything.
Controlling What Gets Included
Every file is tagged with the project it belongs to and any sub-tags that matter. When you load context, you specify scope:
/load-context project:X— only Project X./load-context project:X exclude:billing— Project X minus anything billing-tagged./load-context project:X,Y compare:precedent— pull precedent material from both X and Y for explicit cross-reference./load-context bucket:work— all work projects (their gloss layers only), useful when you're not sure which one is relevant.
The default behavior is conservative — load only what you asked for, nothing else. If you want cross-project reasoning, you ask for it. This is the inverse of vendor memory features, which silently mix everything together.
Physical Setup
The hardware side is intentionally modest. You don't need a data center — a quiet machine in a closet does the job.
A few notes on the physical layout. The server doesn't need to be expensive — any reliable machine with enough storage works. The two external drives give you a live working copy plus a nightly mirror, so a single drive failure is non-fatal. The offsite encrypted backup is your insurance against fire, theft, or anything else that takes out the building. The Git repo is a third line of defense, plus it gives you a full diff history of every context file — useful when you want to see how a project's record evolved.
Sensitive material can be processed entirely locally if needed, using local models instead of cloud APIs for the summarization step. Most material doesn't need this, but the option is there.
If You Want to Build One
This is a months-long build, not a weekend project — but the sequence matters more than the speed. Each phase delivers value on its own, so you don't have to finish the whole thing before benefits start showing up. Roughly:
-
Pick the server
Any always-on machine you control. Mini PC, repurposed desktop, NAS with a compute module — whatever fits the space and budget. Storage matters more than CPU; you want headroom for years of raw files.
-
Define your buckets and projects
Before writing any code, decide how context will be partitioned. Most setups end up with 2–4 top-level buckets (e.g., personal, work, side-project) and projects nested inside. Get this right early — restructuring later is annoying.
-
Build the folder structure and four-tier pipeline
Start manual. Create the folders. Process a handful of files by hand into Raw / Cleaned / Summarized / Gloss versions. You'll learn what the right summary length feels like by doing it yourself before automating.
-
Write the first ingest agent
Pick one source — usually meeting transcripts, since they're high-value and have a clean API. Get the pipeline working end-to-end for that one source: file arrives, gets cleaned, summarized, glossed, indexed, committed. This is the minimum viable system.
-
Build the index and the session scripts
The
/load-contextand/close-sessionflows. Even a basic version — a script that takes a project name and dumps the right files to your clipboard — transforms how chats feel immediately. -
Add data sources one at a time
Email next, probably. Then documents, calendars, notes, code. Each source has its own quirks, so add them serially and let each one prove itself before moving on. Some sources are easy (anything with an API). Some require workarounds (anything Apple makes). Some require migration (move from Notes to Obsidian).
-
Add team access
Once the single-user setup is solid, layer in shared access. Read-only project mounts for team members, role-based scoping, or a small internal web endpoint that serves context to authenticated requests. Start with the simplest thing that works for your team's size.
-
Refine
The system gets better with use. You'll find that some summary formats serve you better than others, that certain tags pay off more than expected, that your archive rules need tuning. Treat the architecture as a living document and revise it as patterns emerge.
A useful rule of thumb: get one source fully automated before adding the second. The temptation is to try to wire up everything at once. Don't. The discipline of completing one source end-to-end teaches you what the system actually needs.
Why It's Worth the Effort
The value compounds with each phase. Even getting meeting transcripts ingested into a summarized + gloss structure — and nothing else — eliminates a whole category of manual work. Each additional source multiplies that.
The deeper payoff is the change in posture toward AI tools. You stop thinking of them as places where work lives and start thinking of them as instruments that act on your work. The work itself — the accumulated context, the project history, the institutional memory — lives somewhere you own. When a better tool shows up next quarter, you adopt it in an afternoon. When a colleague needs to pick up where you left off, they get the same briefing you would have. When you want to be careful about what touches what, you can be — explicitly, every time.
It's a lot of work to set up. It's also the kind of system that you only need to set up once.