原文要点

I've been looking beyond the hardware side of local ai lately, and I am trying to figure out what people are actually using once the server itself is up and running. Running a local model is one thing, but turning it into something useful for private documents seems to involve another layer entirely, document ingestion, embeddings, retrieval, knowledge-base management, and keeping everything updated. Testing fast gpt as one option for this layer because it brings the knowledge-base and AI application workflow together. For those running ai locally, are you building this stack yourself, or using an existing platform? I'm especially interested in what has worked well for larger document collections and what hardware/software bottlenecks you've encountered.

原文完整内容

I've been looking beyond the hardware side of local ai lately, and I am trying to figure out what people are actually using once the server itself is up and running.

Running a local model is one thing, but turning it into something useful for private documents seems to involve another layer entirely, document ingestion, embeddings, retrieval, knowledge-base management, and keeping everything updated.

Testing fast gpt as one option for this layer because it brings the knowledge-base and AI application workflow together.

For those running ai locally, are you building this stack yourself, or using an existing platform?

I'm especially interested in what has worked well for larger document collections and what hardware/software bottlenecks you've encountered.

原帖关键讨论

perseus-computing 2 points

We run a local stack, and the DIY-vs-platform split matters less than which pieces you're willing to maintain yourself.

FastGPT is a reasonable middle ground: knowledge base and workflow UI in one place, and it points at a local model without much fuss. Its limit for bigger collections is retrieval tuning: you get chunk-size control and vector+keyword search, but you outgrow the built-in defaults and the workflow editor starts being the product instead of the tool. On the platform side, RAGFlow is worth a look if document parsing quality is the bottleneck; Open WebUI's RAG is fine for small sets but gets unwieldy as the corpus grows.

For larger collections, the things that reliably move the needle:

  • Hybrid retrieval (dense embeddings + BM25). Pure vector search quietly misses exact matches, IDs, and rare terms.
  • A reranker after first-pass retrieval: the cheapest quality-per-effort win in the whole stack.
  • Chunking tuned per document type instead of one size for everything.
  • Treating stale-document handling as first-class. When a doc gets updated or contradicts an older one, most stacks will happily cite the stale chunk. Provenance should be a requirement, not a feature.

Bottlenecks tend to be: PDF/table parsing at ingestion (CPU-bound), embedding throughput on CPU (fine for hundreds of docs, slow for tens of thousands), RAM pressure from index + model + reranker on one box, and reranker latency on long candidate lists.

The layer I've ended up most interested in is the one between retrieval and the answer: keeping durable facts, corrections, and provenance separate from session context so updates supersede stale content instead of coexisting with it. I'm building Perseus Vault around that: governed facts with correction, supersession, and review rather than treating every retrieved chunk as truth. It complements a retrieval stack rather than replacing one.

CapnAhab_1 1 points

Not quite the answer to your question, but this is an excellent tutorial on setting up your own! It explains a lot of the theory so could give you a great grounding

https://youtu.be/Q1GnzRs3RbI?is=F2mGHHFz1RKLxeaG

来源：https://www.reddit.com/r/LocalAIServers/comments/1vnzygz/what_software_stack_are_you_using_for_private/