Case study · Askbase
A knowledge assistant that answers from company documents, with sources and permissions.
An internal assistant for support and engineering staff that answers questions from manuals, specifications, and past tickets, cites its sources, and only uses documents each person is allowed to see.
- Sector
- Engineering & technical services
- Services
- Scope
- Evaluation to rollout
- Core stack
- Python, LLM APIs, pgvector, PostgreSQL, Next.js
This case study shows how we would approach a representative project of this kind. It is not a record of a specific client engagement: names, interfaces, and plans are examples, and outcomes are described as what would be measured rather than as results.
Scroll sideways to see the full screen
§ 01The situation
Where things stand.
A technical services company supports complex equipment for its customers. The knowledge needed to resolve a support case is spread across product manuals, engineering specifications, a wiki, and thousands of past tickets.
New support staff take months to become productive, and senior engineers are interrupted many times a day with questions that have been answered before. A generic chatbot was tried and abandoned after it gave confident answers that were wrong.
What makes it hard
Accuracy over fluency
A plausible but wrong answer about equipment can cause real damage, so answers must be grounded and verifiable.
Permissions
Some documents are restricted to certain teams or customers and must never appear in another person's answer.
Changing content
Manuals and specifications are revised regularly, and the assistant must not answer from outdated versions.
Proving value
Leadership wants evidence that the assistant helps, not anecdotes.
§ 02The solution
What we would build.
Each part answers one of the problems above, and each can be delivered and tested on its own.
- Content pipeline
- Documents from the wiki, file storage, and ticketing system indexed continuously, with versions tracked and superseded content excluded.
- Permission-aware retrieval
- Search runs with the user's own access rights, so restricted documents are filtered out before the model ever sees them.
- Answers with citations
- Answers are built only from retrieved passages, with numbered citations that link to the exact source section.
- Knowing when not to answer
- When the sources do not support an answer, the assistant says so and suggests who to ask instead.
- Evaluation & feedback
- A test set of real questions with expert-approved answers, plus feedback from users that feeds the next evaluation round.
- In the tools people use
- Available inside the support tool and team chat, so staff do not need to open yet another application.
§ 03Architecture
How the pieces fit.
A layered view of the proposed system. The highlighted component carries the logic the rest of the system depends on, so it is designed and tested first.
01 Sources
- Wiki
- File storage
- Ticketing system
- Product manuals
02 Indexing
- Connectors & sync
- Chunking & versioning
- Embeddings (pgvector)
03 Answering
- Permission-aware retrieval
- LLM with grounded prompts
- Citation builder
04 Delivery & quality
- Web app & chat
- Feedback capture
- Evaluation pipeline
- Usage analytics
§ 04Delivery plan
Delivered in stages you can review.
Each stage ends with something working that you can evaluate before committing to the next.
Stage 01
3 weeks
Evaluation set
Collect real questions and expert answers, and agree what a good answer looks like.
Stage 02
4 weeks
Prototype
Retrieval and answer generation measured against the evaluation set.
Stage 03
6 weeks
Pilot
One support team, with permissions enforced, citations, and a feedback loop.
Stage 04
Ongoing
Rollout
More teams and sources, chat integration, and regular re-evaluation.
§ 05Outcomes
What you receive, and how success is measured.
Targets for each measure are agreed during discovery, using your current figures as the baseline, and reported on throughout delivery.
Deliverables
- Assistant web application and chat integration
- Content connectors with permission sync
- Evaluation set and scoring pipeline
- Usage and quality dashboard
- Administration guide for content owners
Measured by
- Answers rated correct by experts on the evaluation set
- Answers backed by verifiable citations
- Time to resolve support cases
- Questions escalated to senior engineers
§ 06Start a project
Planning something similar?
Describe where things stand today and what needs to change. We will reply with questions and an initial view of scope and approach.
- 01
Share the context
A short description of the project, your goals, and any constraints you already know about.
- 02
We review it
We read it carefully and reply with questions or an initial perspective on the problem.
- 03
A first conversation
A call to explore scope, approach, and whether we are the right fit for the work.