A RAG chatbot is a chatbot that searches your own documents before it answers. RAG stands for retrieval-augmented generation: the system retrieves the passages most relevant to a question and gives them to a language model, which writes the answer from those passages. The result is a chatbot that answers from your policies, pricing, and product documentation instead of from general internet knowledge.
What is retrieval-augmented generation?
Retrieval-augmented generation is a way of answering questions in two moves: first find the relevant text, then write the answer from it. The term was introduced in a 2020 paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. A RAG chatbot is that idea applied to a business: the text being searched is your own content.
Why ordinary AI chatbots get your business wrong
A language model knows what it saw in training. It has never read your refund policy, your onboarding guide, or last month's price list. Asked about them, it does one of two things: gives a generic answer, or produces a confident answer that is wrong.
A RAG chatbot closes that gap. It does not rely on the model's memory for facts about your business. It looks them up.
How a RAG chatbot works
There are five steps, and the first three happen before any customer asks anything. Together they are often called the RAG pipeline.
- Collect. Your source documents are gathered: PDFs, help-centre articles, Word and Google Docs, policy pages.
- Chunk. Each document is split into short passages. Good chunking keeps each passage meaningful on its own, usually with some overlap between neighbours.
- Embed and store. Each passage is turned into an embedding, a numerical representation of its meaning, and saved in a vector database.
- Retrieve. When a question arrives, it is also turned into an embedding, and the database returns the passages closest in meaning.
- Answer. The language model receives the question and the retrieved passages, with an instruction to answer only from them.
Because the knowledge lives in the documents and not in the model, you update the chatbot by updating the documents.
Embeddings are what make step 4 work. Two pieces of text with similar meaning get similar embeddings even when their wording differs, which is why a customer does not have to use the exact words in your documents. The OpenAI embeddings guide explains the idea, and the Pinecone documentation covers how a vector database stores and searches them.
RAG chatbot vs a standard chatbot
| Standard AI chatbot | RAG chatbot | |
|---|---|---|
| Source of answers | The model's training data | Your documents |
| Knows your policies and prices | No | Yes |
| How you correct it | Rewrite the prompt and hope | Fix the document |
| Can cite where an answer came from | No | Yes |
| Stays current | Only as current as the model | As current as your content |
| Setup effort | Low | Higher: needs an ingestion and retrieval pipeline |
What a RAG chatbot is good for
Customer support. Answering questions about orders, policies, features, and setup from your help content, and handing off to a person when the answer is not in the documents. This is the most common use, and it is what our AI chatbots and customer support service is built around.
Internal team assistants. Staff ask how a process works or what a policy says and get the answer from company documentation, instead of searching shared drives.
Sales and pre-sales questions. Visitors ask about services, pricing, and fit and get answers grounded in what you have published.
Portfolio and product assistants. Our founder built Ask Haseeb AI, an open-source RAG assistant that answers questions about a body of documents in real time. It ingests PDF, Markdown, and HTML files automatically from Google Drive, stores OpenAI embeddings in Pinecone, and serves answers through a FastAPI backend. The walkthrough video and source code are on our work page.
What you need before you build one
- Documents that already contain the answers. RAG retrieves; it does not invent. If the answer is not written down, the chatbot cannot give it.
- Reasonably clean content. Remove duplicates and outdated versions. Two documents that disagree produce a chatbot that disagrees with itself.
- A list of real questions. Twenty to fifty questions that customers or staff really ask is enough to test whether retrieval is working.
- A handoff path. Decide what happens when the chatbot does not find an answer: collect contact details, open a ticket, or route to a person.
If your content is a short FAQ and little else, you may not need a full pipeline yet. See how to train your AI chatbot on your FAQs for the simpler starting point.
What a RAG chatbot is made of
| Part | What it does | Common choices |
|---|---|---|
| Ingestion pipeline | Loads, cleans, and chunks documents | Python, LangChain |
| Embedding model | Converts text to vectors | OpenAI embeddings |
| Vector database | Stores and searches the vectors | Pinecone and similar |
| Language model | Writes the answer from retrieved passages | OpenAI models |
| API and interface | Receives questions and returns answers | FastAPI, a web chat widget |
Chunking choices and why they matter
Chunking is the step people skip over, and it has more effect on answer quality than most other choices. The retriever returns chunks, not documents. If the chunks are poorly cut, the right answer may exist in your content and still never reach the model.
| Choice | If you get it wrong | What usually works |
|---|---|---|
| Chunk size | Too small: a passage loses its context. Too large: several topics blur together and the match is vague | Passages long enough to hold one complete idea |
| Overlap | None: an explanation is cut in half at the boundary | Some shared text between neighbouring passages |
| Where to split | Mid-sentence or mid-table splits produce fragments | Split at headings and paragraph breaks where possible |
| Metadata | Without it, the chatbot cannot cite a source or filter old content | Store the source file, section, and date with every passage |
There is no single correct size. It depends on how your documents are written. Ask Haseeb AI uses token-based chunking of about 350 tokens with overlap, which suits its mix of PDF, Markdown, and HTML files. A long legal policy and a short FAQ may need different settings.
Some content needs special handling. A table split row by row loses its column headings. A question-and-answer page works best when each pair stays together. The way to find the right settings is to test with real questions, change one thing, and test again.
How to build a RAG chatbot: a step-by-step outline
This is the order we follow. Each step is small, and skipping one usually shows up later as a wrong answer.
- Define the audience and scope. Who will ask, and about what? One audience and one subject area is enough for a first version.
- Gather and clean the documents. Remove duplicates, old versions, and anything that should not be shown to that audience.
- Write the test questions. Collect real questions and note the expected answer and its source for each.
- Build ingestion. Load the files, extract the text, chunk it, and attach metadata.
- Embed and store. Create an embedding for each passage and save it in the vector database.
- Build retrieval. Embed the incoming question, fetch the closest passages, and decide how many to pass on.
- Write the answer prompt. Tell the model to answer only from the passages, cite the source, and say so when the answer is not there.
- Add the handoff. Define what happens when nothing relevant is found.
- Test against the questions. Fix content, chunking, or prompts, and repeat.
- Connect the channel and launch. Add the chat interface, turn on logging, and review real conversations in the first weeks.
In Ask Haseeb AI these steps map to LangChain for retrieval, a FastAPI backend, and deployment on a VPS with Gunicorn and Nginx.
Evaluating the answers
A RAG chatbot is not finished when it produces fluent answers. It is finished when the answers are correct. Check four things for every test question.
- Did retrieval find the right passage? Look at what was retrieved before you look at the answer. If the right passage is missing, the fix is in content or chunking.
- Is the answer supported by the passages? Every claim in the answer should be traceable to the retrieved text.
- Is the answer complete and correct? Compare it with the expected answer you wrote down.
- Does it refuse properly? Include questions your documents do not cover. The chatbot should say it does not know and offer the handoff.
Keep the question set and run it again whenever documents, chunking, or prompts change. After launch, add the real questions the chatbot struggled with. The set grows into a record of what your customers ask.
Keeping the knowledge base current
A RAG chatbot is as current as its content, so decide early how updates will reach it.
- Name an owner. One person is responsible for the source documents. Without an owner, content goes stale.
- Automate ingestion. New and changed files should be picked up without a developer. Ask Haseeb AI ingests files automatically from a Google Drive folder, so updating the assistant means saving a file in the right place.
- Replace, do not pile up. When a document changes, its old passages must be removed from the vector database. Otherwise the chatbot can retrieve both the old and the new price.
- Review unanswered questions. The questions that trigger a handoff are a list of content you have not written yet.
- Re-run the tests after large changes. A new price list or policy is a reason to run the question set again.
Answering in more than one language
A RAG chatbot can answer in Arabic or English from the same knowledge base. Two things make this possible. Embeddings represent meaning, so with a multilingual embedding model a question in one language can match a passage written in another. And the language model can be instructed to write its answer in the language the customer used.
This means you do not need to maintain a full copy of your documentation in every language to get started. A few cautions apply:
- Test in each language. Run your question set in both, including informal phrasing and mixed-language messages.
- Fix product names and terms. Decide how names, units, and key terms should appear in each language and put that in the prompt.
- Translate what must be exact. For legal or policy wording where the precise phrasing matters, keep an approved version in each language.
This is the basis of our Arabic AI chatbot work for businesses serving Arabic and English speakers.
Where a RAG chatbot can be deployed
The retrieval and answer logic sits behind an API, so the same chatbot can serve more than one channel.
Website widget. The usual starting point. Answers can be longer, include links, and show their sources.
WhatsApp. Businesses connect through the WhatsApp Business Platform, described in the WhatsApp Cloud API documentation. Incoming messages are passed to your backend, which runs the same retrieval and answer steps and sends the reply. Answers should be shorter and written for a phone screen. See our WhatsApp AI chatbot page.
Internal tools. A team assistant can sit in an internal web page or a team chat app, with access limited to staff.
One knowledge base serving every channel keeps answers consistent. What changes per channel is the format of the reply and how the handoff works.
Limits to know about
A RAG chatbot is more accurate than a standard chatbot, but it is not perfect.
- Retrieval can miss. If the question is worded very differently from the document, the right passage may not be returned. Testing with real questions exposes this, and better chunking usually fixes it.
- The model can misread a passage. Rare, but possible. Showing the source alongside the answer lets the reader check.
- It answers only what is documented. Gaps in your content become gaps in the chatbot. This is useful: the questions it cannot answer tell you what to write next.
RAG chatbot or AI agent?
A RAG chatbot answers. An AI agent acts. If customers need accurate answers, build a RAG chatbot. If they need something done, such as a booking changed or a record updated, you need an agent, often with RAG inside it so that it acts on correct information. We compare the two in AI agent vs chatbot, and describe how we build the second on our AI agent development page.
Do you need RAG or fine-tuning?
For business knowledge, RAG. Fine-tuning changes how a model writes, not what it knows about your company. The full comparison is in RAG vs fine-tuning.
What a RAG chatbot costs
A RAG chatbot costs more to set up than a simple FAQ chatbot because of the pipeline, and its running cost depends on how many questions it answers and how much text each one retrieves. Our guide to the cost of AI chatbots explains the parts of that bill.
Getting started
Start small: one set of documents, one audience, one list of real questions. Build the pipeline, test it against the questions, fix the content it struggles with, and launch with a clear handoff to a human. Axenor AI builds this end to end and documents the structure so your team can maintain it. See our RAG knowledge assistants page or book a free automation audit.