I kept seeing "agentic AI" everywhere this year, in job postings, in course catalogues, on LinkedIn. I even picked up a CrewAI certification because it was on the syllabus. But I noticed I could describe what an agent framework does without being able to explain what actually happens underneath it. So I closed the framework docs and built one from scratch instead: ai-research-assistant, a small FastAPI backend that answers questions using a local knowledge base and an LLM, with zero LangChain, zero CrewAI, and no hidden agent loop I couldn't read line by line.

Why build another RAG bot

Retrieval-Augmented Generation is the "hello world" of applied LLM engineering, and that's exactly why I picked it. I didn't want to learn a framework's opinion of what an agent is; I wanted to build the opinion myself, then go back and see how much of it CrewAI and LangChain were doing for me automatically. So the rule I set myself was simple: every step in the pipeline had to be a function I wrote and could step through in a debugger.

The shape of the pipeline

The whole system boils down to one comment I left at the top of agent.py:

Question -> Retriever -> LLM -> Answer + Sources

A question comes in through a /chat endpoint, gets handed to a ResearchAgent, which retrieves relevant documents, builds a prompt out of them, asks the LLM to answer using only that context, and returns the answer alongside the exact sources it used:

class ResearchAgent:
    async def run(self, question: str) -> str:
        docs = retriever.retrieve(question, top_k=settings.top_k)
        context = "\n\n".join(f"[{d.source}]: {d.content}" for d in docs)
        prompt = PROMPT_TEMPLATE.format(context=context, question=question)
        answer = await self.llm_client.generate(prompt)
        return {"answer": answer, "sources": [d.source for d in docs]}

Four steps, no branching, no tool-calling loop deciding what to do next. That was deliberate: "agentic" doesn't have to mean unpredictable. It just means the model's output feeds a workflow instead of a chat bubble.

Retrieval without a vector database

My knowledge base is four markdown files (agentic_ai.md, rag.md, fastapi.md, llm.md) sitting in a docs/ folder. Standing up a vector database for that felt like using a shipping container to move a backpack, so I reached for something older: TF-IDF and cosine similarity, straight out of scikit-learn.

corpus = [doc.content for doc in self.documents]
self.vectorizer = TfidfVectorizer(stop_words="english")
self.doc_matrix = self.vectorizer.fit_transform(corpus)

def retrieve(self, query: str, top_k: int = 2) -> list[Document]:
    query_vec = self.vectorizer.transform([query])
    scores = cosine_similarity(query_vec, self.doc_matrix).flatten()
    ranked_idx = scores.argsort()[::-1]
    return [self.documents[i] for i in ranked_idx[:top_k] if scores[i] > 0]

It has no semantic understanding of synonyms, no embeddings, nothing fancy. But for a small, static knowledge base it's fast, dependency-light, and, crucially for someone learning, easy to reason about when it retrieves the wrong document. Right-sizing the tool to the problem turned out to be its own lesson in engineering judgment.

An LLM client you can swap without touching the agent

I wanted to develop against a free, local model and still be able to point the same code at a hosted one later, so the LLM call sits behind an abstract base class:

class LLMClient(ABC):
    @abstractmethod
    def generate(self, prompt: str) -> str: ...

class OllamaClient(LLMClient):
    async def generate(self, prompt: str) -> str:
        # POST to localhost:11434/api/generate
        ...

class OpenAIClient(LLMClient):
    async def generate(self, prompt: str) -> str:
        # AsyncOpenAI chat.completions.create
        ...

def get_llm_client() -> LLMClient:
    provider = settings.llm_provider.lower()
    return OllamaClient() if provider == "ollama" else OpenAIClient()

Day to day I ran everything against Ollama running llama2 locally, which meant I could iterate on prompts for free without burning API credits. Flipping LLM_PROVIDER=openai in the .env file is the only change needed to swap in GPT-4 instead — the router, the retriever, and the agent don't know or care which one answered.

Grounding the answer, and making it show its work

The part that actually taught me something about "agentic" behavior was the prompt itself:

PROMPT_TEMPLATE = """You are a helpful research assistant. Answer the user's \
question using ONLY the context below. If the context does not contain the \
answer, say you don't have enough information.

Context:
{context}

Question: {question}

Answer:"""

Constraining the model to the retrieved context, and forcing it to admit when it doesn't know, is most of what keeps this system from hallucinating. Pairing that with returning the source filenames in the API response (ChatResponse has answer and sources) means every answer is checkable — if the assistant cites fastapi.md for a question about vector databases, that's an immediate signal something went wrong in retrieval, not generation. Separating those two failure modes was the single most useful debugging habit I picked up from this project.

Testing an agent without paying for tokens

I didn't want CI runs (or my own sanity) depending on a live model, so the API tests mock the agent instead of the LLM call itself:

with patch(
    "app.services.agent.ResearchAgent.run",
    new=AsyncMock(return_value={"answer": "FastAPI is a web framework.", "sources": ["fastapi.md"]}),
):
    resp = client.post("/chat", json={"question": "What is FastAPI?"})
assert resp.status_code == 200

Combined with a Pydantic schema that rejects empty questions with a 422 before the agent is ever called, the whole HTTP layer is testable offline in milliseconds. It's a small thing, but it's the difference between a project I trust and a project I'm afraid to touch.

The one Docker wall I hit

Running the API in a container while Ollama stayed on the host machine broke silently — localhost:11434 inside a container doesn't mean what it means on your machine. The fix was pointing OLLAMA_BASE_URL at host.docker.internal instead, one of those gotchas that's obvious for five seconds after you find it and completely opaque before that.

So what does "agentic" actually mean now

One of the docs in my own knowledge base is the definition I was chasing when I started this:

Agentic AI refers to systems where a language model follows a multi-step
workflow to accomplish a task, often involving tool use and decision-making,
rather than just responding to a single prompt.

There's a small, slightly circular pleasure in the fact that the tool I built to understand that definition can now retrieve and explain that exact definition to someone else. But the real takeaway is that "agentic" isn't a framework feature — it's an architectural choice to make the model one step in a pipeline instead of the whole pipeline. LangChain and CrewAI give you that structure pre-built, with routing, memory, and tool-calling loops handled for you. Having written the four-step version by hand first, I can now open a CrewAI crew and actually recognize what each abstraction is standing in for, instead of trusting it on faith.

What's next

  • Swap TF-IDF for sentence embeddings once the knowledge base outgrows "four markdown files"
  • Let the agent decide whether retrieval is even necessary for a given question, instead of always retrieving
  • Add a second tool (web search or a calculator) so "decision-making" in the pipeline has more than one branch to choose from
  • Stream the answer back token by token instead of waiting for the full response

Try it

The code is on GitHub: github.com/BishantRajbanshi/ai-research-assistant. Clone it, drop your own markdown files into docs/, run Ollama locally, and uvicorn app.main:app --reload gives you a research assistant scoped to whatever you fed it. If you're trying to understand agentic AI the same way I was, I'd skip the framework tutorials for a weekend and just build the four-step pipeline yourself first. It's a much shorter path to actually understanding what you're automating.