Building a Context-Aware AI Doubt Solver with RAG, Knowledge Graphs, and Confidence Scoring

What if ChatGPT existed inside your college trained only on your syllabus, your faculty notes, your PDFs, and your lectures and knew when to say “I’m not confident, let me escalate this to your professor.”?

That was our hackathon challenge.

The problem statement (ED006 — Intelligent Doubt Solving Assistant) asked us to build:

• A chatbot powered by NLP + Knowledge Graphs
• Context-aware doubt resolution
• Academic content database
• Interactive Q&A
• Peer-assisted learning or mentor escalation if AI fails

We didn’t just build a chatbot.

We built a context-aware academic intelligence system.

The Real Problem in Education

Currently, most AI assistants:

• Give generic answers
• Ignore your syllabus + methodologies used by your teachers during lectures.
• Don’t understand your faculty’s teaching context
• Never admit uncertainty

Students don’t need another internet chatbot.

They need a chatbot which can actually study and understands your study material and resolves doubts accordingly.

So we asked:

Can we build a system that behaves like a smart personal Tutor not just an AI?

Our Core Idea: Context + Confidence + Escalation

We designed the system around three pillars:

1. Context-Aware RAG Pipeline

2. Confidence Scoring Engine

3. Human-in-the-Loop Escalation

Let’s break it down.

🏗 High-Level Architecture

Step 1: Faculty Uploads Content

Faculty can upload the content in the different formates like

• PDFs
• Notes
• Videos
• Course materials

We take it store into a media storage db and

• Chunk the content
• Generate embeddings
• Store them in a Vector Database
• Update the Knowledge Graph

After doing these above steps your uploaded content is ready to ask doubt’s around.At this step your AI tutor have been studied this content.

Step 2: Student Asks a Query

Now, Let’s seen How the Student query flows through our System.

If relevant content is found:

• Context is injected into the prompt.
• LLM generates a context-aware answer.

If not found:

• General reasoning fallback is used.

Step 3: Confidence Scoring (Game Changer)

This is where we differentiated ourselves.

While generating the answers AI might not found any reliable source content uploaded by faculty then, Instead of blindly answering, we:

• Compute vector similarity score
• Compare query + retrieved context
• Generate a confidence metric
• Display it on UI

If:

Confidence Score < Threshold (e.g., 60%)

— -> The system escalates to a mentor.

AI knows when it might be wrong. That’s intelligence.At every point of time you have the freedom to escalate the query to the Faculty dashboard.

Two Escalation Approaches We Designed

To answer and solve the student query we explored two models:

Approach 1:👥 Peer-to-Peer Learning

• Student posts doubt publicly
• Similar doubts are auto-linked
• Community answers
• Digital reward points
• Leaderboard & gamification

Think:

StackOverflow for your college syllabus.

We kept this as future scope.

Approach 2: 👨‍🏫Mentor-Guided Escalation (Chosen Approach)

When confidence is low:

• Query is pushed to mentor dashboard
• Faculty resolves it
• Answer updates DB
• Student receives real-time notification
• Knowledge graph gets enriched

Why we chose this:

• High-quality answers
• Contextual alignment
• Trust factor

This creates a self-improving academic AI system.

Supporting Intelligence Layer: Beyond Doubt Solving

We didn’t stop at RAG.

We extended the scope to improve the entire study environment.

1. 🎙 Voice Agents — Making AI Feel Human

To make interaction realistic and accessible:

• Speech-to-text (Whisper)
• Text-to-speech (AWS Polly)
• Conversational voice interface

Students can:

• Speak doubts naturally
• Receive spoken explanations
• Learn hands-free

This isn’t cosmetic.

It improves:

• Accessibility
• Engagement
• Retention
• Real-world usability

AI should feel like a teaching assistant — not a search box.

2. Context-Aware YouTube Feed

Instead of generic recommendations, we built a customized feed based on:

• Doubts asked
• Current course
• Concepts studied
• Weak areas identified

This turns doubt resolution into:

Continuous contextual learning.

The system doesn’t just answer.

It guides exploration.

3. Personalized Progress Analytics

We built a student intelligence dashboard:

• Weak concepts
• Strong areas
• Doubt frequency trends
• Confidence trends
• Concept heatmaps

AI becomes a mirror.

Students don’t just learn they understand how they’re learning. They can track their weak and strong areas and can make effective strategies and prepare accordingly.

4. Faculty Doubt Area Ranking System

For faculty, we added:

• Concept ranking by number of doubts
• Confusion hotspots
• Trend analytics
• Suggestions for content improvement

This transforms the platform from:

Student tool → Institutional improvement engine.

Now AI improves teaching quality.

5. Knowledge Graph Visualization

Faculty can visualize:

• Concept relationships
• Content coverage
• Interconnected topic clusters

This improves:

• Curriculum design
• Content structure
• Teaching clarity
• Feedback loops

This is where AI moves from reactive to strategic.

🛠 Our Tech Stack (Built Like a Production System)

1. Frontend

• React.js Tailwind CSS, React Icons, Video.js

2. Backend

• Node.js, Express.js, Socket.io (Real-time notifications), REST APIs

3. ML & NLP

• PyTorch, Sentence Transformers, OpenAI API, Whisper (speech-to-text), AWS Polly (text-to-speech)

4. Content Extraction

• PyMuPDF, yt-dlp, MoviePy, Playwright

5. Databases

• MongoDB, Neo4j (Knowledge Graph), Qudrant →Vector DB (Embeddings storage)

6. Deployment

• Docker, AWS EC2, ECR, GitHub Actions (CI/CD)

This wasn’t a prototype.

It was engineered like a scalable SaaS.

⚔️ The Real Challenges We Faced

This is where most hackathon blogs stop.

We won’t.

💥 1. Massive ML Docker Images (4.2GB+)

We have take the approach of building the Docker imeges and deploying it on the AWS but on every single change we have to rebuild the docker img,and agian push it on the docker hub.

Every small change meant:

• Rebuilding image
• Pushing to Docker Hub
• Redeploying on EC2

Painful.

Solution:

• CI/CD via GitHub Actions
• Optimized builds
• CPU-optimized models
• Swap memory tuning on EC2

💥 2. Limited EC2 Resources

While deploying the ML service on AWS using EC2, We only had: ~1GB RAM. This is not enough for Running heavy models. To solve this issue we also created a 2 gb Swap but still the limited RAM behaves as a bottleneck.

Fix:

• Reduced model size
• Offloaded some tasks
• Efficient embedding pipeline

This taught us real-world infra constraints > academic theory.

💥 3. YouTube Blocking Video Download Request from backend

We have the YouTube video Doubt resolver feature which get’s stucked in Production. It failed because, to vectorise the youtube video 1st we have to download it and that download is happing using yt-dlp in backend(server side). After deployment, Youtube blocks this download request from our backend server.

Solution:

Shifted download logic to client-side.

This reduced:

• Server load
• Request blocking
• Deployment complexity

What’s Next?

At this point we are done with building the web-app for maximum feasibility and usability. Now to extend the scope of the project , We are currently exploring:

• Open-source LLM deployment
• WhatsApp/Telegram bot integration
• Agent-based architecture
• Reduced model downloading dependency
• Improved accessibility

The vision?

Every college should have its own AI trained on its own brain.

What This Hackathon Really Taught Us

• Infra limits shape architecture.
• AI without confidence scoring is dangerous.
• Human-in-the-loop is not a fallback ,it’s a feature.
• Context is everything.

And most importantly:

Building is the fastest way to learn.

If you Like to read the similar blogs around Tech,Marketing and Learning Journey Make sure to Follow Rohit Mane on Medium. :)

See you in Next One!
Keep Learning !!
Keep Building!!!