Building an Intelligent Book Recommendation System
A deep dive into the complete lifecycle of a data product: from data ingestion and embedding generation to serving real-time ML recommendations with Django, PyTorch, and FAISS.
The book catalog used in this project was built from the Best Books 10k Multi-Genre Data dataset, available on Kaggle.
Hi, I'm Eduardo Sousa
I'm a Data & AI Engineer focused on building production-ready systems that turn machine learning into real-world applications. With a bachelor’s and a master’s degree in Mechanical Engineering, I developed a strong foundation in mathematical modeling and system dynamics, which shaped my analytical and problem-solving mindset. Today, I design data-driven solutions that help businesses solve complex problems, optimize processes, and transform data into scalable products that deliver real value.
Beyond CRUD: Building a True Data Product
Most applications simply store and retrieve data. Smart Library was built to answer a different question: what if your application could understand the content it hosts? What if it could learn relationships between books, understand user preferences, and make intelligent recommendations - all in real-time?
This project explores the complete data product lifecycle: from raw book data to semantic embeddings, from FAISS similarity search to fine-tuned PyTorch models, and finally to a conversational AI that can discuss books naturally. Every component was built with production readiness in mind.
🧠 The ML Architecture: Dual Recommendation System
1 2688-Dimensional Embeddings
Each book is represented by concatenating 7 different 384-dimensional embeddings:
2 Two Approaches, One Goal
Baseline: FAISS IndexFlatIP
Quick semantic similarity using raw 2688d embeddings with cosine similarity
Improved: PyTorch Fine-tuned
Contrastive learning reduces to 256d while capturing deeper patterns
Training: Contrastive Learning
Books from same genre pulled closer
Different genres pushed apart
🤖 Smart Library Agent: Conversational AI
Natural Language
Ask about books, authors, genres in plain English or Portuguese
Contextual Memory
Maintains conversation context across sessions
Real-time Streaming
Typing indicators and streaming responses for better UX
Try asking the agent:
⚙️ MLOps: From Data to Production
🛠️ Technology Stack
🎓 Key Learnings & Challenges
Embedding Generation Pipeline
Building a robust system to generate and store 2688-dimensional embeddings for thousands of books, with automatic GCP integration
Dual Approach Architecture
Implementing and comparing two recommendation strategies (FAISS baseline vs PyTorch fine-tuned) to understand trade-offs
Real-time Updates
Creating Django signals that automatically generate embeddings for new books and keep the FAISS index synchronized
MLOps Fundamentals
Versioning knowledge bases, managing model files, and creating admin interfaces for ML system management