Building a Multi Retail GenAI Assistant Using Web Scraping, RAG Fusion & Conversational Memory
In today’s competitive fashion retail industry, customers expect instant, intelligent and aggregated answers across multiple brands. Traditional search on e-commerce websites is limited to single brand queries, leaving users to manually compare prices, styles and availability across retailers.
To address this, we built a GenAI powered assistant that scrapes product data from M&S, Zara and H&M websites in real time and uses Query Transformation + RAG Fusion to deliver accurate, multi brand retail insights. This assistant also maintains conversational memory, enabling it to answer follow up queries seamlessly.
Author
Ashutosh
Suryawanshi
I’m an AI Engineer with over 8 years of experience in the tech industry. I began my career as a Full-stack Developer, building end-to-end applications across various platforms. Over time, I transitioned into AI Engineering, focusing on developing production-ready AI solutions using tools like Databricks Mosaic AI Framework, LangChain, and MLflow. Focusing on building practical AI applications for real-world use cases.
Connect with
Ashutosh
Suryawanshi
Get a free recap to share with colleagues
What is Lorem Ipsum?
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged.
Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged.
Retail customers are increasingly turning to AI powered shopping assistants for speed and convenience. By solving the multi brand query challenge, businesses can provide personalised and faster shopping experiences, drive customer loyalty through accurate and aggregated insights, achieve scalability and automation in catalog management and reduce manual comparison effort while enhancing decision making. This solution is directly applicable to retail aggregators, fashion marketplaces, and virtual stylists who want to deliver differentiated user experiences.
Solution
The GenAI assistant was designed using a modular, scalable pipeline.
Architecture Workflow:
Data Collection: Scrape structured data (products, designers, categories, prices) from Zara, H&M and M&S using Playwright, Selenium and BeautifulSoup.
Data Processing: Clean, enrich and chunk product metadata.
Embedding + Storage: Create vector embeddings with Azure OpenAI and store them in ChromaDB.
Query Processing: Apply Query Transformation + RAG Fusion to generate precise and multi source answers.
Conversational Memory: Maintain context for follow up queries.
User Interface: Deploy as a chatbot (Gradio / Streamlit) for interactive exploration.
Step by Step Walkthrough
Step 1: Retail Web Scraping Pipeline
Extracts product data from Zara, H&M and M&S websites.
Handles dynamic content, infinite scrolling and structured HTML parsing.
Normalises metadata into a unified product schema.
Implements retry mechanisms, proxy rotation and dynamic waits for robustness.
Stores enriched data in Delta Lake for history tracking and retrieval.
Step 2: Chunking & Embedding
Logical chunking of data (per product, per category).
Uses Azure OpenAI embeddings for vector representation.
Stored in ChromaDB for retrieval augmented generation.
Step 3: Query Transformation + RAG Fusion
Handles vague or shorthand queries by reformulating them into precise prompts.
RAG Fusion merges brand specific answers and reranks results based on confidence and brand tags.
Ensures consistent, comparative insights (e.g., price ranges across brands).
Step 4: Conversational Memory
Uses LangChain memory to maintain session history.
Supports natural follow ups like:
“Show me similar jackets under ₹4000”
“Now compare only Zara and M&S.”
Step 5: User Interface
Deployed as a chatbot for real time retail exploration.
Built using Gradio or Streamlit for accessibility.
Results
Unified search across multiple retail sites.
Near real time updates with live scraping.
Higher accuracy using RAG Fusion + Query Transformation.
Context aware conversations with memory.
Scalable, modular pipeline for future retail domains.
Conclusion
By combining web scraping, vector search, RAG Fusion and conversational memory, we developed a multi retail GenAI assistant that bridges the gap between fragmented brand data and aggregated customer insights.
This solution can be extended beyond fashion into electronics, groceries or travel, any sector where multi source product comparison is valuable. Want to explore how this could work for you? Book a quick call.
