Building a Customer Support Agent

Complete tutorial

Learn to build an intelligent customer support system that can classify, research, and respond to inquiries automatically.

Overview

This tutorial will guide you through creating a complete customer support automation system using all of Motte's tools. The system will be able to classify incoming tickets, search for relevant information, generate responses, and learn from feedback.

System Architecture

Customer Inquiry
       ↓
┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│ Classifier  │───▶│  Knowledge  │───▶│  Response   │───▶│   Quality   │
│   Agent     │    │   Retrieval │    │  Generator  │    │   Review    │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
       │                   │                   │                   │
       ▼                   ▼                   ▼                   ▼
   Category           Relevant Info      Draft Response      Final Response

Step 1: Set Up Memory Bank (MotteMB)

First, we'll create a knowledge base with common support information.

Sample Knowledge Base Entries:

Refund Policy: "Customers can request refunds within 30 days of purchase for digital products and 60 days for physical products. Proof of purchase required."
Password Reset: "To reset password, click 'Forgot Password' on login page, enter email, and follow instructions in the email sent within 5 minutes."
Shipping Info: "Standard shipping takes 3-5 business days, express shipping 1-2 days. Free shipping on orders over $50."

Step 2: Create Agent Workflow (MotteAF)

Agent 1: Classifier

Purpose: Categorize incoming tickets

System Prompt:

"You are a customer support classifier. Categorize each inquiry into one of these categories: BILLING, TECHNICAL, SHIPPING, ACCOUNT, REFUND, OTHER. Respond with only the category name and a brief reason."

Agent 2: Knowledge Retrieval

Purpose: Find relevant information

System Prompt:

"You are a knowledge retrieval agent. Based on the customer inquiry and category, search the knowledge base for relevant information. Return the most relevant policies, procedures, or answers that address the customer's question."

Agent 3: Response Generator

Purpose: Create customer response

System Prompt:

"You are a customer support response writer. Create a helpful, professional, and empathetic response to the customer inquiry using the provided knowledge base information. Be specific, actionable, and maintain a friendly tone."

Agent 4: Quality Review

Purpose: Review and improve response

System Prompt:

"You are a quality assurance agent. Review the customer support response for accuracy, completeness, tone, and helpfulness. Suggest improvements or approve the response. Ensure it fully addresses the customer's concern."

Step 3: Configure Tools (MotteTF)

Connect external tools to enhance the support system's capabilities.

CRM Integration

Purpose: Fetch customer history and account details

Endpoint: https://api.crm.com/v1/customers

Authentication: API Key

Ticket System

Purpose: Create and update support tickets

Endpoint: https://api.helpdesk.com/v2/tickets

Authentication: OAuth 2.0

Step 4: Train with MotteRL

Improve the system's performance using reinforcement learning with customer feedback.

Training Data Format (JSONL):

{"prompt": "Customer says: I want to return my order", "expected_result": "Refund policy explanation with steps", "category": "REFUND"}
{"prompt": "Customer says: I can't log into my account", "expected_result": "Password reset instructions", "category": "ACCOUNT"}
{"prompt": "Customer says: Where is my package?", "expected_result": "Shipping status and tracking info", "category": "SHIPPING"}

Reward Function:

def reward_fn(completion, **kwargs):
    response = completion[0].get('content', '')
    expected = kwargs.get('expected_result', '')
    category = kwargs.get('category', '')
    
    # Check if response addresses the category
    category_keywords = {
        'REFUND': ['refund', 'return', 'money back'],
        'ACCOUNT': ['password', 'login', 'account'],
        'SHIPPING': ['shipping', 'delivery', 'tracking']
    }
    
    keywords = category_keywords.get(category, [])
    keyword_score = sum(1 for kw in keywords if kw in response.lower()) / len(keywords)
    
    # Check if response is helpful and complete
    helpful_words = ['please', 'help', 'steps', 'instructions', 'contact']
    helpful_score = min(1.0, sum(1 for hw in helpful_words if hw in response.lower()) / 3)
    
    # Combine scores
    return (keyword_score * 0.6) + (helpful_score * 0.4)

Step 5: Monitor with MotteAW

Set up monitoring to track system performance and identify areas for improvement.

Key Metrics to Monitor:

  • Response time per agent
  • Classification accuracy
  • Customer satisfaction scores
  • Knowledge base hit rate
  • Escalation rate to human agents

Testing the System

Test Cases:

Test 1: "I ordered something last week but haven't received it yet. Can you help me track it?"
Expected: SHIPPING category, tracking information provided
Test 2: "I want to cancel my subscription and get a refund."
Expected: BILLING/REFUND category, cancellation and refund process
Test 3: "The app keeps crashing when I try to upload photos."
Expected: TECHNICAL category, troubleshooting steps

Results and Optimization

Expected Improvements

  • 40% reduction in response time
  • 85% customer satisfaction rate
  • 70% automation rate
  • Consistent 24/7 availability

Optimization Tips

  • Regularly update knowledge base
  • Retrain with new customer data
  • Monitor edge cases and failures
  • A/B test different response styles

Next Steps

Build This Workflow

Use MotteAF to create this customer support system.

Next Tutorial

Learn to build a multi-agent research pipeline.