👾How it works

Botbee leverages state-of-the-art AI technologies to provide a seamless, multilingual customer support experience. Here’s a detailed breakdown of how Botbee works, including the technical architecture and sample code snippets.

1. Data Ingestion and Indexing

Web Crawler: The process begins with a web crawler that scrapes data from the B2B company's public website. This data includes text content, images, and other relevant information.

LlamaIndex API: The scraped data is then sent to the LlamaIndex API for processing and indexing. LlamaIndex helps in ingesting, structuring, and retrieving data efficiently.

import os
from llama_index import SimpleDirectoryReader, GPTSimpleVectorIndex

# Set up environment variables
os.environ["OPENAI_API_KEY"] = "your_openai_api_key"
os.environ["LLAMA_CLOUD_API_KEY"] = "your_llama_cloud_api_key"

# Load data from the website
documents = SimpleDirectoryReader('./data').load_data()

# Create an index
index = GPTSimpleVectorIndex(documents)
index.save_to_disk('index.json')

Pinecone Vector DB: The indexed data, including text embeddings and metadata, is stored in Pinecone, a vector database optimized for semantic search and retrieval.

import pinecone

# Initialize Pinecone
pinecone.init(api_key="your_pinecone_api_key", environment="your_pinecone_environment")

# Create an index
index_name = "botbee-index"
pinecone.create_index(index_name, dimension=1536)

# Upsert data into Pinecone
index = pinecone.Index(index_name)
index.upsert(vectors=[(doc.id, doc.embedding) for doc in documents])

2. Avatar Generation

NVIDIA ACE: NVIDIA's Avatar Cloud Engine (ACE) is used to create a custom AI avatar. This avatar is tailored to match the B2B company's branding and persona.

from nvidia_ace import AvatarBuilder

# Create a custom avatar
avatar = AvatarBuilder()
avatar.set_persona("business_persona")
avatar.set_appearance("professional")
avatar.save("custom_avatar")

3. Query Processing

Voice Input: Users interact with Botbee through voice input, which is captured and processed by the system.

OpenAI GPT-4o: The voice input is transcribed into text and sent as a query to the OpenAI GPT-4o model. GPT-4o processes the query and generates a response.

import openai

# Transcribe voice input to text
voice_input = "path_to_voice_input_file"
transcription = openai.Audio.transcribe("whisper-1", voice_input)

# Generate response using GPT-4o
response = openai.Completion.create(
    model="gpt-4o",
    prompt=transcription['text'],
    max_tokens=150
)

Pinecone Retrieval: GPT-4o sends a semantic search query to Pinecone to retrieve relevant text embeddings and metadata from the indexed company data.

# Retrieve relevant data from Pinecone
query_embedding = openai.Embedding.create(input=transcription['text'], model="text-embedding-ada-002")
results = index.query(queries=[query_embedding['data'][0]['embedding']], top_k=5)

Meta CM3leon: For queries requiring visual output, GPT-4o leverages Meta's CM3leon multimodal model to generate images based on text prompts.

from cm3leon import CM3leon

# Generate image based on text prompt
cm3leon_model = CM3leon()
image = cm3leon_model.generate_image(prompt=response['choices'][0]['text'])

Voice Output: The text response generated by GPT-4o (and any images generated by CM3leon) is passed to the NVIDIA ACE avatar, which delivers the response verbally and with appropriate facial expressions and gestures.

from nvidia_ace import Audio2Face

# Convert text response to speech
speech = openai.TextToSpeech.create(text=response['choices'][0]['text'], voice="en-US-Wavenet-D")

# Animate avatar with speech
avatar.animate(speech)
avatar.display()

Language Translation: If the user selects a different language, the response can be translated using GPT-4o's multilingual capabilities or a dedicated translation service.

# Translate response to selected language
translated_response = openai.Translation.create(
    text=response['choices'][0]['text'],
    target_language="es"
)

Mixture of Experts (MoE) with Expert Choice Routing

Botbee's uniqueness lies in its use of a novel Mixture of Experts (MoE) architecture combined with Expert Choice Routing for multilingual interpretation and avatar interaction. This approach ensures optimal load balancing and efficient utilization of specialized models for different languages and domains.

Algorithm Overview:

1. Data Ingestion: Use a web crawler to collect data and LlamaIndex API to index it.

2. Vector Storage: Store indexed data in Pinecone for efficient retrieval.

3. Expert Selection: Implement Expert Choice Routing to dynamically select the most appropriate expert models based on the query's context and language.

4. Response Generation: Use OpenAI GPT-4o to generate responses, leveraging the selected expert models.

5. Multimodal Output: Integrate Meta's CM3leon for visual responses and NVIDIA ACE for avatar interaction.

Mathematical Model:

• Expert Choice Routing: Optimize the routing of tokens to experts using a combination of k-means clustering and linear assignment to maximize token-expert affinities.

• Load Balancing: Ensure balanced training and inference loads across experts to prevent under or over-specialization.

import numpy as np
from sklearn.cluster import KMeans
from scipy.optimize import linear_sum_assignment

# Example of Expert Choice Routing
def expert_choice_routing(embeddings, experts):
    # Cluster embeddings
    kmeans = KMeans(n_clusters=len(experts))
    clusters = kmeans.fit_predict(embeddings)
    
    # Assign tokens to experts
    cost_matrix = np.zeros((len(embeddings), len(experts)))
    for i, embedding in enumerate(embeddings):
        for j, expert in enumerate(experts):
            cost_matrix[i, j] = np.linalg.norm(embedding - expert.centroid)
    
    row_ind, col_ind = linear_sum_assignment(cost_matrix)
    assignments = col_ind
    
    return assignments

# Example usage
embeddings = np.random.rand(100, 768)  # Example embeddings
experts = [ExpertModel() for _ in range(10)]  # Example expert models
assignments = expert_choice_routing(embeddings, experts)

Conclusion

Botbee's innovative use of Mixture of Experts with Expert Choice Routing, combined with advanced AI technologies from OpenAI, Meta, and NVIDIA, provides a unique and powerful solution for multilingual customer support. This approach ensures efficient, personalized, and scalable support experiences, setting Botbee apart in the market. The proposed algorithm and architecture offer a strong basis for patent consideration, highlighting Botbee's commitment to cutting-edge innovation in AI-driven customer support.