Multimodal AI Applications

Multimodal AI processes multiple types of input — text, images, audio, and video — in a single model. This enables powerful applications that understand and generate across modalities.

What is Multimodal AI?

Traditional LLMs work with text only. Multimodal models can:

Input:  Image + Text → Output: Text
Input:  Audio        → Output: Text (transcription)
Input:  Text         → Output: Image (generation)
Input:  Video + Text → Output: Text (analysis)

Image Understanding

Analyzing Screenshots

python

from openai import OpenAI
import base64

client = OpenAI()

def analyze_image(image_path, question):
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode("utf-8")

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": question},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/png;base64,{image_data}"
                        }
                    }
                ]
            }
        ],
        max_tokens=1000
    )
    return response.choices[0].message.content

# Usage
result = analyze_image(
    "ui-mockup.png",
    "Describe this UI design. What improvements would you suggest for UX?"
)

Comparing Images

python

def compare_images(image1_path, image2_path, comparison_prompt):
    images = []
    for path in [image1_path, image2_path]:
        with open(path, "rb") as f:
            images.append(base64.b64encode(f.read()).decode("utf-8"))

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": comparison_prompt},
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{images[0]}"}
                    },
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{images[1]}"}
                    }
                ]
            }
        ]
    )
    return response.choices[0].message.content

# Usage
diff = compare_images(
    "design-v1.png",
    "design-v2.png",
    "Compare these two UI designs. What changed? Which is better?"
)

Practical Vision Applications

Document Processing (OCR + Understanding)

python

def process_document(image_path):
    result = analyze_image(
        image_path,
        """Extract all text from this document image.
        Then organize it into a structured JSON format with:
        - document_type (invoice, receipt, form, letter)
        - key_fields (extracted key-value pairs)
        - total_amount (if applicable)
        - date (if found)"""
    )
    return result

Chart/Graph Analysis

python

def analyze_chart(chart_path):
    return analyze_image(
        chart_path,
        """Analyze this chart/graph:
        1. What type of chart is it?
        2. What data does it show?
        3. What are the key trends or insights?
        4. Extract approximate data points if possible."""
    )

UI to Code

python

def ui_to_code(screenshot_path, framework="React with Tailwind CSS"):
    return analyze_image(
        screenshot_path,
        f"""Convert this UI screenshot into {framework} code.
        - Match the layout, colors, and typography as closely as possible
        - Use responsive design
        - Add appropriate class names
        - Include placeholder content where text is visible"""
    )

Audio Processing

Speech-to-Text with Whisper

python

def transcribe_audio(audio_path):
    with open(audio_path, "rb") as audio_file:
        transcript = client.audio.transcriptions.create(
            model="whisper-1",
            file=audio_file,
            response_format="text"
        )
    return transcript

# With timestamps
def transcribe_with_timestamps(audio_path):
    with open(audio_path, "rb") as audio_file:
        transcript = client.audio.transcriptions.create(
            model="whisper-1",
            file=audio_file,
            response_format="verbose_json",
            timestamp_granularities=["segment"]
        )
    return transcript

Audio Analysis Pipeline

python

def analyze_meeting(audio_path):
    # Step 1: Transcribe
    transcript = transcribe_audio(audio_path)

    # Step 2: Analyze with LLM
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "Analyze meeting transcripts and extract key information."
            },
            {
                "role": "user",
                "content": f"""Analyze this meeting transcript:

{transcript}

Provide:
1. **Summary** (2-3 sentences)
2. **Key Decisions** made
3. **Action Items** (who, what, deadline)
4. **Open Questions** that weren't resolved"""
            }
        ]
    )
    return response.choices[0].message.content

Text-to-Speech

python

def text_to_speech(text, voice="alloy", output_path="output.mp3"):
    response = client.audio.speech.create(
        model="tts-1",
        voice=voice,  # alloy, echo, fable, onyx, nova, shimmer
        input=text
    )
    response.stream_to_file(output_path)
    return output_path

Image Generation

With DALL-E

python

def generate_image(prompt, size="1024x1024"):
    response = client.images.generate(
        model="dall-e-3",
        prompt=prompt,
        size=size,
        quality="standard",
        n=1
    )
    return response.data[0].url

Building a Multimodal Application

Example: Product Description Generator

python

def generate_product_listing(image_path, category=None):
    """Analyzes a product image and generates a complete listing."""

    # Step 1: Analyze the product image
    analysis = analyze_image(
        image_path,
        "Describe this product in detail: color, material, size, "
        "features, condition. Be specific."
    )

    # Step 2: Generate listing content
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": "You are an e-commerce product listing expert."
            },
            {
                "role": "user",
                "content": f"""Based on this product analysis:
{analysis}

Category: {category or 'Auto-detect'}

Generate a complete product listing with:
1. Title (SEO-optimized, under 80 characters)
2. Description (3 paragraphs: overview, features, specifications)
3. Bullet points (5 key selling points)
4. Suggested tags (5-8 keywords)
5. Suggested price range"""
            }
        ],
        temperature=0.6
    )
    return response.choices[0].message.content

Multimodal Model Comparison

Capability	GPT-4o	Claude Sonnet	Gemini 1.5 Pro
Image input	Yes	Yes	Yes
Image generation	Yes (DALL-E)	No	Yes (Imagen)
Audio input	Yes (Whisper)	No	Yes (native)
Audio output	Yes (TTS)	No	Yes
Video input	No	No	Yes
PDF input	Yes	Yes	Yes

Best Practices

Compress images before sending — reduce API costs without losing quality
Use the right modality — don't send an image when text suffices
Combine modalities thoughtfully — image + specific question > image alone
Handle errors gracefully — models may refuse unclear or inappropriate images
Consider privacy — be careful with images containing personal information

Key Takeaways

Multimodal AI unlocks new applications that text-only models can't handle
Vision + LLM is the most mature combination — use it for document processing, UI analysis, and product descriptions
Audio pipelines combine transcription + LLM analysis for meeting notes and content creation
Gemini leads in multimodal breadth (video, audio), while GPT-4o and Claude excel in quality
Build pipelines that chain modalities for complex workflows

With a solid foundation in building AI applications, the next module dives into RAG — connecting LLMs to your own data for accurate, grounded responses.