vertex-ai-media-master - Comprehensive Multimodal AI Operations

This Agent Skill provides comprehensive mastery of Google Vertex AI multimodal capabilities for video, audio, image, and text processing with focus on marketing applications.

Core Capabilities

🎥 Video Processing (Gemini 2.0/2.5)

• Video Understanding: Process videos up to 6 hours at low resolution or 2 hours at default resolution
• 2M Context Window: Gemini 2.5 Pro handles massive video content
• Audio Track Processing: Automatic audio transcription from video
• Multi-video Analysis: Process multiple videos in single request
• Video Summarization: Extract key moments, scenes, and insights
• Marketing Use Cases:
• Analyze competitor video ads
• Extract highlights from long-form content
• Generate video summaries for social media
• Transcribe and caption video content
• Identify brand mentions and product placements

🎵 Audio Generation & Processing

• Lyria Model (2025): Native audio and music generation
• Speech-to-Text: Transcribe audio with speaker diarization
• Text-to-Speech: Generate natural voiceovers
• Music Composition: Background music for campaigns
• Audio Enhancement: Noise reduction and quality improvement
• Marketing Use Cases:
• Generate podcast scripts and voiceovers
• Create audio ads and radio spots
• Produce background music for video campaigns
• Transcribe customer interviews
• Generate multilingual voiceovers

🖼️ Image Generation (Imagen 4 & Gemini 2.5 Flash Image)

• Imagen 4: Highest quality text-to-image generation
• Gemini 2.5 Flash Image: Interleaved image generation with text
• Style Transfer: Apply brand styles to generated images
• Product Visualization: Generate product mockups
• Campaign Assets: Create ad creatives and social media graphics
• Marketing Use Cases:
• Generate personalized ad images (Adios solution)
• Create social media graphics at scale
• Produce product lifestyle images
• Generate A/B test variations
• Create branded campaign visuals

📢 Marketing Campaign Automation

• ViGenAiR: Convert long-form video ads to short formats automatically
• Adios: Generate personalized ad images tailored to audience context
• Campaign Asset Generation: Photos, soundtracks, voiceovers from prompts
• Content Pipeline: Email copy, blog posts, social media, PMax assets
• Catalog Enrichment: Multi-agent workflow for product onboarding
• Marketing Use Cases:
• Automated campaign asset production
• Personalized content at scale
• Multi-channel content distribution
• Product catalog enhancement
• Visual merchandising automation

🔧 Technical Implementation

API Integration:

python
from google.cloud import aiplatform
from vertexai.preview.generative_models import GenerativeModel


# Initialize Vertex AI
aiplatform.init(project="your-project", location="us-central1")


# Gemini 2.5 Pro for video
model = GenerativeModel("gemini-2.5-pro")


# Process video with audio
response = model.generate_content([
    "Analyze this video and extract key marketing insights",
    video_file,  # Up to 6 hours
])


# Imagen 4 for image generation
from vertexai.preview.vision_models import ImageGenerationModel
imagen = ImageGenerationModel.from_pretrained("imagen-4")
images = imagen.generate_images(
    prompt="Professional product photo, studio lighting, white background",
    number_of_images=4
)

Gemini 2.5 Flash Image (Interleaved Generation):

python
# Generate images within text responses
model = GenerativeModel("gemini-2.5-flash-image")
response = model.generate_content([
    "Create a 5-step recipe with images for each step"
])
# Returns text + images interleaved

Audio Generation (Lyria):

python
from vertexai.preview.audio_models import AudioGenerationModel
lyria = AudioGenerationModel.from_pretrained("lyria")
audio = lyria.generate_audio(
    prompt="Upbeat background music for product launch video, 30 seconds",
    duration=30
)

📊 Marketing Workflow Automation

1. Multi-Channel Campaign Creation:

python
# Single prompt generates all assets
campaign = model.generate_content([
    """Create a product launch campaign for [product]:
    - Hero image (1920x1080)
    - 3 social media graphics (1080x1080)
    - 30-second video script
    - Background music description
    - Email marketing copy
    - Instagram caption"""
])

2. Video Repurposing Pipeline:

python
# Long-form to short-form conversion (ViGenAiR approach)
long_video = "gs://bucket/original-ad-60s.mp4"
response = model.generate_content([
    f"Extract 3 engaging 15-second clips from this video for TikTok/Reels",
    long_video
])
# Auto-generates format-specific versions

3. Personalized Ad Generation:

python
# Context-aware image generation (Adios approach)
for audience in audiences:
    ad_image = imagen.generate_images(
        prompt=f"Product ad for {product}, targeting {audience.demographics}, {audience.style_preference}",
        aspect_ratio="16:9"
    )

🎯 Best Practices for Jeremy

1. Project Setup:

bash
# Set environment variables
export GOOGLE_CLOUD_PROJECT="your-project-id"
export GOOGLE_APPLICATION_CREDENTIALS="path/to/service-account.json"


# Install SDK
pip install google-cloud-aiplatform[vision,audio] google-generativeai

2. Rate Limits & Quotas:

• Gemini 2.5 Pro: 2M tokens/min (video processing)
• Imagen 4: 100 images/min
• Monitor usage in Cloud Console

3. Cost Optimization:

• Use Gemini 2.5 Flash for faster, cheaper operations
• Batch image generation requests
• Cache video embeddings for repeated analysis
• Use low-resolution video setting when appropriate

4. Security & Compliance:

• Keep API keys in Secret Manager, never in code
• Use service accounts with minimal permissions
• Enable VPC Service Controls for data residency
• Log all API calls for audit trails

🚀 Advanced Marketing Use Cases

1. Campaign Performance Analysis:

python
# Analyze competitor campaigns
competitor_videos = ["gs://bucket/competitor1.mp4", "gs://bucket/competitor2.mp4"]
analysis = model.generate_content([
    "Compare these competitor videos: themes, messaging, CTAs, production quality",
    *competitor_videos
])

2. Content Localization:

python
# Generate multilingual campaigns
for lang in ["en", "es", "fr", "de", "ja"]:
    localized_content = model.generate_content([
        f"Translate and culturally adapt this campaign for {lang} market:",
        campaign_brief,
        hero_image
    ])

3. A/B Test Generation:

python
# Generate variations automatically
variations = []
for style in ["minimalist", "bold", "luxury", "playful"]:
    variation = imagen.generate_images(
        prompt=f"Product ad, {style} style, {brand_guidelines}",
        number_of_images=1
    )
    variations.append(variation)

📚 Reference Documentation

Official Documentation:

• Vertex AI Multimodal: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/overview
• Gemini 2.5 Pro: https://cloud.google.com/vertex-ai/generative-ai/docs/models
• Imagen 4: https://cloud.google.com/vertex-ai/generative-ai/docs/image/overview
• Video Understanding: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/video-understanding

Marketing Solutions:

• GenAI for Marketing: https://github.com/GoogleCloudPlatform/genai-for-marketing
• ViGenAiR (video repurposing)
• Adios (personalized ad images)

Pricing:

• Gemini 2.5 Pro: $3.50/1M input tokens, $10.50/1M output tokens
• Imagen 4: $0.04/image
• Video processing: Included in Gemini token pricing

When This Skill Activates

This skill automatically activates when you mention:

• Video processing, analysis, or understanding
• Audio generation, music composition, or voiceovers
• Image generation, ad creatives, or visual content
• Marketing campaigns, content automation, or asset production
• Gemini multimodal capabilities
• Vertex AI media operations
• Social media content, email marketing, or PMax campaigns

Integration with Other Tools

Google Cloud Services:

• Cloud Storage for media asset management
• BigQuery for campaign analytics
• Cloud Functions for automation triggers
• Vertex AI Pipelines for content workflows

Third-Party Integrations:

• Social media APIs (LinkedIn, Twitter, Instagram)
• Marketing automation platforms (HubSpot, Marketo)
• CMS integrations (WordPress, Contentful)
• DAM systems (Bynder, Cloudinary)

Success Metrics

Track These KPIs:

• Asset generation speed (baseline: 5 images/min)
• Content approval rate (target: >80%)
• Campaign personalization scale (target: 1000+ variants)
• Cost per asset (target: <$0.10/image)
• Time saved vs manual production (target: 90% reduction)

This skill makes Jeremy a Vertex AI multimodal expert with instant access to video processing, audio generation, image creation, and marketing automation capabilities.

Prerequisites

• Access to project files in {baseDir}/
• Required tools and dependencies installed
• Understanding of skill functionality
• Permissions for file operations

Instructions

1. Identify skill activation trigger and context
2. Gather required inputs and parameters
3. Execute skill workflow systematically
4. Validate outputs meet requirements
5. Handle errors and edge cases appropriately
6. Provide clear results and next steps

Output

• Primary deliverables based on skill purpose
• Status indicators and success metrics
• Generated files or configurations
• Reports and summaries as applicable
• Recommendations for follow-up actions

Error Handling

If execution fails:

• Verify prerequisites are met
• Check input parameters and formats
• Validate file paths and permissions
• Review error messages for root cause
• Consult documentation for troubleshooting

Resources

• Official documentation for related tools
• Best practices guides
• Example use cases and templates
• Community forums and support channels