azure-aigateway

Azure AI Gateway

Bootstrap and configure Azure API Management (APIM) as an AI Gateway for securing, observing, and controlling AI models, tools (MCP Servers), and agents.

Skill Activation Triggers

Use this skill immediately when the user asks to:

• "Set up a gateway for my model"
• "Set up a gateway for my tools"
• "Set up a gateway for my agents"
• "Add a gateway to my MCP server"
• "Protect my AI model with a gateway"
• "Secure my AI agents"
• "Ratelimit my model requests"
• "Ratelimit my tool requests"
• "Limit tokens for my model"
• "Add rate limiting to my MCP server"
• "Enable semantic caching for my AI API"
• "Add content safety to my AI endpoint"
• "Add my model behind gateway"
• "Import API from OpenAPI spec"
• "Add API to gateway from swagger"
• "Convert my API to MCP"
• "Expose my API as MCP server"

Key Indicators:

• User deploying Azure OpenAI, AI Foundry, or other AI models
• User creating or managing MCP servers
• User needs token limits, rate limiting, or quota management
• User wants to cache AI responses to reduce costs
• User needs content filtering or safety controls
• User wants load balancing across multiple AI backends

Secondary Triggers (Proactive Recommendations):

• After model creation: Recommend AI Gateway for security, caching, and token limits
• After MCP server creation: Recommend AI Gateway for rate limiting, content safety, and auth

Overview

Azure API Management serves as an AI Gateway that provides:

• Security: Authentication, authorization, and content safety
• Observability: Token metrics, logging, and monitoring
• Control: Rate limiting, token limits, and load balancing
• Optimization: Semantic caching to reduce costs and latency

AI Models ──┐                      ┌── Azure OpenAI
MCP Tools ──┼── AI Gateway (APIM) ──┼── AI Foundry
Agents ─────┘                      └── Custom Models

Key Resources

• GitHub Repo: https://github.com/Azure-Samples/AI-Gateway (aka.ms/aigateway)
• Docs:
• GenAI Gateway Capabilities
• MCP Server Overview
• Azure AI Foundry API
• Semantic Caching
• Token Limits & LLM Logs

Configuration Rules

Default to Basicv2 SKU when creating new APIM instances:

• Cheaper than other tiers
• Creates quickly (~5-10 minutes vs 30+ for Premium)
• Supports all AI Gateway policies

Pattern 1: Quick Bootstrap AI Gateway

Deploy APIM with Basicv2 SKU for AI workloads.

# Create resource group
az group create --name rg-aigateway --location eastus2

# Deploy APIM with Bicep
az deployment group create \
  --resource-group rg-aigateway \
  --template-file main.bicep \
  --parameters apimSku=Basicv2

Bicep Template

param location string = resourceGroup().location
param apimSku string = 'Basicv2'
param apimManagedIdentityType string = 'SystemAssigned'

// NOTE: Using 2024-06-01-preview because Basicv2 SKU support currently requires this preview API version.
//       Update to the latest stable (GA) API version once Basicv2 is available there.
resource apimService 'Microsoft.ApiManagement/service@2024-06-01-preview' = {
  name: 'apim-aigateway-${uniqueString(resourceGroup().id)}'
  location: location
  sku: {
    name: apimSku
    capacity: 1
  }
  properties: {
    publisherEmail: 'admin@contoso.com'
    publisherName: 'Contoso'
  }
  identity: {
    type: apimManagedIdentityType
  }
}

output gatewayUrl string = apimService.properties.gatewayUrl
output principalId string = apimService.identity.principalId

Pattern 2: Semantic Caching

Cache similar prompts to reduce costs and latency.

<policies>
    <inbound>
        <base />
        <!-- Cache lookup with 0.8 similarity threshold -->
        <azure-openai-semantic-cache-lookup 
            score-threshold="0.8" 
            embeddings-backend-id="embeddings-backend" 
            embeddings-backend-auth="system-assigned" />
        <set-backend-service backend-id="{backend-id}" />
    </inbound>
    <outbound>
        <!-- Cache responses for 120 seconds -->
        <azure-openai-semantic-cache-store duration="120" />
        <base />
    </outbound>
</policies>

Options: | Parameter | Range | Description | |-----------|-------|-------------| | score-threshold | 0.7-0.95 | Higher = stricter matching | | duration | 60-3600 | Cache TTL in seconds |

Pattern 3: Token Rate Limiting

Limit tokens per minute to control costs and prevent abuse.

<policies>
    <inbound>
        <base />
        <set-backend-service backend-id="{backend-id}" />
        <!-- Limit to 500 tokens per minute per subscription -->
        <azure-openai-token-limit 
            counter-key="@(context.Subscription.Id)"
            tokens-per-minute="500" 
            estimate-prompt-tokens="false" 
            remaining-tokens-variable-name="remainingTokens" />
    </inbound>
</policies>

Options: | Parameter | Values | Description | |-----------|--------|-------------| | counter-key | Subscription.Id, Request.IpAddress, custom | Grouping key for limits | | tokens-per-minute | 100-100000 | Token quota | | estimate-prompt-tokens | true/false | true = faster but less accurate |

Pattern 4: Content Safety

Filter harmful content and detect jailbreak attempts.

<policies>
    <inbound>
        <base />
        <set-backend-service backend-id="{backend-id}" />
        <!-- Block severity 4+ content, detect jailbreaks -->
        <llm-content-safety backend-id="content-safety-backend" shield-prompt="true">
            <categories output-type="EightSeverityLevels">
                <category name="Hate" threshold="4" />
                <category name="Sexual" threshold="4" />
                <category name="SelfHarm" threshold="4" />
                <category name="Violence" threshold="4" />
            </categories>
            <blocklists>
                <id>custom-blocklist</id>
            </blocklists>
        </llm-content-safety>
    </inbound>
</policies>

Options: | Parameter | Range | Description | |-----------|-------|-------------| | threshold | 0-7 | 0=safe, 7=severe | | shield-prompt | true/false | Detect jailbreak attempts |

Pattern 5: Rate Limits for MCPs/OpenAPI Tools

Protect MCP servers and tools with request rate limiting.

<policies>
    <inbound>
        <base />
        <!-- 10 calls per 60 seconds per IP -->
        <rate-limit-by-key 
            calls="10" 
            renewal-period="60" 
            counter-key="@(context.Request.IpAddress)" 
            remaining-calls-variable-name="remainingCalls" />
    </inbound>
    <outbound>
        <set-header name="X-Rate-Limit-Remaining" exists-action="override">
            <value>@(context.Variables.GetValueOrDefault<int>("remainingCalls", 0).ToString())</value>
        </set-header>
        <base />
    </outbound>
</policies>

Pattern 6: Managed Identity Authentication

Secure backend access with managed identity instead of API keys.

<policies>
    <inbound>
        <base />
        <!-- Managed identity auth to Azure OpenAI -->
        <authentication-managed-identity 
            resource="https://cognitiveservices.azure.com" 
            output-token-variable-name="managed-id-access-token" 
            ignore-error="false" />
        <set-header name="Authorization" exists-action="override">
            <value>@("Bearer " + (string)context.Variables["managed-id-access-token"])</value>
        </set-header>
        <set-backend-service backend-id="{backend-id}" />
        <!-- Emit token metrics for m