LLMs Hosted on Azure & AWS Bedrock

Microsoft Azure and AWS Bedrock offer the ability to host private clouds with OpenAI models. You can add these models in Stack AI using an "Azure" node or "Bedrock" node. Hosting models in Azure/Bedrock has benefits:

Azure

1. Lower and Consistent Latency: Cloud hosted models are not affected by public API traffic. This is a great option if latency is a real concern.

2. Higher Rate Limits: Models in Azure offer higher rate limits of up to 240,000 tokens per minute and 1440 requests per minute.

3. Data Privacy and Compliance: data sent to Azure is kept under the private cloud and is not sent to OpenAI or any external service. These models are covered under Azure's Business Associate Agreement (BAA) and are HIPPA compliant.

AWS Bedrock

1. AWS Security & Compliance: Bedrock leverages AWS’s security, IAM, and compliance features. You can use AWS IAM roles, VPC endpoints, and audit logging for enterprise-grade security.

2. Data Privacy & Residency: Data processed through Bedrock stays within AWS infrastructure. You can choose the AWS region for data residency requirements.

How It Works

Microsoft Azure

• Azure OpenAI Service: Azure provides access to OpenAI models (like GPT-3.5, GPT-4, etc.) through its Azure OpenAI Service.

• How it works: You provision an Azure OpenAI resource, get an endpoint and API key, and can then use these credentials to access models via Azure’s API.

AWS Bedrock

• Amazon Bedrock: AWS offers access to multiple foundation models (including Anthropic Claude, AI21, Cohere, and Amazon’s own Titan models) through the Amazon Bedrock service.

• How it works: You provision an AWS Bedrock resource, get an endpoint and API key, and can then use these credentials to access the models you have enabled via AWS's API.