Creating a Custom Provider for AI SDK

Understand the Laravel AI SDK source code and implement custom providers for AI services not supported out of the box.

When you need a custom provider

Laravel AI SDK supports the major AI services out of the box — OpenAI, Anthropic, Gemini, and Mistral. However, the standard providers cannot handle cases like:

• An emerging AI service that is not yet officially supported
• An internal model gateway or billing management layer you want to route requests through
• An on-premises inference server that uses a proprietary protocol or authentication method

In these cases, implement a custom provider and register it with the SDK’s AiManager to use it through the same API as the standard providers.

Use the built-in driver for OpenAI-compatible APIs

As of SDK 0.9, the openai-compatible driver is included out of the box. For an internal OpenAI-compatible inference server, such as Ollama’s OpenAI-compatible endpoint, you only need to add its configuration to config/ai.php. You do not need to implement a custom provider.

'my-server' => [
    'driver' => 'openai-compatible',
    'key'    => env('MY_SERVER_API_KEY'),
    'url'    => env('MY_SERVER_URL', 'http://localhost:8080/v1'),
    'models' => [
        'text' => ['default' => 'llama3.3-70b'],
    ],
],

Architecture overview

Two-layer structure

The Laravel AI SDK consists of two layers: providers and gateways.

Layer	Role	Examples
Provider	Application-side interface. Resolves model names and holds configuration.	OpenAiProvider, AnthropicProvider
Gateway	Sends the actual API request for a single step.	AnthropicGateway, OpenAiCompatibleGateway

All providers extend the abstract class Laravel\Ai\Providers\Provider and implement feature-specific contracts (interfaces). For multi-step flows that include tool calls, TextGenerationLoop repeatedly invokes the gateway.

Available contracts

Implement only the contracts for the features you want to provide.

Contract	Namespace	Feature
TextProvider	Laravel\Ai\Contracts\Providers\TextProvider	Text generation and agents
EmbeddingProvider	Laravel\Ai\Contracts\Providers\EmbeddingProvider	Vector embedding generation
ImageProvider	Laravel\Ai\Contracts\Providers\ImageProvider	Image generation
AudioProvider	Laravel\Ai\Contracts\Providers\AudioProvider	Text-to-speech (TTS)
TranscriptionProvider	Laravel\Ai\Contracts\Providers\TranscriptionProvider	Speech-to-text (STT)

In most cases, implementing TextProvider alone is sufficient.

The TextProvider contract

The interface that text generation providers implement (src/Contracts/Providers/TextProvider.php).

interface TextProvider extends Provider
{
    public function prompt(AgentPrompt $prompt): AgentResponse;
    public function stream(AgentPrompt $prompt): StreamableAgentResponse;
    public function useTextGateway(StepTextGateway $gateway): self;
    public function textGenerationLoop(): TextGenerationLoop;
    public function defaultTextModel(): string;
    public function cheapestTextModel(): string;
    public function smartestTextModel(): string;
}

The prompt() and stream() implementations can be delegated to existing traits (GeneratesText, StreamsText), so you only need to implement four methods: the three model name methods and textGateway(). TextGenerationLoop manages the multi-step tool loop, so the gateway only processes a single request step.

Implementation example: custom provider

This example registers an inference service with a proprietary API as a provider named my-inference.

1. Create the provider class

   Create app/Ai/Providers/MyInferenceProvider.php.

   <?php
   
   declare(strict_types=1);
   
   namespace App\Ai\Providers;
   
   use Illuminate\Contracts\Events\Dispatcher;
   use Laravel\Ai\Contracts\Gateway\StepTextGateway;
   use Laravel\Ai\Contracts\Providers\TextProvider;
   use Laravel\Ai\Providers\Concerns\GeneratesText;
   use Laravel\Ai\Providers\Concerns\HasTextGateway;
   use Laravel\Ai\Providers\Concerns\StreamsText;
   use Laravel\Ai\Providers\Provider;
   
   class MyInferenceProvider extends Provider implements TextProvider
   {
       use GeneratesText;
       use HasTextGateway;
       use StreamsText;
   
       public function __construct(protected array $config, protected Dispatcher $events)
       {
           //
       }
   
       /**
        * Get the credentials for the provider.
        */
       public function providerCredentials(): array
       {
           return ['key' => $this->config['key'] ?? null];
       }
   
       /**
        * Get the provider's text gateway.
        */
       public function textGateway(): StepTextGateway
       {
           return $this->textGateway ??= new \App\Ai\Gateway\MyInferenceGateway($this->events);
       }
   
       public function defaultTextModel(): string
       {
           return $this->config['models']['text']['default'] ?? 'my-model-v1';
       }
   
       public function cheapestTextModel(): string
       {
           return $this->config['models']['text']['cheapest'] ?? 'my-model-v1';
       }
   
       public function smartestTextModel(): string
       {
           return $this->config['models']['text']['smartest'] ?? 'my-model-v1';
       }
   }
   
2. Register in AppServiceProvider

   Register the provider in the boot method of App\Providers\AppServiceProvider using extend().

   <?php
   
   namespace App\Providers;
   
   use App\Ai\Providers\MyInferenceProvider;
   use Illuminate\Contracts\Events\Dispatcher;
   use Illuminate\Support\ServiceProvider;
   use Laravel\Ai\AiManager;
   
   class AppServiceProvider extends ServiceProvider
   {
       public function boot(): void
       {
           $this->app->make(AiManager::class)->extend(
               'my-inference',
               fn (array $config) => new MyInferenceProvider(
                   $config,
                   $this->app->make(Dispatcher::class)
               )
           );
       }
   }
   
3. Add the provider to config/ai.php

   'providers' => [
       // ...existing providers...
   
       'my-inference' => [
           'driver' => 'my-inference',
           'key'    => env('MY_INFERENCE_API_KEY'),
           'url'    => env('MY_INFERENCE_URL', 'http://localhost:8080'),
       ],
   ],
   

   Add the values to your .env file as well.

   MY_INFERENCE_API_KEY=your-api-key
   MY_INFERENCE_URL=https://inference.example.internal
   
4. Use it from an agent

   After registration, specify the provider name in the provider argument of prompt() to use it just like a standard provider.

   use App\Ai\Agents\SummaryAgent;
   
   $response = SummaryAgent::make()->prompt('Summarize this article.', provider: 'my-inference');
   
   echo $response->text;
   

   To use it as the default provider, change the default key in config/ai.php.

   'default' => 'my-inference',
   

Implementing a custom gateway

Services with a proprietary non-OpenAI-compatible API require a custom gateway that implements the StepTextGateway contract.

The StepTextGateway contract

The interface is defined in src/Contracts/Gateway/StepTextGateway.php. A gateway processes one step of a conversation and returns a StepResponse. The calling TextGenerationLoop manages the tool call loop.

interface StepTextGateway
{
    public function generateTextStep(
        TextProvider $provider,
        string $model,
        ?string $instructions,
        array $messages,
        array $tools,
        ?array $schema,
        ?TextGenerationOptions $options,
        ?int $timeout,
        StepContext $stepContext,
    ): StepResponse;

    public function generateStreamStep(
        string $invocationId,
        TextProvider $provider,
        string $model,
        ?string $instructions,
        array $messages,
        array $tools,
        ?array $schema,
        ?TextGenerationOptions $options,
        ?int $timeout,
        StepContext $stepContext,
    ): Generator;
}

The pre-0.9 TextGateway contract and its generateText(), stream(), and onToolInvocation() methods were removed in 0.9. If you have a custom gateway, migrate it to StepTextGateway. Tool call handling from onToolInvocation() now lives in TextGenerationLoop.

Custom gateway implementation example

A skeleton of a simple gateway that sends HTTP requests to a proprietary inference API.

<?php

declare(strict_types=1);

namespace App\Ai\Gateway;

use Generator;
use Illuminate\Contracts\Events\Dispatcher;
use Illuminate\Support\Facades\Http;
use Laravel\Ai\Contracts\Gateway\StepTextGateway;
use Laravel\Ai\Contracts\Providers\TextProvider;
use Laravel\Ai\Gateway\StepContext;
use Laravel\Ai\Gateway\StepResponse;
use Laravel\Ai\Gateway\TextGenerationOptions;
use Laravel\Ai\Responses\Data\FinishReason;
use Laravel\Ai\Responses\Data\Meta;
use Laravel\Ai\Responses\Data\Usage;

class MyInferenceGateway implements StepTextGateway
{
    public function __construct(protected Dispatcher $events) {}

    public function generateTextStep(
        TextProvider $provider,
        string $model,
        ?string $instructions,
        array $messages,
        array $tools,
        ?array $schema,
        ?TextGenerationOptions $options,
        ?int $timeout,
        StepContext $stepContext,
    ): StepResponse {
        $credentials = $provider->providerCredentials();
        $config      = $provider->additionalConfiguration();

        $response = Http::withToken($credentials['key'])
            ->baseUrl($config['url'])
            ->timeout($timeout ?? 30)
            ->post('/generate', [
                'model'        => $model,
                'instructions' => $instructions,
                'messages'     => $this->formatMessages($messages),
            ]);

        $data = $response->json();

        return new StepResponse(
            text: $data['output']['text'] ?? '',
            toolCalls: [],
            finishReason: FinishReason::Stop,
            usage: new Usage(
                promptTokens: $data['usage']['input_tokens'] ?? 0,
                completionTokens: $data['usage']['output_tokens'] ?? 0,
            ),
            meta: new Meta(
                provider: $provider->name(),
                model: $model,
            ),
        );
    }

    public function generateStreamStep(
        string $invocationId,
        TextProvider $provider,
        string $model,
        ?string $instructions,
        array $messages,
        array $tools,
        ?array $schema,
        ?TextGenerationOptions $options,
        ?int $timeout,
        StepContext $stepContext,
    ): Generator {
        // Streaming implementation (omitted)
        yield from [];

        return null;
    }

    protected function formatMessages(array $messages): array
    {
        return array_map(fn ($message) => [
            'role'    => $message->role->value,
            'content' => $message->content,
        ], $messages);
    }
}

To support tool calls (function calling), include the tool call results in toolCalls from generateTextStep() and set finishReason to FinishReason::ToolCalls. TextGenerationLoop automatically executes the tools and advances to the next step. Refer to AnthropicGateway.php for an implementation example.

Testing

Using Agent::fake()

To test agents that use your custom provider, call the fake method on the agent class. Regardless of whether a custom provider is in use, the fake gateway is set on the provider.

<?php

namespace Tests\Feature;

use App\Ai\Agents\SummaryAgent;
use Illuminate\Foundation\Testing\RefreshDatabase;
use Tests\TestCase;

class SummaryAgentTest extends TestCase
{
    use RefreshDatabase;

    public function test_summary_agent_returns_text(): void
    {
        SummaryAgent::fake(['This is a summary.']);

        $response = SummaryAgent::make()->prompt('Long article text...', provider: 'my-inference');

        $this->assertEquals('This is a summary.', $response->text);

        SummaryAgent::assertPrompted('Long article text...');
    }
}

As of 0.9, responses from Agent::fake() pass through the same TextGenerationLoop as real providers. If you configure a fake tool call for an agent that has not registered that tool, the SDK throws a NoSuchToolException.

Mock provider using extend()

You can also use extend() to register a test provider from the container.

public function test_with_mock_provider(): void
{
    $this->app->make(AiManager::class)->extend(
        'my-inference',
        function (array $config) {
            $events = $this->app->make(\Illuminate\Contracts\Events\Dispatcher::class);

            return new MyInferenceProvider($config, $events);
        }
    );

    // Test code
}

Reference links

OllamaProvider.php — simple provider implementation example

A minimal provider that connects to a local model server. A good starting point for custom provider implementations.

AnthropicGateway.php — gateway implementation example

A gateway implementation of StepTextGateway, including generateTextStep() and generateStreamStep().

StepTextGateway Contract

The interface definition that text generation gateways must implement.

TextProvider Contract

The interface definition that text generation providers must implement.