Usage & Cost

AI Gateway Metering Policy

AI Gateway Policy

This policy is for use with the AI Gateway. See the AI Gateway documentation to learn how to configure and govern AI models with Zuplo.

The AI Gateway Metering policy records each application's spend, tokens, and requests and enforces spend, token, and request budgets. When a limit is exceeded, it uses the model selection's quota fallback when one is configured or returns 429 Too Many Requests.

Place it after Model Filtering and Fallback Model, and before policies such as Semantic Cache that may answer without calling a provider. Metering fails open by default when its service is unavailable; set throwOnFailure to true to reject requests instead.

Configuration

The configuration shows how to configure the policy in the 'policies.json' document.


Code
{
  "name": "my-ai-gateway-metering-v2-inbound-policy",
  "policyType": "ai-gateway-metering-v2-inbound",
  "handler": {
    "export": "AIGatewayMeteringV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "budgetRules": [
        {
          "budgetBy": "app",
          "meters": [
            {
              "meter": "cost",
              "period": "monthly",
              "value": 100,
              "action": "block"
            },
            {
              "meter": "cost",
              "period": "monthly",
              "value": 80,
              "action": "warn"
            }
          ]
        }
      ],
      "throwOnFailure": false
    }
  }
}

Policy Configuration

name <string> - The name of your policy instance. This is used as a reference in your routes.
policyType <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be ai-gateway-metering-v2-inbound.
handler.export <string> - The name of the exported type. Value should be AIGatewayMeteringV2InboundPolicy.
handler.module <string> - The module containing the policy. Value should be $import(@zuplo/runtime).
handler.options <object> - The options for this policy. See Policy Options below.

Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

throwOnFailure <boolean> - Throw when the metering service is unavailable instead of failing open. Defaults to false.
budgetRules <object[]> - Budget rules for this application. Each rule budgets the whole app ("app") or each distinct value of an expression ("expression"), with warn and block thresholds per meter and period.
- budgetBy (required) <string> - No description available. Allowed values are app, expression.
- expression <string> - Required when budgetBy is "expression"; forbidden for "app". Each distinct value gets its own budget.
- meters (required) <object[]> - No description available.
  - meter (required) <string> - No description available. Allowed values are cost, requests, tokens.
  - period (required) <string> - No description available. Allowed values are hourly, daily, weekly, monthly.
  - value (required) <number> - No description available.
  - action (required) <string> - No description available. Allowed values are warn, block.
limits <object> - Usage limits grouped by meter.
- costs <object> - No description available.
  - daily <object> - No description available.
    - enabled (required) <boolean> - No description available.
    - limit <number> - No description available.
    - warning <object> - No description available.
      - enabled (required) <boolean> - No description available.
      - threshold <number> - Percentage of the limit at which to emit a warning event.
  - monthly <object> - No description available.
    - enabled (required) <boolean> - No description available.
    - limit <number> - No description available.
    - warning <object> - No description available.
      - enabled (required) <boolean> - No description available.
      - threshold <number> - Percentage of the limit at which to emit a warning event.
- tokens <object> - No description available.
  - daily <object> - No description available.
    - enabled (required) <boolean> - No description available.
    - limit <number> - No description available.
    - warning <object> - No description available.
      - enabled (required) <boolean> - No description available.
      - threshold <number> - Percentage of the limit at which to emit a warning event.
  - monthly <object> - No description available.
    - enabled (required) <boolean> - No description available.
    - limit <number> - No description available.
    - warning <object> - No description available.
      - enabled (required) <boolean> - No description available.
      - threshold <number> - Percentage of the limit at which to emit a warning event.
- requests <object> - No description available.
  - daily <object> - No description available.
    - enabled (required) <boolean> - No description available.
    - limit <number> - No description available.
    - warning <object> - No description available.
      - enabled (required) <boolean> - No description available.
      - threshold <number> - Percentage of the limit at which to emit a warning event.
  - monthly <object> - No description available.
    - enabled (required) <boolean> - No description available.
    - limit <number> - No description available.
    - warning <object> - No description available.
      - enabled (required) <boolean> - No description available.
      - threshold <number> - Percentage of the limit at which to emit a warning event.

Using the Policy

AI Gateway Metering

AI Gateway Metering records application usage and configures the app's own budgets before the provider request runs. It meters spend, tokens, and requests. Budget rules can cover the whole application or each distinct value of an expression.

When a limit is exceeded, the policy activates the model selection's quotaFallback when AI Gateway Fallback Model supplied one. Otherwise it returns 429 Too Many Requests.

Place Metering after Model Filtering and Fallback Model so an exceeded budget can activate the quota fallback. Put policies that may answer early, such as Semantic Cache, after Metering so cache hits still count toward request limits.

Example


Code
{
  "name": "ai-gateway-metering-v2-inbound",
  "options": {
    "throwOnFailure": false,
    "budgetRules": [
      {
        "budgetBy": "app",
        "meters": [
          {
            "meter": "cost",
            "period": "monthly",
            "value": 100,
            "action": "block"
          },
          {
            "meter": "cost",
            "period": "monthly",
            "value": 80,
            "action": "warn"
          }
        ]
      },
      {
        "budgetBy": "expression",
        "expression": "request.headers.get(\"x-user\")",
        "meters": [
          {
            "meter": "cost",
            "period": "daily",
            "value": 5,
            "action": "block"
          }
        ]
      }
    ]
  }
}

The supported meters are cost, requests, and tokens. The supported periods are hourly, daily, weekly, and monthly. An application can have at most five rules. An expression rule creates a separate budget for every distinct value of its expression. An action of "warn" notifies without blocking. An action of "block" activates the configured quota fallback or returns 429 Too Many Requests when usage reaches the value.

Budgets fail open by default. When throwOnFailure is false, a metering service failure lets the request proceed unmetered and no limit is checked. Set it to true to reject the request instead.

Team limits

Budgets configured on this application govern only this app. Limits configured on a parent team or the gateway root are enforced centrally after the application's policy chain, whether or not this policy appears in that chain. An inherited limit activates the selected model's quota fallback when available and otherwise returns 429 Too Many Requests.

An application cannot disable inherited enforcement through its policy chain. If the central hierarchical check is unavailable, the request proceeds. The policy's throwOnFailure option controls failures while checking or recording the app's own limits; it does not change inherited-limit behavior.