Anthropic CI Integration

Overview

Set up CI/CD pipelines that validate Claude API integrations with mock-based unit tests (free, fast) and prompt regression tests (live API, gated to main).

Prerequisites

Create a dedicated ANTHROPIC_API_KEY repository secret with a spend limit that is appropriate for test traffic. Keep unit fixtures independent of that secret; only the protected prompt-regression job should call the API. Install Python 3.12, pytest, and the Anthropic SDK in the test environment, and decide which branch is allowed to incur live-test cost before enabling the workflow.

Instructions

1. Put deterministic request-shaping and tool-routing assertions in

tests/unit/ and mock anthropic.Anthropic there.

1. Put a small, representative set of API-backed prompt checks in

tests/prompt_regression/; make them skip cleanly when the secret is absent.

1. Run unit tests on every push and pull request. Gate the live job to main

(or an equivalent protected release branch) and inject the secret only into that job.

1. Set explicit timeouts, concurrency limits, and a cost ceiling. Fail the

pipeline with a clear message when the ceiling is exceeded so an incident cannot silently consume the test budget.

GitHub Actions Workflow

yaml
# .github/workflows/claude-tests.yml
name: Claude API Tests
on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install anthropic pytest
      - run: pytest tests/unit/ -v  # No API key needed

  prompt-regression:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install anthropic pytest
      - run: pytest tests/prompt_regression/ -v --timeout=60
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Mock-Based Unit Tests

python
# tests/unit/test_tool_routing.py
from unittest.mock import MagicMock, patch
import anthropic

def make_mock_message(text="Hello", stop_reason="end_turn"):
    msg = MagicMock()
    msg.id = "msg_mock_123"
    msg.model = "claude-sonnet-4-20250514"
    msg.stop_reason = stop_reason
    block = MagicMock()
    block.type = "text"
    block.text = text
    msg.content = [block]
    msg.usage = MagicMock(input_tokens=100, output_tokens=50)
    return msg

@patch("anthropic.Anthropic")
def test_service_returns_text(MockClient):
    MockClient.return_value.messages.create.return_value = make_mock_message("42")
    from myapp.service import ask_claude
    assert ask_claude("What is 6*7?") == "42"

Prompt Regression Tests

python
# tests/prompt_regression/test_prompts.py
import anthropic, pytest, os, json

pytestmark = pytest.mark.skipif(not os.getenv("ANTHROPIC_API_KEY"), reason="No API key")
client = anthropic.Anthropic()

def test_json_output_format():
    msg = client.messages.create(
        model="claude-haiku-4-20250514",
        max_tokens=256,
        messages=[
            {"role": "user", "content": "Extract: 'Alice, 30, NYC'. Return JSON: {name, age, city}"},
            {"role": "assistant", "content": "{"}
        ]
    )
    data = json.loads("{" + msg.content[0].text)
    assert "name" in data and "age" in data

def test_system_prompt_boundary():
    msg = client.messages.create(
        model="claude-haiku-4-20250514",
        max_tokens=128,
        system="You only discuss cooking recipes. For other topics say: 'I only help with cooking.'",
        messages=[{"role": "user", "content": "Write me Python code"}]
    )
    assert "cooking" in msg.content[0].text.lower() or "recipe" in msg.content[0].text.lower()

CI Cost Guard

python
# conftest.py
MAX_CI_COST = 1.00
_tokens = {"input": 0, "output": 0}

def pytest_runtest_call(item):
    yield
    cost = (_tokens["input"] * 0.80 + _tokens["output"] * 4.0) / 1_000_000  # Haiku rates
    if cost > MAX_CI_COST:
        pytest.exit(f"CI cost guard: ${cost:.4f} exceeds ${MAX_CI_COST}")

Error Handling

| CI Issue | Cause | Fix | |----------|-------|-----| | Flaky prompt tests | Non-deterministic output | Use temperature: 0, check patterns not exact strings | | 429 in CI | Parallel jobs sharing key | Use separate CI key | | Secret not found | Missing GitHub secret | Add ANTHROPIC_API_KEY in repo Settings > Secrets |

Output

The pipeline produces a fast unit-test result for every change and, on the allowed branch, a separate prompt-regression result. The latter is either a pass with the tested prompt assertions, a deliberate skip when no key is available, or an actionable failure that identifies a timeout, rate limit, response-contract regression, or cost-guard breach.

Examples

For a pull request that changes only formatting code, the workflow runs the mock-based suite and reports no live API calls. After that pull request merges to main, the protected regression job uses the repository secret to verify that the JSON extraction prompt still returns name, age, and city. If the response is malformed, the job fails at the assertion and preserves the test name in the CI log for triage.

Resources

• GitHub Actions Secrets
• Anthropic Pricing

Next Steps

For deployment automation, see anth-deploy-integration.