How to set up LLM analytics for Anthropic's Claude

Apr 09, 2024

Posted by

Lior Neu-ner

On this page

1. Create the demo app
2. Capture chat completion events
3. Create insights
Total cost per model
Average cost per user
Average API response time
Next steps
Further reading

Tracking your Claude usage, costs, and latency is crucial to understanding how your users are interacting with your AI and LLM powered features. In this tutorial, we show you how to monitor important metrics such as:

Total cost per model
Average cost per user
Average API response time

We'll build a basic Next.js app, implement the Claude API, and capture these events using PostHog.

1. Create the demo app

To showcase how to track important metrics, we create a simple one-page Next.js app with the following:

A form with textfield and button for user input.
A label to show Claude's output.
A dropdown to select different Anthropic models.

First, ensure Node.js is installed (version 18.0 or newer). Then run the following script to create a new Next.js app and install both the Anthropic JavaScript and PostHog Web SDKs:

Terminal

npx create-next-app@latest anthropic-analytics
cd anthropic-analytics
npm install --save @anthropic-ai/sdk
npm install --save posthog-js
cd ./src/app
touch providers.js # we set up PostHog in this file below

When prompted, select No for TypeScript, Yes for use app router, No for Tailwind CSS and the defaults for every other option.

Next, we set up Posthog using our API key and host (You can find these in your project settings). Add the below code to app/providers.js:

app/providers.js

'use client'
import posthog from 'posthog-js'
import { PostHogProvider } from 'posthog-js/react'
import { useEffect } from 'react'

export function PHProvider({ children }) {
  useEffect(() => {
    posthog.init('<ph_project_api_key>', {
      api_host: 'https://us.i.posthog.com'
    })
  }, []);
  return <PostHogProvider client={posthog}>{children}</PostHogProvider>
}

Then we import the PHProvider component into app/layout.js and wrap our app with it:

app/layout.js

import "./globals.css";
import { PHProvider } from './providers'



export default function RootLayout({ children }) {
  return (
    <html lang="en">
      <PHProvider>
        <body>{children}</body>
      </PHProvider>
    </html>
  );
}

Then replace the code in page.js with our basic layout and functionality. You can find your Anthropic API key here.

app/page.js

'use client'

import { useState } from 'react';
import { usePostHog } from 'posthog-js/react'
import Anthropic from '@anthropic-ai/sdk';

const models = [
  {
    name: 'claude-3-opus-20240229',
    token_input_cost: 0.00001500,
    token_output_cost: 0.00007500
  },
  {
    name: 'claude-3-sonnet-20240229',
    token_input_cost: 0.00000300,
    token_output_cost:  0.00001500
  },
    {
    name: 'claude-3-haiku-20240307',
    token_input_cost: 0.00000025,
    token_output_cost: 0.00000125
  },
]

export default function Home() {
  const [userInput, setUserInput] = useState('');
  const [response, setResponse] = useState('');
  const [selectedModel, setSelectedModel] = useState(models[0]);
  const posthog = usePostHog()

  const fetchResponse = async () => {
    try {
      const anthropic = new Anthropic({
        apiKey: '<your_anthropic_api_key>', 
        baseURL: window.location.origin + '/anthropic/',
      });  
      
      setResponse('Generating...');
      const msg = await anthropic.messages.create({
        model: selectedModel.name,
        max_tokens: 4096,
        messages: [{ role: "user", content: userInput }],
      });

      const response = msg.content[0].text
      setResponse(response);
    } catch (error) {
      setResponse(error.message);
    }
  };

  const handleInputChange = (event) => {
    setUserInput(event.target.value);
  };

  const handleModelChange = (event) => {
    setSelectedModel(models.filter(m => (m.name === event.target.value))[0]);
  };

  const handleSubmit = (event) => {
    event.preventDefault();
    fetchResponse();
  };

  return (
    <div style={{ display: 'flex', flexDirection: 'column', alignItems: 'center', justifyContent: 'center', minHeight: '100vh', gap: '20px' }}>
      <form onSubmit={handleSubmit}>
        <input
          type="text"
          value={userInput}
          onChange={handleInputChange}
          placeholder="Type your message"
        />
        <button type="submit">Send</button>
      </form>
      <select value={selectedModel.name} onChange={handleModelChange}>
        {models.map((model, index) => (
          <option key={index} value={model.name}>
            {model.name}
          </option>
        ))}
      </select>     
      <label>API Response:</label>
      <label>{response}</label>
    </div>
  );
};

Lastly, replace the code in next.config.mjs with the following:

next.config.mjs

const nextConfig = {
  rewrites: async () => [
      {
        source: "/anthropic/:path*",
        destination: "https://api.anthropic.com/:path*"
      },
    ],
  
};
export default nextConfig;

Our basic app is now set up. Run npm run dev to see it in app action.

2. Capture chat completion events

With our app set up, we can begin capturing events with PostHog. To start, we capture a claude_message_completion event with properties related to the API request. We find the following properties most useful to capture:

prompt
model
input_tokens
output_tokens
input_cost_in_dollars i.e. input_tokens * token_input_cost
output_cost_in_dollars i.e. output_tokens * token_output_cost
total_cost_in_dollars i.e. input_cost_in_dollars + output_cost_in_dollars

Update your fetchResponse() function in page.js to capture this event:

App.js

const fetchResponse = async () => {
    try {

      // your existing code...

      setResponse('Generating...');
      const msg = await anthropic.messages.create({
        model: selectedModel.name,
        max_tokens: 4096,
        messages: [{ role: "user", content: userInput }],
      });

      const inputCostInDollars = msg.usage.input_tokens * selectedModel.token_input_cost
      const outputCostInDollars = msg.usage.output_tokens * selectedModel.token_output_cost
      posthog.capture('claude_message_completion', {
        model: selectedModel.name,
        prompt: userInput,
        input_tokens: msg.usage.input_tokens,
        output_tokens: msg.usage.output_tokens,
        input_cost_in_dollars: inputCostInDollars,
        output_cost_in_dollars: outputCostInDollars,
        total_cost_in_dollars: inputCostInDollars + outputCostInDollars
      })

      // your existing code...

Refresh your app and submit a few prompts. You should then see your events captured in the PostHog activity tab.

3. Create insights

Now that we're capturing events, we can create insights. Below are three examples of useful metrics you should monitor:

Total cost per model

To create this insight, go the Product analytics tab and click + New insight. Then:

Set the event to claude_message_completion
Click on Total count to show a dropdown. Click on Property value (sum).
Select the total_cost_in_dollars property.
Click + Add breakdown and select model from the event properties list.

Note: Insights may show 0 if the total cost is less than 0.01.

Average cost per user

This metric helps give you an idea of how your costs will scale as your product grows. Creating this insight is similar to creating the one above, however we use formula mode to divide the total cost by the total number of users:

Set the event to claude_message_completion
Click on Total count to show a dropdown. Click on Property value (sum).
Select the total_cost_in_dollars property.
Click + Add graph series (if your visual is set to number, switch it back to trend first).
Change the event name to claude_message_completion. Then change the value from Total count to Unique users.
Click Enable formula mode.
In the formula box, enter A/B.

Total Anthropic costs per user in PostHog

Average API response time

Anthropics's API response time can take long, especially for longer outputs, so it's useful to keep an eye on this. To do this, first we need to modify our event capturing to also include the response time:

page.js

const fetchResponse = async () => {
    try {

      // your existing code...

      const startTime = performance.now(); 
      const msg = await anthropic.messages.create({
        model: selectedModel.name,
        max_tokens: 4096,
        messages: [{ role: "user", content: userInput }],
      });
      const endTime = performance.now();
      const responseTime = endTime - startTime;

      const inputCostInDollars = msg.usage.input_tokens * selectedModel.token_input_cost
      const outputCostInDollars = msg.usage.output_tokens * selectedModel.token_output_cost
      posthog.capture('claude_message_completion', {
        model: selectedModel.name,
        prompt: userInput,
        input_tokens: msg.usage.input_tokens,
        output_tokens: msg.usage.output_tokens,
        input_cost_in_dollars: inputCostInDollars,
        output_cost_in_dollars: outputCostInDollars,
        total_cost_in_dollars: inputCostInDollars + outputCostInDollars,
        response_time_in_ms: responseTime
      });

      // your existing code...

Then, after capturing a few events, create a new insight to calculate the average response time:

Set the event to claude_message_completion
Click on Total count to show a dropdown. Click on Property value (average).
Select the response_time_in_ms property.

Next steps

We've shown you the basics of creating insights from your product's Claude usage. Below are more examples of product questions you may want to investigate:

How many of my users are interacting with my LLM features?
Are there generation latency spikes?
Does interacting with LLM features correlate with other metrics e.g. retention, usage, or revenue?

How to set up LLM analytics for Anthropic's Claude

1. Create the demo app

2. Capture chat completion events

3. Create insights

Total cost per model

Average cost per user

Average API response time

Next steps

Further reading

Product for Engineers

Questions? Ask Max AI.

Comments