When working with large language models like Claude from Anthropic, understanding how many tokens your input consumes is essential for managing costs and optimizing performance. In this blog, we’ll explore how to calculate token counts for a given input using Go. We’ll use Anthropic’s count_tokens API endpoint, which provides a straightforward way to determine token usage.

Prerequisites

Before diving in, ensure you have:

1. A working Go environment (installation guide).
2. An Anthropic API key. Set it as an environment variable (ANTHROPIC_API_KEY).
3. Basic knowledge of Go HTTP requests and JSON handling.

The Code

Here’s a complete Go function to calculate the number of tokens for Claude:

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
)

// RequestPayload defines the structure of the JSON payload
type RequestPayload struct {
	Model    string `json:"model"`
	System   string `json:"system"`
	Messages []struct {
		Role    string `json:"role"`
		Content string `json:"content"`
	} `json:"messages"`
}

// Response defines the structure of the response from the API
type Response struct {
	InputTokens int `json:"input_tokens"`
}

// CalculateToken calculates the token count for a given input content
func CalculateToken(ctx context.Context, content string, encoding string) (int, error) {
	// Define the API endpoint and headers
	url := "https://api.anthropic.com/v1/messages/count_tokens"
	apiKey := os.Getenv("ANTHROPIC_API_KEY") // Ensure this is set in your environment
	headers := map[string]string{
		"x-api-key":         apiKey,
		"content-type":      "application/json",
		"anthropic-version": "2023-06-01",
		"anthropic-beta":    "token-counting-2024-11-01",
	}

	// Create the request payload
	payload := RequestPayload{
		Model:  encoding,
		System: "You are a scientist",
		Messages: []struct {
			Role    string `json:"role"`
			Content string `json:"content"`
		}{
			{Role: "user", Content: content},
		},
	}

	// Serialize the payload to JSON
	payloadBytes, err := json.Marshal(payload)
	if err != nil {
		log.Fatalf("Error marshalling payload: %v", err)
	}

	// Create an HTTP POST request
	req, err := http.NewRequest("POST", url, bytes.NewBuffer(payloadBytes))
	if err != nil {
		log.Fatalf("Error creating HTTP request: %v", err)
	}

	// Add headers to the request
	for key, value := range headers {
		req.Header.Set(key, value)
	}

	// Send the request
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("Error sending request: %v", err)
	}
	defer resp.Body.Close()

	// Read and parse the response
	body, err := io.ReadAll(resp.Body)
	if err != nil {
		log.Printf("Error reading response body: %v", err)
		return 0, err
	}

	respData := &Response{}
	_ = json.Unmarshal(body, respData)
	fmt.Printf("Response Status: %s\n", resp.Status)
	fmt.Printf("Response Body: %s\n", body)

	return respData.InputTokens, nil
}

Explanation

1. Structuring the Payload

The RequestPayload struct defines the JSON payload required by the count_tokens API. The Messages field contains the conversation history, where each message has a role (e.g., user) and content.

2. Defining the API Endpoint and Headers

We use the url for the token-counting endpoint and provide necessary headers, including:

• x-api-key: Your API key.
• content-type: Set to application/json.
• anthropic-version and anthropic-beta: Specify the API version and beta feature for token counting.

3. Making the API Request

The function:

1. Serializes the payload to JSON.
2. Creates a POST request with the serialized payload.
3. Sets the required headers.
4. Handling the Response

The response is read and unmarshaled into the Response struct, extracting the input_tokens field.

Example Usage

Here’s how you can use the CalculateToken function:

func main() {
	ctx := context.Background()
	content := "Hello! How many tokens does this text consume?"
	encoding := "claude-2" // Replace with the desired model name

	tokens, err := CalculateToken(ctx, content, encoding)
	if err != nil {
		log.Fatalf("Failed to calculate tokens: %v", err)
	}

	fmt.Printf("Input content consumed %d tokens.\n", tokens)
}

Output

Running the example will output something like:

Response Status: 200 OK
Response Body: {"input_tokens": 12}
Input content consumed 12 tokens.

By following this guide, you can seamlessly integrate token counting into your Go applications and better manage your interaction with Claude.