ok i want you to create a cot with tools can be used during thinking with a native cot :
import json
import os
from duckduckgo_search import DDGS
from openai import OpenAI, APIConnectionError, APIError

A tool function to perform web searches

def web_search(query):
"""Perform a DuckDuckGo web search for the given query."""
try:
with DDGS() as ddgs:
results = [r for r in ddgs.text(query, max_results=5)]
if results:
return "\n".join([f"{i+1}. {r['title']}" for i, r in enumerate(results)])
else:
return "No results found."
except Exception as e:
return f"Search error: {e}"

Tools configuration that may be used by the CoT process

tools = [
{
"type": "function",
"function": {
"name": "web_search",
"description": "Searches for information on the web.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search terms"}
},
"required": ["query"]
}
}
}
]

Configuration for the API connection (replace with your own API key and URL)

client_config = {
"base_url": "https://api.groq.com/openai/v1",
"api_key": "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",
"model": "deepseek-r1-distill-llama-70b"
}

Create an API client instance

client = OpenAI(
base_url=client_config["base_url"],
api_key=client_config["api_key"]
)

def test_api_connection():
"""Minimal test to check API connectivity."""
try:
response = client.chat.completions.create(
model=client_config["model"],
messages=[{"role": "user", "content": "Test connection"}]
)
print("Connection successful! Test response:")
print(response.choices[0].message.content)
return True
except (APIConnectionError, APIError) as e:
print(f"Connection error: {e}")
except Exception as e:
print(f"Unexpected error: {e}")
return False

def main_loop():
"""
Main loop for a CoT process that:
1. Accepts a user question.
2. Uses internal chain-of-thought reasoning to decide if a tool (web_search) should be used.
3. If needed, calls the tool and appends the result to the conversation.
4. Provides a final answer based on hidden internal reasoning.

text
Note: The internal reasoning is not exposed.
"""
messages = [
    {"role": "system", "content": (
        "You are an intelligent assistant that reasons step by step before giving a final answer. "
        "Use tools if necessary, but do not reveal your internal chain of thought."
    )}
]

print("Enter your question (or type 'exit' to quit):")
while True:
    question = input("\n==> ").strip()
    if question.lower() in ["exit", "quit"]:
        break

    messages.append({"role": "user", "content": question})

    try:
        # Call the API with tools enabled; the assistant internally decides on tool usage.
        response = client.chat.completions.create(
            model=client_config["model"],
            messages=messages,
            tools=tools,
            tool_choice="auto",
            reasoning_format = "parsed"  # The model automatically selects a tool if needed.
        )
        response_message = response.choices[0].message
        # Display the hidden chain-of-thought internally (only for developer logging)
        print("\n[REASONING (hidden)]")
        print(response_message.content or "No explicit reasoning provided.")
        # Process any tool calls made by the model.
        if response_message.tool_calls:
            for tool_call in response_message.tool_calls:
                if tool_call.function.name == "web_search":
                    args = json.loads(tool_call.function.arguments)
                    print(f"\n[TOOL CALL] Searching for: {args['query']}")
                    tool_result = web_search(args["query"])
                    print(f"[TOOL RESULT]\n{tool_result[:500]}...")
                    messages.append({
                        "role": "tool",
                        "content": tool_result,
                        "tool_call_id": tool_call.id
                    })
        # Display final answer
        print("\n[FINAL ANSWER]")
        print(response_message.content)
    except (APIConnectionError, APIError) as e:
        print(f"API error: {e}")
    except Exception as e:
        print(f"Error: {e}")

    messages.append({"role": "assistant", "content": response_message.content})

if name == "main":
print("Testing API connection...")
if test_api_connection():
main_loop()

Playground
Documentation
API Reference
Metrics
API Keys
Settings
Status
discord logoDiscord
Chat with us

XELOSZ
Personal
Documentation
Get Started
Overview
Quickstart
OpenAI Compatibility
Models
Rate Limits
Features
Text
Speech
Reasoning
Vision
Advanced Features
Batch Processing
Flex Processing
Content Moderation
Prefilling
Tool Use
Developer Resources
Groq Libraries
Groq Badge
Examples
Applications Showcase
Resources
Prompting Guide
Integrations
Agno
AutoGen
Arize
Composio
CrewAI
E2B
Gradio
JigsawStack
LangChain
LlamaIndex
LiteLLM
LiveKit
Toolhouse
Vercel
xRx
Support & Guidelines
Errors
Changelog
Policies & Notices
Reasoning
Reasoning models excel at complex problem-solving tasks that require step-by-step analysis, logical deduction, and structured thinking and solution validation. With Groq inference speed, these types of models can deliver instant reasoning capabilities critical for real-time applications.

Why Speed Matters for Reasoning
Reasoning models are capable of complex decision making with explicit reasoning chains that are part of the token output and used for decision-making, which make low-latency and fast inference essential. Complex problems often require multiple chains of reasoning tokens where each step build on previous results. Low latency compounds benefits across reasoning chains and shaves off minutes of reasoning to a response in seconds.

Supported Model
Model ID Model
deepseek-r1-distill-qwen-32b DeepSeek R1 Distill Qwen 32B
deepseek-r1-distill-llama-70b DeepSeek R1 Distil Llama 70B
Reasoning Format
Groq API supports explicit reasoning formats through the reasoning_format parameter, giving you fine-grained control over how the model's reasoning process is presented. This is particularly valuable for valid JSON outputs, debugging, and understanding the model's decision-making process.

Note: The format defaults to raw or parsed when JSON mode or tool use are enabled as those modes do not support raw. If reasoning is explicitly set to raw with JSON mode or tool use enabled, we will return a 400 error.

Options for Reasoning Format
reasoning_format Options Description
parsed Separates reasoning into a dedicated field while keeping the response concise.
raw Includes reasoning within think tags in the content.
hidden Returns only the final answer for maximum efficiency.
Quick Start
Python
JavaScript
curl

from groq import Groq

client = Groq()
completion = client.chat.completions.create(
model="deepseek-r1-distill-llama-70b",
messages=[
{
"role": "user",
"content": "How many r's are in the word strawberry?"
}
],
temperature=0.6,
max_completion_tokens=1024,
top_p=0.95,
stream=True,
reasoning_format="raw"
)

for chunk in completion:
print(chunk.choices[0].delta.content or "", end="")
Quick Start with Tool use

curl https://api.groq.com//openai/v1/chat/completions -s
-H "authorization: bearer $GROQ_API_KEY"
-d '{
"model": "deepseek-r1-distill-llama-70b",
"messages": [
{
"role": "user",
"content": "What is the weather like in Paris today?"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current temperature for a given location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "City and country e.g. Bogotá, Colombia"
}
},
"required": [
"location"
],
"additionalProperties": false
},
"strict": true
}
}
]}'
Recommended Configuration Parameters
Parameter Default Range Description
messages - - Array of message objects. Important: Avoid system prompts - include all instructions in the user message!
temperature 0.6 0.0 - 2.0 Controls randomness in responses. Lower values make responses more deterministic. Recommended range: 0.5-0.7 to prevent repetitions or incoherent outputs
max_completion_tokens 1024 - Maximum length of model's response. Default may be too low for complex reasoning - consider increasing for detailed step-by-step solutions
top_p 0.95 0.0 - 1.0 Controls diversity of token selection
stream false boolean Enables response streaming. Recommended for interactive reasoning tasks
stop null string/array Custom stop sequences
seed null integer Set for reproducible results. Important for benchmarking - run multiple tests with different seeds
json_mode - boolean Set to enable JSON mode for structured output.
reasoning_format raw "parsed", "raw", "hidden" Controls how model reasoning is presented in the response. Must be set to either parsed or hidden when using tool calling or JSON mode.
Optimizing Performance
Temperature and Token Management
The model performs best with temperature settings between 0.5-0.7, with lower values (closer to 0.5) producing more consistent mathematical proofs and higher values allowing for more creative problem-solving approaches. Monitor and adjust your token usage based on the complexity of your reasoning tasks - while the default max_completion_tokens is 1024, complex proofs may require higher limits.

Prompt Engineering
To ensure accurate, step-by-step reasoning while maintaining high performance:

DeepSeek-R1 works best when all instructions are included directly in user messages rather than system prompts.
Structure your prompts to request explicit validation steps and intermediate calculations.
Avoid few-shot prompting and go for zero-shot prompting only.
GroqCloud

Playground
Documentation
API Reference
Metrics
API Keys
Settings
Status
discord logoDiscord
Chat with us

XELOSZ
Personal
Documentation
Get Started
Overview
Quickstart
OpenAI Compatibility
Models
Rate Limits
Features
Text
Speech
Reasoning
Vision
Advanced Features
Batch Processing
Flex Processing
Content Moderation
Prefilling
Tool Use
Developer Resources
Groq Libraries
Groq Badge
Examples
Applications Showcase
Resources
Prompting Guide
Integrations
Agno
AutoGen
Arize
Composio
CrewAI
E2B
Gradio
JigsawStack
LangChain
LlamaIndex
LiteLLM
LiveKit
Toolhouse
Vercel
xRx
Support & Guidelines
Errors
Changelog
Policies & Notices
Chat Completion Models
The Groq Chat Completions API processes a series of messages and generates output responses. These models can perform multi-turn discussions or tasks that require only one interaction.

For details about the parameters, visit the reference page.

JSON mode (beta)
JSON mode is a beta feature that guarantees all chat completions are valid JSON.

Usage:

Set "response_format": {"type": "json_object"} in your chat completion request
Add a description of the desired JSON structure within the system prompt (see below for example system prompts)
Recommendations for best beta results:

Mixtral performs best at generating JSON, followed by Gemma, then Llama
Use pretty-printed JSON instead of compact JSON
Keep prompts concise
Beta Limitations:

Does not support streaming
Does not support stop sequences
Error Code:

Groq will return a 400 error with an error code of json_validate_failed if JSON generation fails.
Example system prompts:

You are a legal advisor who summarizes documents in JSON

You are a data analyst API capable of sentiment analysis that responds in JSON. The JSON schema should include
{
"sentiment_analysis": {
"sentiment": "string (positive, negative, neutral)",
"confidence_score": "number (0-1)"
# Include additional fields as required
}
}
Generating Chat Completions with groq SDK
Code Overview
Python
JavaScript

pip install groq

Performing a basic Chat Completion

from groq import Groq

client = Groq()

chat_completion = client.chat.completions.create(
#
# Required parameters
#
messages=[
# Set an optional system message. This sets the behavior of the
# assistant and can be used to provide specific instructions for
# how it should behave throughout the conversation.
{
"role": "system",
"content": "you are a helpful assistant."
},
# Set a user message for the assistant to respond to.
{
"role": "user",
"content": "Explain the importance of fast language models",
}
],

text
# The language model which will generate the completion.
model="llama-3.3-70b-versatile",

#
# Optional parameters
#

# Controls randomness: lowering results in less random completions.
# As the temperature approaches zero, the model will become deterministic
# and repetitive.
temperature=0.5,

# The maximum number of tokens to generate. Requests can use up to
# 32,768 tokens shared between prompt and completion.
max_completion_tokens=1024,

# Controls diversity via nucleus sampling: 0.5 means half of all
# likelihood-weighted options are considered.
top_p=1,

# A stop sequence is a predefined or user-specified text string that
# signals an AI to stop generating content, ensuring its responses
# remain focused and concise. Examples include punctuation marks and
# markers like "[end]".
stop=None,

# If set, partial message deltas will be sent.
stream=False,

)

Print the completion returned by the LLM.

print(chat_completion.choices[0].message.content)

Streaming a Chat Completion
To stream a completion, simply set the parameter stream=True. Then the completion function will return an iterator of completion deltas rather than a single, full completion.

from groq import Groq

client = Groq()

stream = client.chat.completions.create(
#
# Required parameters
#
messages=[
# Set an optional system message. This sets the behavior of the
# assistant and can be used to provide specific instructions for
# how it should behave throughout the conversation.
{
"role": "system",
"content": "you are a helpful assistant."
},
# Set a user message for the assistant to respond to.
{
"role": "user",
"content": "Explain the importance of fast language models",
}
],

text
# The language model which will generate the completion.
model="llama-3.3-70b-versatile",

#
# Optional parameters
#

# Controls randomness: lowering results in less random completions.
# As the temperature approaches zero, the model will become deterministic
# and repetitive.
temperature=0.5,

# The maximum number of tokens to generate. Requests can use up to
# 2048 tokens shared between prompt and completion.
max_completion_tokens=1024,

# Controls diversity via nucleus sampling: 0.5 means half of all
# likelihood-weighted options are considered.
top_p=1,

# A stop sequence is a predefined or user-specified text string that
# signals an AI to stop generating content, ensuring its responses
# remain focused and concise. Examples include punctuation marks and
# markers like "[end]".
stop=None,

# If set, partial message deltas will be sent.
stream=True,

)

Print the incremental deltas returned by the LLM.

for chunk in stream:
print(chunk.choices[0].delta.content, end="")

Performing a Chat Completion with a stop sequence

from groq import Groq

client = Groq()

chat_completion = client.chat.completions.create(
#
# Required parameters
#
messages=[
# Set an optional system message. This sets the behavior of the
# assistant and can be used to provide specific instructions for
# how it should behave throughout the conversation.
{
"role": "system",
"content": "you are a helpful assistant."
},
# Set a user message for the assistant to respond to.
{
"role": "user",
"content": "Count to 10. Your response must begin with "1, ". example: 1, 2, 3, ...",
}
],

text
# The language model which will generate the completion.
model="llama-3.3-70b-versatile",

#
# Optional parameters
#

# Controls randomness: lowering results in less random completions.
# As the temperature approaches zero, the model will become deterministic
# and repetitive.
temperature=0.5,

# The maximum number of tokens to generate. Requests can use up to
# 2048 tokens shared between prompt and completion.
max_completion_tokens=1024,

# Controls diversity via nucleus sampling: 0.5 means half of all
# likelihood-weighted options are considered.
top_p=1,

# A stop sequence is a predefined or user-specified text string that
# signals an AI to stop generating content, ensuring its responses
# remain focused and concise. Examples include punctuation marks and
# markers like "[end]".
# For this example, we will use ", 6" so that the llm stops counting at 5.
# If multiple stop values are needed, an array of string may be passed,
# stop=[", 6", ", six", ", Six"]
stop=", 6",

# If set, partial message deltas will be sent.
stream=False,

)

Print the completion returned by the LLM.

print(chat_completion.choices[0].message.content)
Performing an Async Chat Completion
Simply use the Async client to enable asyncio

import asyncio

from groq import AsyncGroq

async def main():
client = AsyncGroq()

text
chat_completion = await client.chat.completions.create(
    #
    # Required parameters
    #
    messages=[
        # Set an optional system message. This sets the behavior of the
        # assistant and can be used to provide specific instructions for
        # how it should behave throughout the conversation.
        {
            "role": "system",
            "content": "you are a helpful assistant."
        },
        # Set a user message for the assistant to respond to.
        {
            "role": "user",
            "content": "Explain the importance of fast language models",
        }
    ],

    # The language model which will generate the completion.
    model="llama-3.3-70b-versatile",

    #
    # Optional parameters
    #

    # Controls randomness: lowering results in less random completions.
    # As the temperature approaches zero, the model will become
    # deterministic and repetitive.
    temperature=0.5,

    # The maximum number of tokens to generate. Requests can use up to
    # 2048 tokens shared between prompt and completion.
    max_completion_tokens=1024,

    # Controls diversity via nucleus sampling: 0.5 means half of all
    # likelihood-weighted options are considered.
    top_p=1,

    # A stop sequence is a predefined or user-specified text string that
    # signals an AI to stop generating content, ensuring its responses
    # remain focused and concise. Examples include punctuation marks and
    # markers like "[end]".
    stop=None,

    # If set, partial message deltas will be sent.
    stream=False,
)

# Print the completion returned by the LLM.
print(chat_completion.choices[0].message.content)

asyncio.run(main())
Streaming an Async Chat Completion

import asyncio

from groq import AsyncGroq

async def main():
client = AsyncGroq()

text
stream = await client.chat.completions.create(
    #
    # Required parameters
    #
    messages=[
        # Set an optional system message. This sets the behavior of the
        # assistant and can be used to provide specific instructions for
        # how it should behave throughout the conversation.
        {
            "role": "system",
            "content": "you are a helpful assistant."
        },
        # Set a user message for the assistant to respond to.
        {
            "role": "user",
            "content": "Explain the importance of fast language models",
        }
    ],

    # The language model which will generate the completion.
    model="llama-3.3-70b-versatile",

    #
    # Optional parameters
    #

    # Controls randomness: lowering results in less random completions.
    # As the temperature approaches zero, the model will become
    # deterministic and repetitive.
    temperature=0.5,

    # The maximum number of tokens to generate. Requests can use up to
    # 2048 tokens shared between prompt and completion.
    max_completion_tokens=1024,

    # Controls diversity via nucleus sampling: 0.5 means half of all
    # likelihood-weighted options are considered.
    top_p=1,

    # A stop sequence is a predefined or user-specified text string that
    # signals an AI to stop generating content, ensuring its responses
    # remain focused and concise. Examples include punctuation marks and
    # markers like "[end]".
    stop=None,

    # If set, partial message deltas will be sent.
    stream=True,
)

# Print the incremental deltas returned by the LLM.
async for chunk in stream:
    print(chunk.choices[0].delta.content, end="")

asyncio.run(main())
JSON Mode

from typing import List, Optional
import json

from pydantic import BaseModel
from groq import Groq

groq = Groq()

Data model for LLM to generate

class Ingredient(BaseModel):
name: str
quantity: str
quantity_unit: Optional[str]

class Recipe(BaseModel):
recipe_name: str
ingredients: List[Ingredient]
directions: List[str]

def get_recipe(recipe_name: str) -> Recipe:
chat_completion = groq.chat.completions.create(
messages=[
{
"role": "system",
"content": "You are a recipe database that outputs recipes in JSON.\n"
# Pass the json schema to the model. Pretty printing improves results.
f" The JSON object must use the schema: {json.dumps(Recipe.model_json_schema(), indent=2)}",
},
{
"role": "user",
"content": f"Fetch a recipe for {recipe_name}",
},
],
model="llama3-70b-8192",
temperature=0,
# Streaming is not supported in JSON mode
stream=False,
# Enable JSON mode by setting the response format
response_format={"type": "json_object"},
)
return Recipe.model_validate_json(chat_completion.choices[0].message.content)

def print_recipe(recipe: Recipe):
print("Recipe:", recipe.recipe_name)

text
print("\nIngredients:")
for ingredient in recipe.ingredients:
    print(
        f"- {ingredient.name}: {ingredient.quantity} {ingredient.quantity_unit or ''}"
    )
print("\nDirections:")
for step, direction in enumerate(recipe.directions, start=1):
    print(f"{step}. {direction}")

recipe = get_recipe("apple pie")
print_recipe(recipe)
GroqCloud

Playground
Documentation
API Reference
Metrics
API Keys
Settings
Status
discord logoDiscord
Chat with us

XELOSZ
Personal
Documentation
Get Started
Overview
Quickstart
OpenAI Compatibility
Models
Rate Limits
Features
Text
Speech
Reasoning
Vision
Advanced Features
Batch Processing
Flex Processing
Content Moderation
Prefilling
Tool Use
Developer Resources
Groq Libraries
Groq Badge
Examples
Applications Showcase
Resources
Prompting Guide
Integrations
Agno
AutoGen
Arize
Composio
CrewAI
E2B
Gradio
JigsawStack
LangChain
LlamaIndex
LiteLLM
LiveKit
Toolhouse
Vercel
xRx
Support & Guidelines
Errors
Changelog
Policies & Notices
Introduction to Tool Use
Tool use is a powerful feature that allows Large Language Models (LLMs) to interact with external resources, such as APIs, databases, and the web, to gather dynamic data they wouldn't otherwise have access to in their pre-trained (or static) state and perform actions beyond simple text generation.

Tool use bridges the gap between the data that the LLMs were trained on with dynamic data and real-world actions, which opens up a wide array of realtime use cases for us to build powerful applications with, especially with Groq's insanely fast inference speed. 🚀

Supported Models
Model ID Tool Use Support? Parallel Tool Use Support? JSON Mode Support?
qwen-2.5-32b Yes Yes Yes
deepseek-r1-distill-qwen-32b Yes Yes Yes
deepseek-r1-distill-llama-70b Yes Yes Yes
llama-3.3-70b-versatile Yes Yes Yes
llama-3.1-8b-instant Yes Yes Yes
mixtral-8x7b-32768 Yes No Yes
gemma2-9b-it Yes No Yes
How Tool Use Works
Groq API tool use structure is compatible with OpenAI's tool use structure, which allows for easy integration. See the following cURL example of a tool use request:

curl https://api.groq.com/openai/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer $GROQ_API_KEY"
-d '{
"model": "llama-3.3-70b-versatile",
"messages": [
{
"role": "user",
"content": "What'''s the weather like in Boston today?"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"]
}
},
"required": ["location"]
}
}
}
],
"tool_choice": "auto"
}'

To integrate tools with Groq API, follow these steps:

Provide tools (or predefined functions) to the LLM for performing actions and accessing external data in real-time in addition to your user prompt within your Groq API request
Define how the tools should be used to teach the LLM how to use them effectively (e.g. by defining input and output formats)
Let the LLM autonomously decide whether or not the provided tools are needed for a user query by evaluating the user query, determining whether the tools can enhance its response, and utilizing the tools accordingly
Extract tool input, execute the tool code, and return results
Let the LLM use the tool result to formulate a response to the original prompt
This process allows the LLM to perform tasks such as real-time data retrieval, complex calculations, and external API interaction, all while maintaining a natural conversation with our end user.

Tool Use with Groq
Groq API endpoints support tool use to almost instantly deliver structured JSON output that can be used to directly invoke functions from desired external resources.

Tools Specifications
Tool use is part of the Groq API chat completion request payload.

Tool Call and Tool Response Structure
Tool Call Structure

Groq API tool calls are structured to be OpenAI-compatible. The following is an example tool call structure:

{
"model": "llama-3.3-70b-versatile",
"messages": [
{
"role": "system",
"content": "You are a weather assistant. Use the get_weather function to retrieve weather information for a given location."
},
{
"role": "user",
"content": "What's the weather like in New York today?"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The unit of temperature to use. Defaults to fahrenheit."
}
},
"required": ["location"]
}
}
}
],
"tool_choice": "auto",
"max_completion_tokens": 4096
}'

Tool Call Response

The following is an example tool call response based on the above:

"model": "llama-3.3-70b-versatile",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"tool_calls": [{
"id": "call_d5wg",
"type": "function",
"function": {
"name": "get_weather",
"arguments": "{"location": "New York, NY"}"
}
}]
},
"logprobs": null,
"finish_reason": "tool_calls"
}],

When a model decides to use a tool, it returns a response with a tool_calls object containing:

id: a unique identifier for the tool call
type: the type of tool call, i.e. function
name: the name of the tool being used
parameters: an object containing the input being passed to the tool
Setting Up Tools
To get started, let's go through an example of tool use with Groq API that you can use as a base to build more tools on your own.

Step 1: Create Tool
Let's install Groq SDK, set up our Groq client, and create a function called calculate to evaluate a mathematical expression that we will represent as a tool.

Note: In this example, we're defining a function as our tool, but your tool can be any function or an external resource (e.g. dabatase, web search engine, external API).

Python
JavaScript

pip install groq

from groq import Groq
import json

Initialize the Groq client

client = Groq()

Specify the model to be used (we recommend Llama 3.3 70B)

MODEL = 'llama-3.3-70b-versatile'

def calculate(expression):
"""Evaluate a mathematical expression"""
try:
# Attempt to evaluate the math expression
result = eval(expression)
return json.dumps({"result": result})
except:
# Return an error message if the math expression is invalid
return json.dumps({"error": "Invalid expression"})
Step 2: Pass Tool Definition and Messages to Model
Next, we'll define our calculate tool within an array of available tools and call our Groq API chat completion. You can read more about tool schema and supported required and optional fields above in Tool Specifications.

By defining our tool, we'll inform our model about what our tool does and have the model decide whether or not to use the tool. We should be as descriptive and specific as possible for our model to be able to make the correct tool use decisions.

In addition to our tools array, we will provide our messages array (e.g. containing system prompt, assistant prompt, and/or user prompt).

Step 3: Receive and Handle Tool Results
After executing our chat completion, we'll extract our model's response and check for tool calls.

If the model decides that no tools should be used and does not generate a tool or function call, then the response will be a normal chat completion (i.e. response_message = response.choices[0].message) with a direct model reply to the user query.

If the model decides that tools should be used and generates a tool or function call, we will:

Define available tool or function,
Add the model's response to the conversation by appending our message
Process the tool call and add the tool response to our message
Make a second Groq API call with the updated conversation
Return the final response
Python
JavaScript

imports calculate function from step 1

def run_conversation(user_prompt):
# Initialize the conversation with system and user messages
messages=[
{
"role": "system",
"content": "You are a calculator assistant. Use the calculate function to perform mathematical operations and provide the results."
},
{
"role": "user",
"content": user_prompt,
}
]
# Define the available tools (i.e. functions) for our model to use
tools = [
{
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate a mathematical expression",
"parameters": {
"type": "object",
"properties": {
"expression": {
"type": "string",
"description": "The mathematical expression to evaluate",
}
},
"required": ["expression"],
},
},
}
]
# Make the initial API call to Groq
response = client.chat.completions.create(
model=MODEL, # LLM to use
messages=messages, # Conversation history
stream=False,
tools=tools, # Available tools (i.e. functions) for our LLM to use
tool_choice="auto", # Let our LLM decide when to use tools
max_completion_tokens=4096 # Maximum number of tokens to allow in our response
)
# Extract the response and any tool call responses
response_message = response.choices[0].message
tool_calls = response_message.tool_calls
if tool_calls:
# Define the available tools that can be called by the LLM
available_functions = {
"calculate": calculate,
}
# Add the LLM's response to the conversation
messages.append(response_message)

text
    # Process each tool call
    for tool_call in tool_calls:
        function_name = tool_call.function.name
        function_to_call = available_functions[function_name]
        function_args = json.loads(tool_call.function.arguments)
        # Call the tool and get the response
        function_response = function_to_call(
            expression=function_args.get("expression")
        )
        # Add the tool response to the conversation
        messages.append(
            {
                "tool_call_id": tool_call.id, 
                "role": "tool", # Indicates this message is from tool use
                "name": function_name,
                "content": function_response,
            }
        )
    # Make a second API call with the updated conversation
    second_response = client.chat.completions.create(
        model=MODEL,
        messages=messages
    )
    # Return the final response
    return second_response.choices[0].message.content

Example usage

user_prompt = "What is 25 * 4 + 10?"
print(run_conversation(user_prompt))
Routing System
If you use our models fine-tuned for tool use, we recommended to use them as part of a routing system:

Query Analysis: Implement a routing system that analyzes incoming user queries to determine their nature and requirements.
Model Selection: Based on the query analysis, route the request to the most appropriate model:
For queries involving function calling, API interactions, or structured data manipulation, use the Llama 3 Groq Tool Use models.
For general knowledge, open-ended conversations, or tasks not specifically related to tool use, route to a general-purpose language model, such as Llama 3 70B.
The following is the calculate tool we built in the above steps enhanced to include a routing system that routes our request to Llama 3.3 70B if the user query does not require the tool:

Python
JavaScript

from groq import Groq
import json

Initialize the Groq client

client = Groq()

Define models

ROUTING_MODEL = "llama3-70b-8192"
TOOL_USE_MODEL = "llama-3.3-70b-versatile"
GENERAL_MODEL = "llama3-70b-8192"

def calculate(expression):
"""Tool to evaluate a mathematical expression"""
try:
result = eval(expression)
return json.dumps({"result": result})
except:
return json.dumps({"error": "Invalid expression"})

def route_query(query):
"""Routing logic to let LLM decide if tools are needed"""
routing_prompt = f"""
Given the following user query, determine if any tools are needed to answer it.
If a calculation tool is needed, respond with 'TOOL: CALCULATE'.
If no tools are needed, respond with 'NO TOOL'.

text
User query: {query}

Response:
"""

response = client.chat.completions.create(
    model=ROUTING_MODEL,
    messages=[
        {"role": "system", "content": "You are a routing assistant. Determine if tools are needed based on the user query."},
        {"role": "user", "content": routing_prompt}
    ],
    max_completion_tokens=20  # We only need a short response
)

routing_decision = response.choices[0].message.content.strip()

if "TOOL: CALCULATE" in routing_decision:
    return "calculate tool needed"
else:
    return "no tool needed"

def run_with_tool(query):
"""Use the tool use model to perform the calculation"""
messages = [
{
"role": "system",
"content": "You are a calculator assistant. Use the calculate function to perform mathematical operations and provide the results.",
},
{
"role": "user",
"content": query,
}
]
tools = [
{
"type": "function",
"function": {
"name": "calculate",
"description": "Evaluate a mathematical expression",
"parameters": {
"type": "object",
"properties": {
"expression": {
"type": "string",
"description": "The mathematical expression to evaluate",
}
},
"required": ["expression"],
},
},
}
]
response = client.chat.completions.create(
model=TOOL_USE_MODEL,
messages=messages,
tools=tools,
tool_choice="auto",
max_completion_tokens=4096
)
response_message = response.choices[0].message
tool_calls = response_message.tool_calls
if tool_calls:
messages.append(response_message)
for tool_call in tool_calls:
function_args = json.loads(tool_call.function.arguments)
function_response = calculate(function_args.get("expression"))
messages.append(
{
"tool_call_id": tool_call.id,
"role": "tool",
"name": "calculate",
"content": function_response,
}
)
second_response = client.chat.completions.create(
model=TOOL_USE_MODEL,
messages=messages
)
return second_response.choices[0].message.content
return response_message.content

def run_general(query):
"""Use the general model to answer the query since no tool is needed"""
response = client.chat.completions.create(
model=GENERAL_MODEL,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": query}
]
)
return response.choices[0].message.content

def process_query(query):
"""Process the query and route it to the appropriate model"""
route = route_query(query)
if route == "calculate":
response = run_with_tool(query)
else:
response = run_general(query)

text
return {
    "query": query,
    "route": route,
    "response": response
}

Example usage

if name == "main":
queries = [
"What is the capital of the Netherlands?",
"Calculate 25 * 4 + 10"
]

text
for query in queries:
    result = process_query(query)
    print(f"Query: {result['query']}")
    print(f"Route: {result['route']}")
    print(f"Response: {result['response']}\n")

Parallel Tool Use
We learned about tool use and built single-turn tool use examples above. Now let's take tool use a step further and imagine a workflow where multiple tools can be called simultaneously, enabling more efficient and effective responses.

This concept is known as parallel tool use and is key for building agentic workflows that can deal with complex queries, which is a great example of where inference speed becomes increasingly important (and thankfully we can access fast inference speed with Groq API).

Here's an example of parallel tool use with a tool for getting the temperature and the tool for getting the weather condition to show parallel tool use with Groq API in action:

Python
JavaScript

import json
from groq import Groq
import os

Initialize Groq client

client = Groq()
model = "llama-3.3-70b-versatile"

Define weather tools

def get_temperature(location: str):
# This is a mock tool/function. In a real scenario, you would call a weather API.
temperatures = {"New York": 22, "London": 18, "Tokyo": 26, "Sydney": 20}
return temperatures.get(location, "Temperature data not available")

def get_weather_condition(location: str):
# This is a mock tool/function. In a real scenario, you would call a weather API.
conditions = {"New York": "Sunny", "London": "Rainy", "Tokyo": "Cloudy", "Sydney": "Clear"}
return conditions.get(location, "Weather condition data not available")

Define system messages and tools

messages = [
{"role": "system", "content": "You are a helpful weather assistant."},
{"role": "user", "content": "What's the weather like in New York and London?"},
]

tools = [
{
"type": "function",
"function": {
"name": "get_temperature",
"description": "Get the temperature for a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The name of the city",
}
},
"required": ["location"],
},
},
},
{
"type": "function",
"function": {
"name": "get_weather_condition",
"description": "Get the weather condition for a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The name of the city",
}
},
"required": ["location"],
},
},
}
]

Make the initial request

response = client.chat.completions.create(
model=model, messages=messages, tools=tools, tool_choice="auto", max_completion_tokens=4096
)

response_message = response.choices[0].message
tool_calls = response_message.tool_calls

Process tool calls

messages.append(response_message)

available_functions = {
"get_temperature": get_temperature,
"get_weather_condition": get_weather_condition,
}

for tool_call in tool_calls:
function_name = tool_call.function.name
function_to_call = available_functions[function_name]
function_args = json.loads(tool_call.function.arguments)
function_response = function_to_call(**function_args)

text
messages.append(
    {
        "role": "tool",
        "content": str(function_response),
        "tool_call_id": tool_call.id,
    }
)

Make the final request with tool call results

final_response = client.chat.completions.create(
model=model, messages=messages, tools=tools, tool_choice="auto", max_completion_tokens=4096
)

print(final_response.choices[0].message.content)

Error Handling
Groq API tool use is designed to verify whether a model generates a valid tool call object. When a model fails to generate a valid tool call object, Groq API will return a 400 error with an explanation in the "failed_generation" field of the JSON body that is returned.

Next Steps
For more information and examples of working with multiple tools in parallel using Groq API and Instructor, see our Groq API Cookbook tutorial here.

Tool Use with Structured Outputs (Python)
Groq API offers best-effort matching for parameters, which means the model could occasionally miss parameters or misinterpret types for more complex tool calls. We recommend the Instuctor library to simplify the process of working with structured data and to ensure that the model's output adheres to a predefined schema.

Here's an example of how to implement tool use using the Instructor library with Groq API:

pip install instructor pydantic

import instructor
from pydantic import BaseModel, Field
from groq import Groq

Define the tool schema

tool_schema = {
"name": "get_weather_info",
"description": "Get the weather information for any location.",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The location for which we want to get the weather information (e.g., New York)"
}
},
"required": ["location"]
}
}

Define the Pydantic model for the tool call

class ToolCall(BaseModel):
input_text: str = Field(description="The user's input text")
tool_name: str = Field(description="The name of the tool to call")
tool_parameters: str = Field(description="JSON string of tool parameters")

class ResponseModel(BaseModel):
tool_calls: list[ToolCall]

Patch Groq() with instructor

client = instructor.from_groq(Groq(), mode=instructor.Mode.JSON)

def run_conversation(user_prompt):
# Prepare the messages
messages = [
{
"role": "system",
"content": f"You are an assistant that can use tools. You have access to the following tool: {tool_schema}"
},
{
"role": "user",
"content": user_prompt,
}
]

text
# Make the Groq API call
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    response_model=ResponseModel,
    messages=messages,
    temperature=0.7,
    max_completion_tokens=1000,
)

return response.tool_calls

Example usage

user_prompt = "What's the weather like in San Francisco?"
tool_calls = run_conversation(user_prompt)

for call in tool_calls:
print(f"Input: {call.input_text}")
print(f"Tool: {call.tool_name}")
print(f"Parameters: {call.tool_parameters}")
print()
Benefits of Using Structured Outputs
Type Safety: Pydantic models ensure that output adheres to the expected structure, reducing the risk of errors.
Automatic Validation: Instructor automatically validates the model's output against the defined schema.
Next Steps
For more information and examples of working with structured outputs using Groq API and Instructor, see our Groq API Cookbook tutorial here.

Best Practices
Provide detailed tool descriptions for optimal performance.
We recommend tool use with the Instructor library for structured outputs.
Use the fine-tuned Llama 3 models by Groq or the Llama 3.1 models for your applications that require tool use.
Implement a routing system when using fine-tuned models in your workflow.
Handle tool execution errors by returning error messages with "is_error": true.
GroqCloud

VERY VERY IMPORTANT I WANT THE CODE IN STREAMING PLS and i want to see the cot with the tools call like this pls :
"""
I should start with a web search to get an overview of current cancer treatments and any recent advancements.

So, my first action would be:

Searching... Okay, so from the web search, I've got a bunch of results talking about recent advances in cancer treatment. There's mention of cancer vaccines, precision medicine, immunotherapy, AI in cancer research, and some specific types of treatments like carbon ion therapy. It seems like there are multiple promising leads.
"""