A tool is a capability an agent invokes to perform an operation or obtain information. Tools extend agents beyond conversation by enabling code execution, API calls, inspection, retrieval, and other task-directed actions. section 4.1.2 section 1.2
Tools commonly support three kinds of agent activity:
- Task completion, such as saving content to a file or generating an image. section 6.2.2 section 4.1.2
- Exploration and information gathering, such as fetching web content or inspecting an image. section 4.1.2 section 1.2
- Communication or coordination, including calls to other assistants and specialized agents. section 6.2.2 section 1.2
A tool may be semantic, using prompts or model behavior, or native, using executable code. In either form, it needs a clear name, useful documentation, and attachment to the agent or workflow that will invoke it. Providing maintained tools can be more reliable than letting agents generate code for every goal. section 7.1 section 4.1.2
Image generation with validation
Create a describe_image function that encodes an image, sends it to a GPT-4 vision model, and returns a description. Save the skill in AutoGen Studio, attach it to the primary assistant workflow, then ask the agent to generate a book cover and check whether its title is spelled correctly. The inspection tool acts as a quality check for the image-generation tool. section 4.1.2
Listing 4.2 describe_image.py
import base64
import requests
import os
def describe_image(image_path='animals.png') -> str:
"""
Uses GPT-4 Vision to inspect and describe the contents of the image.
:param input_path: str, the name of the PNG file to describe.
"""
api_key = os.environ['OPEN_API_KEY']
# Function to encode the image
def encode_image(image_path): #1
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode('utf-8')
# Getting the base64 string
base64_image = encode_image(image_path)
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}"
}
payload = {
"model": "gpt-4-turbo",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What’s in this image?"
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{base64_image}" #2
}
}
]
}
],
"max_tokens": 300
}
response = requests.post(
"https://api.openai.com/v1/chat/completions",
headers=headers,
json=payload)
return response.json()["choices"][0]["message"] #3
["content"] #3
Use the annotated Python listing to inspect a complete image-inspection tool, including file encoding, authentication, request construction, and model selection. section 4.1.2
