Visual question answering answers a natural-language question about an image. It combines an image URL and question as multimodal input, then returns the answer as text. section 6.2.1
A visual question-answering request contains:
- A URL leading to an image. section 6.2.1
- A natural-language question about the image. section 6.2.1
- A user message whose content list identifies each element as text or image_url. section 6.2.1
- A text response generated by GPT-4o. section 6.2.2
Classifying fruit images
To find banana pictures for a logo project, ask “Is this a banana ("Yes","No")?” for each image URL and retain images receiving “Yes.” This turns visual question answering into a simple classification method. section 6.2.3
Listing 6.1 Answering questions about images via language models
import argparse
import openai
import time
client = openai.OpenAI()
def analyze_image(image_url, question): #1
""" Use language model to answer questions about image.
Args:
image_url: URL leading to image.
question: question about image.
Returns:
Answer generated by the language model.
"""
for nr_retries in range(1, 4):
try:
response = client.chat.completions.create(
model='gpt-4o',
messages=[ #2
{'role':'user', 'content':[
{'type':'text', 'text':question},
{'type':'image_url', 'image_url':{
'url':image_url
}
}]
}]
)
return response.choices[0].message.content
except:
time.sleep(nr_retries * 2)
raise Exception('Cannot query OpenAI model!')
if __name__ == '__main__':
parser = argparse.ArgumentParser() #3
parser.add_argument('imageurl', type=str, help='URL to image')
parser.add_argument('question', type=str, help='Question about image')
args = parser.parse_args()
answer = analyze_image(args.imageurl, args.question)
print(answer)
A short Python function passes the image URL and question to the chat completions endpoint, then prints the generated answer. section 6.2.2
