If you want ChatGPT to output in a predictable format, you can use the OpenAI API to output JSON. There are two ways to get JSON output. The first method, released around June 2023, was initially called “function calling”, but has now been renamed to “tool_choice”.
The second method that can be used is the newer JSON mode, which was announced on Nov 6, 2023 at the OpenAI DevDay. JSON mode is only available in the newer gpt-3.5-turbo-1106 or gpt-4-1106-preview models.
How to use JSON mode with OpenAI
JSON mode is available as part of the Chat Completions API. You’ll notice in the command below that line 6 specifies the response format: "response_format": { "type": "json_object" }
Another requirement to JSON mode is that you’ll also need to remember to instruct the model to produce JSON through either a message in the conversation prompt or through the system message. I like always having it in the system message in case I forget to add it in the user conversation.
Below is an example that works consistently well for me. The prompt asks ChatGPT/OpenAI to return 5 random names that sound well together and output the results as JSON. You can run the below curl command in your terminal or import it into your API client. For API clients, I’ve been switching between RapidAPi’s tool (formerly known as Paw) and Insomnium. Be sure to replace $OPENAI_API_KEY with your own OpenAI API key.
The prompt I’m using is:
Create a list of 5 random first and last names that sound good together. Include the reason why they harmonize well together. Also include what gender the name is associated with. Output an object of ‘names’ in json with an array of ‘first’ and ‘last’ and ‘reason’ and ‘gender’ as the keys.
curl https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-3.5-turbo-1106",
"response_format": { "type": "json_object" },
"messages": [
{
"role": "system",
"content": "You are a helpful assistant designed to output JSON."
},
{
"role": "user",
"content": "Create a list of 5 random first and last names that sound good together. Include the reason why they harmonize well together. Also include what gender the name is associated with. Output an object of 'names' in json with an array of 'first' and 'last' and 'reason' and 'gender' as the keys."
}
]
}'
You’ll see the the JSON response we want in the “content” field of the response. The output will look something like this:
{
"id": "chatcmpl-123",
"object": "chat.completion",
"created": 1703803056,
"model": "gpt-3.5-turbo-1106",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "{\n \"names\": [\n {\n \"first\": \"Evelyn\",\n \"last\": \"Harrison\",\n \"reason\": \"Both names have a classic and timeless quality, and they flow smoothly together.\",\n \"gender\": \"female\"\n },\n {\n \"first\": \"Dexter\",\n \"last\": \"Archer\",\n \"reason\": \"Both names have a strong and assertive sound, and they complement each other well.\",\n \"gender\": \"male\"\n },\n {\n \"first\": \"Lena\",\n \"last\": \"Bennett\",\n \"reason\": \"Both names have a soft and elegant feel, and they create a harmonious combination.\",\n \"gender\": \"female\"\n },\n {\n \"first\": \"Elias\",\n \"last\": \"Porter\",\n \"reason\": \"Both names have a sophisticated and refined sound, and they sound great when said together.\",\n \"gender\": \"male\"\n },\n {\n \"first\": \"Aria\",\n \"last\": \"Davis\",\n \"reason\": \"Both names have a modern yet graceful vibe, and they blend well together.\",\n \"gender\": \"female\"\n }\n ]\n}"
},
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 85,
"completion_tokens": 251,
"total_tokens": 336
},
"system_fingerprint": "fp_44709d6fcb"
}
Here is the actual JSON from within the content field. I’s been formatted so it’s a bit easier to read:
{
"names": [
{
"first": "Evelyn",
"last": "Harrison",
"reason": "Both names have a classic and timeless quality, and they flow smoothly together.",
"gender": "female"
},
{
"first": "Dexter",
"last": "Archer",
"reason": "Both names have a strong and assertive sound, and they complement each other well.",
"gender": "male"
},
{
"first": "Lena",
"last": "Bennett",
"reason": "Both names have a soft and elegant feel, and they create a harmonious combination.",
"gender": "female"
},
{
"first": "Elias",
"last": "Porter",
"reason": "Both names have a sophisticated and refined sound, and they sound great when said together.",
"gender": "male"
},
{
"first": "Aria",
"last": "Davis",
"reason": "Both names have a modern yet graceful vibe, and they blend well together.",
"gender": "female"
}
]
}
You can experiment with changing the names of the keys or leaving some out completely from the prompt to see how the chat completion responds. I’ve had good luck getting consistently structured outputs without needing to use the seed parameter. If you do find yourself getting unpredictable results, try using the function calling method below.
How to use function calling (renamed as “tool use”) for JSON output
Before JSON mode existed, outputting JSON from OpenAI’s API would involve a feature called “function calling”. When I first started reading about this I thought it would actually call a function in my code somehow. Instead, function calling let’s OpenAI’s API respond with JSON, and you can use your backend to see if that returned JSON message has any data that can be handled with an existing API.
In OpenAI’s docs, and Henrik’s video below, they both use weather examples. Your user will use natural language and ask for the weather in a certain city and the OpenAI’s API will see that you have a function description available for that and return a JSON with the input data and function name. Your internal API can watch to see if that function name exists in the returned JSON, and if it does, you can run your API to grab weather information and send that JSON back to OpenAI where it will respond in natural language to your user using the info contained in that JSON object.
Henrik Knibert’s YouTube vid describes how to use function calling in conjunction with third-party APIs better than I can. The video also shows you how to set the function calling so you can just use OpenAI to only return JSON. The specific part about outputting JSON only is at timestamp: 11:33, however, it is worth watching the whole video to better understand other ways to use function calling for calling an external API.
Recently function_call has been depreciated in favor of tools in the Chat Completions API and Assitants API. Google and Claude have “function calling” features now as well, so it looks like the original naming might have stuck.
Using the same prompt example above, we ask for 5 random first and last names that sound good together. Here are the steps to use functions to get the same JSON output as the JSON mode example above.
First we’ll need to define the JSON schema of the response you’d like to see.
I asked ChatGPT to convert the above JSON data to schema. It might take a couple of tries to get a result that matches the examples in the cookbook docs. The schema has descriptions that seems to help the LLM understand the structure better by providing it an example similar to few-shot prompting:
{
"type": "object",
"properties": {
"names": {
"type": "array",
"items": {
"type": "object",
"properties": {
"first": {
"type": "string",
"description": "First name of the person."
},
"last": {
"type": "string",
"description": "Last name of the person."
},
"reason": {
"type": "string",
"description": "The rationale behind the name choice."
},
"gender": {
"type": "string",
"enum": ["male", "female"],
"description": "Gender of the person."
}
},
"required": ["first", "last", "reason", "gender"]
}
}
},
"required": ["names"]
}
The above schema goes into the parameters key within the newly added “tools” section as shown below.
curl https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-3.5-turbo-1106",
"messages": [
{
"role": "user",
"content": "Create a list of 5 random first and last names that sound good together. Include the reason why they harmonize well together. Also include what gender the name is associated with. "
}
],
"tools": [
{
"type": "function",
"function": {
"name": "generate_names",
"description": "Create a list of 5 random first and last names that sound good together.",
"parameters": {
"type": "object",
"properties": {
"names": {
"type": "array",
"items": {
"type": "object",
"properties": {
"first": {
"type": "string",
"description": "First name of the person."
},
"last": {
"type": "string",
"description": "Last name of the person."
},
"reason": {
"type": "string",
"description": "The rationale behind the name choice."
},
"gender": {
"type": "string",
"enum": ["male", "female"],
"description": "Gender of the person."
}
},
"required": ["first", "last", "reason", "gender"]
}
}
},
"required": ["names"]
}
}
}
],
"tool_choice": "auto"
}'
Running the above prompt results in the below response from the API. You’ll see the array of names that we want in the “arguments” key. The tool_choice parameter is used to enforce the usage of the function that we just created.
{
"id": "chatcmpl-321",
"object": "chat.completion",
"created": 1703830524,
"model": "gpt-3.5-turbo-1106",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_VVSd7PdTQceyeLZ2q6pYdss7",
"type": "function",
"function": {
"name": "generate_names",
"arguments": "{\"names\":[{\"first\":\"Ethan\",\"last\":\"Adams\",\"reason\":\"Ethan Adams has a strong, classic sound that appeals to both modern and traditional tastes.\",\"gender\":\"male\"},{\"first\":\"Olivia\",\"last\":\"Miller\",\"reason\":\"Olivia Miller has a soft, elegant sound that evokes grace and sophistication.\",\"gender\":\"female\"},{\"first\":\"Caleb\",\"last\":\"Roberts\",\"reason\":\"Caleb Roberts has a strong, assertive sound and creates a sense of confidence and leadership.\",\"gender\":\"male\"},{\"first\":\"Aria\",\"last\":\"Johnson\",\"reason\":\"Aria Johnson has a melodic and rhythmic flow that feels harmonious and balanced.\",\"gender\":\"female\"},{\"first\":\"Sebastian\",\"last\":\"Cole\",\"reason\":\"Sebastian Cole has a distinguished and refined sound that exudes sophistication and charm.\",\"gender\":\"male\"}]}"
}
}
]
},
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 144,
"completion_tokens": 177,
"total_tokens": 321
},
"system_fingerprint": "fp_44709d6fcc"
}
As you can see function calling is a bit more complicated than JSON mode, but can be very useful in ensuring reliable JSON responses from your OpenAI prompts. It can be powerful when expanded to include third-party API responses as described in the YouTube video above.
The next step for me is to figure out if streaming and rendering partial JSON data is possible. Looks like there is at least a few libraries (1, 2, 3, 4) that can from LLMs.