Prediction stability measures whether a model produces the same output when given the same input and configuration repeatedly. In knowledge-graph extraction, unstable predictions can omit entities or relations on subsequent runs, reducing confidence in the resulting graph. section 5.3.2
Evaluate and improve prediction stability through a practical testing loop:
- Run the same prompt on the same test dataset multiple times when using non-zero temperature. section 5.3.3
- Measure output overlap across runs rather than trusting a single generated result. section 5.3.3
- Reduce variation with an unambiguous task formulation and very low temperature; factual extraction uses temperature 0. section 5.3.3
- Unit-test prompt changes with snippets and expected outputs, then inspect failures and success rates in bulk. section 5.3.3
Repeated knowledge-graph extraction
A prompt extracts entities and relations from the same paragraph twice with the same model configuration. The second output omits the theory of Raman spectra and the straight calorimetric method, while handling Mayer’s personality traits differently. The disagreement exposes low prediction stability and motivates constrained entity and relation lists. section 5.3.2
Listing 5.2 OpenAI API call
import os
import time
from dotenv import load_dotenv
from openai import OpenAI
from listing_2 import prompt_segments
_ = load_dotenv() #1
OPENAI_MODEL = "gpt-4o-mini"
def openai_query(client, prompt_segments: dict, #2
query: str):
messages = [
{"role": "system", "content": prompt_segments['task']},
{"role": "user", "content": prompt_segments['example']},
{"role": "assistant", "content": prompt_segments['example_output']},
{"role": "user", "content": query}
]
t_start = time.time()
response = client.chat.completions.create(model=OPENAI_MODEL,
messages=messages, temperature=0., max_tokens=2000) #3
print(response.choices[0].message.content)
print(f"\nTime: {round(time.time() - t_start, 1)} sec\n")
return response.choices[0].message.content
if __name__ == "__main__":
client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])
text = """ #4
JOHNS HOPKINS UNIVERSITY Chemistry Department:
Wednesday, November 9, 1932
WW visits the Dept. with Dr. Frazer (Baker Prof. Chem.). There are 239
undergraduate and 116 graduate students of chemistry, the latter group in-
cluding holders of the special State fellowships in chemistry under the
New Plan. D.H. Andrews (Prof.Chem.) is a physical chemist specializing in
thermodynamics. He is not present at the time of WW's call, but one of his
assistants explains his work. He is measuring specific heats of organic
compounds by a straight calorimetric method. This work is in its early
stages. He is also interested in making mechanical models of various atoms
from which can be demonstrated the theory of the Raman spectra.
J.B.Mayer (Assoc. in Chem.) is a former student of G. N. Lewis and works
with Max Born at Gottingen summers. He specializes in the energetics of
crystal lattices. His wife, last summer, prepared the new edition of
Born's treatise on this subject. In Mayer's laboratory Mrs. Wintner, wife
of the mathematician, is working on an experimental problem. Andrews says
that Mayer is young and impresses one as an enthusiastic and able man.
"""
openai_query(client, prompt_segments, text)
Inspect the API pattern for deterministic extraction: the call fixes the model configuration and sets temperature to 0 before returning generated relations. section 5.3.3
