A novel task is sufficiently unlike the examples and structures represented in an LLM’s training data. LLMs often struggle with such tasks because useful patterns become scarce as the task grows more unique. section 4.3
Judge a task by how closely it resembles existing examples:
- Treat a task as less novel when abundant examples or structurally similar solutions exist. section 4.3
- Expect degradation when the task requires unusual extrapolation, even if the underlying problem seems manageable. section 4.3
- Verify outputs with citations or other checks, and avoid relying on LLMs for highly novel problems. section 4.3
Python versus Modula-3
Generating Python code to calculate π is familiar and usually works because many similar examples exist. Translating that function into obscure Modula-3 requires more extrapolation, so a reasonable-looking result needs careful checking. section 4.3
Listing 4.1 ChatGPT calculating pi in Python
def calculate_pi(terms):
pi = 0
for i in range(terms):
term = ((-1) ** i) / (2 * i + 1)
pi += term
pi *= 4
return pi
print(calculate_pi(1000000)) #1
Inspect the familiar Python implementation used as the low-novelty comparison. section 4.3
Listing 4.2 ChatGPT calculates pi in Modula-3
MODULE CalculatePi; #1
IMPORT IO, Real;
PROCEDURE CalculatePi(terms: INTEGER): REAL;
VAR
pi: REAL := 0.0;
i: INTEGER;
BEGIN
FOR i := 0 TO terms - 1 DO
pi := pi + ((-1.0) ** FLOAT(i)) / (2.0 * FLOAT(i) + 1.0); #2
END;
RETURN 4.0 * pi;
END CalculatePi;
BEGIN
IO.PutReal(CalculatePi(1000000), 0, 15); #3
END CalculatePi.
Compare the Modula-3 translation, where sparse examples make extrapolation more demanding. section 4.3
