Overfitting occurs when a model learns training-specific patterns that fail to transfer to unseen data. In receipt forgery detection, training heatmaps can align with forgery boxes while test heatmaps miss the regions and repeatedly assign high probabilities to superficial areas such as item prices or receipt notes. section 6.3.5
Recognize overfitting by comparing model behavior across splits:
- Compare training and test predictions against the actual forgery bounding boxes; weaker overlap on test receipts indicates poorer generalization. section 6.3.5
- Inspect where predicted probabilities concentrate; recurring high-probability regions unrelated to the forgery suggest reliance on training-set artifacts. section 6.3.5
- Reduce overfitting by adding labeled data when available or by revisiting the training pipeline and model architecture; multi-task learning adds text detection as a related auxiliary task. section 6.3.5 section 6.4.5
Receipt forgery heatmaps
A segmentation model closely matches forgery boxes on training receipts but highlights item prices and textual notes on test receipts. A multi-task model trained to detect both forgery regions and receipt text produces stronger overlaps on both splits. section 6.3.5 section 6.4.5
mode = "test"
transform = T.Compose([
T.Resize((256, 256)), #A
])
image_path = f'./findit2/torchdataset/test/1/<choose a file>.png' #B
original_image = Image.open(image_path).convert('RGB')
original_size = original_image.size #C
transformed_image = transform(original_image) #D
model.eval() #E
with torch.no_grad():
input_tensor = T.functional.to_tensor(transformed_image).unsqueeze(0).to(device) #F
output = model(input_tensor)['out']
predictions = torch.argmax(output.squeeze(), dim=0).detach().cpu().numpy() # Convert to numpy array for visualization
predictions_prob = torch.softmax(output, dim=1)[0][1]
predictions_resized = resize(predictions_prob, (original_size[1], original_size[0]), #G
order=0, preserve_range=True, anti_aliasing=False) #G
def draw_bounding_boxes(image, annotations):
draw = ImageDraw.Draw(image)
for ann in annotations['regions']:
shape_attr = ann['shape_attributes']
x, y, w, h = shape_attr['x'], shape_attr['y'], shape_attr['width'], shape_attr['height']
draw.rectangle([(x, y), (x+w, y+h)], outline="red", width=4)
return image
annotations = df[mode][df[mode]["image"]==image_path.split("/")[-1]]["forgery annotations"].iloc[0]
original_image = Image.open(image_path).convert('RGB') #H
drawn_image = draw_bounding_boxes(original_image, annotations) #I
drawn_image_array = np.array(drawn_image) #J
fig, ax = plt.subplots(1, 1, figsize=(10, 10)) #K
ax.imshow(drawn_image_array) #L
heatmap = ax.imshow(predictions_resized, cmap='hot', interpolation='nearest', alpha=0.5) #M
plt.colorbar(heatmap)
plt.show()
Run the test-image inference and heatmap workflow to inspect predictions on an unseen receipt and compare highlighted regions with annotations. section 6.3.5
































