The F1-score is the harmonic mean of precision and recall for a classification class. It summarizes whether a model both finds relevant cases and avoids incorrect alerts, making it useful when class distributions are uneven. section 12.4.1
Cybersecurity article classification
A model classifying news articles as high or low risk reports an F1-score of 0.67 for each class. For low-risk articles, precision is 1.00 and recall is 0.50; for high-risk articles, precision is 0.50 and recall is 1.00. The shared F1-score exposes the trade-off that accuracy alone would hide. section 12.4.1
Interpret the F1-score alongside the other classification outputs:
- Compare class-level F1-scores rather than relying only on overall accuracy, especially when classes are imbalanced. section 12.4.1
- Use precision to assess incorrect alerts and recall to assess missed threats; the F1-score balances both concerns. section 12.5.1
- Inspect the confusion matrix before deciding whether the model meets operational requirements or needs optimization. section 12.5.1
Listing 12.4 NLP exercise
# Import required libraries
import pandas as pd #1
from sklearn.feature_extraction.text import #1
TfidfVectorizer #1
from sklearn.model_selection import train_test_split #1
from sklearn.linear_model import LogisticRegression #1
from sklearn.metrics import classification_report, #1
confusion_matrix #1
import seaborn as sns #1
import matplotlib.pyplot as plt #1
#1
# Step 1: Generate a balanced dataset #1
data = { #2
'Article': [ #2
"Ransomware targets financial institutions", #2
"Security patch released for major software vulnerability", #2
"Hackers steal millions of customer data in breach", #2
"Low-risk malware detected in corporate emails", #2
"Phishing campaign aims to steal credentials", #2
"Database vulnerability in hospital systems", #2
"Encryption policies updated to prevent breaches", #2
"Zero-day vulnerability in common software", #2
"Mandatory security training after an attack", #2
"Phishing attacks target large organizations" #2
] * 2, # Duplicate entries to balance classes #2
'Label': ['high-risk', 'low-risk', 'high-risk', #2
'low-risk', 'high-risk', #2
'low-risk', 'low-risk', 'high-risk', 'low-risk',
'high-risk'] * 2 # Balanced labels
}
df = pd.DataFrame(data) #3
# Step 2: Convert labels to numeric
df['Label'] = df['Label'].apply(lambda x:
1 if x == 'high-risk' else 0) #4
# Step 3: Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
df['Article'], df['Label'], test_size=0.3,
random_state=42) #5
# Step 4: Apply TF-IDF and Logistic Regression
vectorizer = TfidfVectorizer(stop_words='english') #6
X_train_tfidf = vectorizer.fit_transform(X_train) #7
X_test_tfidf = vectorizer.transform(X_test) #7
model = LogisticRegression() #8
model.fit(X_train_tfidf, y_train) #8
# Step 5: Make predictions
y_pred = model.predict(X_test_tfidf) #9
# Step 6: Evaluate the model
print("Classification Report:\n",
classification_report(y_test, y_pred)) #10
conf_matrix = confusion_matrix(y_test, y_pred) #11
# Step 7: Plot the confusion matrix (Grayscale)
plt.figure(figsize=(6, 4)) #12
sns.heatmap(conf_matrix, annot=True, cmap='Greys', #12
fmt='d', #12
xticklabels=['Low-Risk', 'High-Risk'], #12
yticklabels=['Low-Risk', 'High-Risk']) #12
plt.title('Confusion Matrix (Grayscale)') #12
plt.xlabel('Predicted Labels') #12
plt.ylabel('True Labels') #12
plt.show() #12
Run the TF-IDF and logistic-regression exercise to generate a classification report and confusion matrix for cybersecurity articles. section 12.4.1
