Precision measures how many instances predicted as a class actually belong to that class. For a high-risk classifier, it answers: how many high-risk alerts are correct? section 12.4.1
Use precision to judge the reliability of model alerts. Higher precision means fewer false positives, reducing unnecessary investigations; interpret it alongside recall, F1-score, accuracy, and the confusion matrix. section 12.4.1 section 12.5.1
Cybersecurity article classification
If precision for high-risk articles is 80%, four out of every five articles predicted as high-risk are actually high-risk. A classification report can expose different precision for low-risk and high-risk classes; one example reports 1.00 for class 0 and 0.50 for class 1. section 12.4.1
Listing 12.4 NLP exercise
# Import required libraries
import pandas as pd #1
from sklearn.feature_extraction.text import #1
TfidfVectorizer #1
from sklearn.model_selection import train_test_split #1
from sklearn.linear_model import LogisticRegression #1
from sklearn.metrics import classification_report, #1
confusion_matrix #1
import seaborn as sns #1
import matplotlib.pyplot as plt #1
#1
# Step 1: Generate a balanced dataset #1
data = { #2
'Article': [ #2
"Ransomware targets financial institutions", #2
"Security patch released for major software vulnerability", #2
"Hackers steal millions of customer data in breach", #2
"Low-risk malware detected in corporate emails", #2
"Phishing campaign aims to steal credentials", #2
"Database vulnerability in hospital systems", #2
"Encryption policies updated to prevent breaches", #2
"Zero-day vulnerability in common software", #2
"Mandatory security training after an attack", #2
"Phishing attacks target large organizations" #2
] * 2, # Duplicate entries to balance classes #2
'Label': ['high-risk', 'low-risk', 'high-risk', #2
'low-risk', 'high-risk', #2
'low-risk', 'low-risk', 'high-risk', 'low-risk',
'high-risk'] * 2 # Balanced labels
}
df = pd.DataFrame(data) #3
# Step 2: Convert labels to numeric
df['Label'] = df['Label'].apply(lambda x:
1 if x == 'high-risk' else 0) #4
# Step 3: Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
df['Article'], df['Label'], test_size=0.3,
random_state=42) #5
# Step 4: Apply TF-IDF and Logistic Regression
vectorizer = TfidfVectorizer(stop_words='english') #6
X_train_tfidf = vectorizer.fit_transform(X_train) #7
X_test_tfidf = vectorizer.transform(X_test) #7
model = LogisticRegression() #8
model.fit(X_train_tfidf, y_train) #8
# Step 5: Make predictions
y_pred = model.predict(X_test_tfidf) #9
# Step 6: Evaluate the model
print("Classification Report:\n",
classification_report(y_test, y_pred)) #10
conf_matrix = confusion_matrix(y_test, y_pred) #11
# Step 7: Plot the confusion matrix (Grayscale)
plt.figure(figsize=(6, 4)) #12
sns.heatmap(conf_matrix, annot=True, cmap='Greys', #12
fmt='d', #12
xticklabels=['Low-Risk', 'High-Risk'], #12
yticklabels=['Low-Risk', 'High-Risk']) #12
plt.title('Confusion Matrix (Grayscale)') #12
plt.xlabel('Predicted Labels') #12
plt.ylabel('True Labels') #12
plt.show() #12
Trace precision through a TF-IDF and logistic-regression exercise that classifies cybersecurity articles and produces a classification report and confusion matrix. section 12.4.1
