liveBook · Manning

Feature selection

As explained in Build AI Drug Discovery Pipelines

Feature selection is the process of choosing a subset of input features for model construction or prediction.

Feature selection chooses a smaller set of molecular descriptors or fingerprint features before modeling. It improves interpretability and limits overfitting when datasets contain many correlated features or few ligands. section 6.3.1 section 3.4.1

A practical selection workflow can combine:

  • L1-regularized Lasso, which shrinks unimportant coefficients to zero and retains the nonzero features. section 6.3.1 section 3.4.1
  • Cross-validation and grid search to choose the regularization strength rather than selecting features from a single fit. section 6.3.1
  • A constrained downstream model, such as multiple linear regression using descriptor combinations limited by the available training data. section 6.3.1
  • Ridge regularization when preserving all features while reducing coefficient magnitude is preferable to removing features. section 4.3 section 3.4.1

Small-data QSAR workflow

For roughly 48 ligands with nearly four times as many descriptors, Lasso selects descriptors before multiple linear regression. Candidate models are screened using training R2, test-set Q2, descriptor significance, and finally simplicity when performance ties. section 6.3.1

Listing 6.7 Lasso Feature Selection and Exhaustive MLR Search
def mask_train_test_split(X, y, test_idxs):
    mask = y.index.isin(test_idxs)
    sc = StandardScaler() 
    X_train = sc.fit_transform(X[~mask])
    X_test = sc.transform(X[mask])
    return X_train, X_test, y[~mask], y[mask]
 
def lasso_feature_selection(X_train, X_test, y_train, y_test):
 
    param_grid = {'alpha': np.logspace(2, -6, num=1000)}
    lasso = Lasso()
 
    grid_search = GridSearchCV(
        lasso, param_grid, cv=5, scoring='r2', n_jobs=-1
    )
    grid_search.fit(X_train, y_train)
 
    best_alpha = grid_search.best_params_['alpha']
    best_model = grid_search.best_estimator_
 
    y_pred = best_model.predict(X_test)
    r2 = r2_score(y_test, y_pred)
 
    print(f"Best alpha: {best_alpha}")
    print(f"Best R2: {r2}")
 
    selected_features = np.where(best_model.coef_ != 0)[0]
    return best_alpha, selected_features
 
def compute_feature_pvals(model, X, y):
    # Calculate the standard errors of the coefficients
    n = len(y)
    p = X.shape[1]
    y_pred = model.predict(X)
    residuals = y - y_pred
    mse = np.sum(residuals**2) / (n - p - 1)
    var_b = mse * (np.linalg.inv(np.dot(X.T, X)).diagonal())
    std_err = np.sqrt(var_b)
 
    # Calculate the t-statistics and p-values
    t_stat = model.coef_ / std_err
    p_values = 2 * (1 - stats.t.cdf(np.abs(t_stat), n - p - 1))
 
    return np.round(p_values, 3)
 
def MLR_exhaustive_search(
    X_train, X_test, y_train, y_test,
    selected_features, threshold=0.7, max_descs=7
):
 
    best_score = -np.inf
    best_model = None
    best_p_values = None
    best_features = None
 
    X_train = X_train[:, selected_features]
    X_test = X_test[:, selected_features]
 
    for r in range(2, max_descs+1):
        for feature_combination in itertools.combinations(
            range(X_train.shape[1]), r
        ):
            model = LinearRegression()
            model.fit(X_train[:, feature_combination], y_train)
 
            y_train_pred = model.predict(X_train[:, feature_combination])
            y_test_pred = model.predict(X_test[:, feature_combination])
 
            r2_train = r2_score(y_train, y_train_pred)
            q2_test = r2_score(y_test, y_test_pred)
 
            if r2_train <= threshold or q2_test <= threshold:
                continue
 
            p_values = compute_feature_pvals(
                model, X_train[:, feature_combination], y_train
            )
            if q2_test > best_score and np.all(p_values < 0.05):
                best_score = q2_test
                best_model = model
                best_features = feature_combination
                best_p_values = p_values
 
    best_selected_features = [
        selected_features[feature] for feature in best_features
    ]
 
    print(f"Best Q2: {best_score}")
    print(f"Best Features: {best_features}")
    return best_model, best_selected_features, best_score, best_p_values
 
def paper_MLR_workflow(X, y, test_idxs, threshold=0.7, max_descs=7):
 
    X_train, X_test, y_train, y_test = mask_train_test_split(
        X, y, test_idxs
    )
 
    print("Begin Lasso Feature Selection")
    best_alpha, selected_features = lasso_feature_selection(
        X_train, X_test, y_train, y_test
    )
 
    print("Screen Performative MLR Models")
    return MLR_exhaustive_search(
        X_train, X_test, y_train, y_test,
        selected_features, threshold, max_descs
    )
 
best_model_kon, best_features_kon, best_score_kon, best_p_values_kon = \
    paper_MLR_workflow(
        hiv_tar1_lnkon.drop(columns=["lnkon"]),
        hiv_tar1_lnkon["lnkon"], test_idxs_paper
    )
 
best_model_koff, best_features_koff, best_score_koff, best_p_values_koff = paper_MLR_workflow(hiv_tar1_lnkoff.drop(columns=["lnkoff"]), hiv_tar1_lnkoff["lnkoff"], test_idxs_paper, threshold=0.6)
 
best_model_kd, best_features_kd, best_score_kd, best_p_values_kd = \
    paper_MLR_workflow(
        hiv_tar1_lnkd.drop(columns=["lnKD"]),
        hiv_tar1_lnkd["lnKD"], test_idxs_paper, max_descs=4
    )

The listing demonstrates standardized train/test handling, five-fold Lasso grid search, test prediction, and extraction of nonzero-coefficient feature indices. section 6.3.1

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains L1 and L2 penalties, cross-validated Lasso selection, descriptor-combination search, and performance filtering, but not selection theory beyond these workflows.
Examples
Demonstrates molecular descriptors, sparse fingerprint bits, a 48-ligand QSAR dataset, and a scikit-learn Lasso search listing.
What you can do after
Apply it via scikit-learn to select nonzero-coefficient features before fitting and evaluating a simpler regression model.
How it connects
Connects feature selection to regularization, Lasso regression, ridge regression, multicollinearity, QSAR modeling, and cross-validation.

Also explained in

Compare how 11 books explain feature selection →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.