In this class, we would cover Feature Selection. This class follows from Class 3 and Class 4 which discussed Data Preprocessing<\/p>\n
The following are covered here:<\/p>\n
<\/p>\n
Feature Selection also known as Variable selection is the method of selecting a subset of variables (or features) to be used for building a model. In order words, we want to reduce the number of features by selecting only features that are expected to produce the best performance.<\/p>\n
Here are some reason why we do feature selection<\/p>\n
<\/p>\n
Univariate feature selection is a technique that helps to select the variables that are strongly related with the output variable(predictor or dependent variable). In this demo, we would use the SelectKBest module from scikit-learn library.<\/p>\n
Follow the step below<\/p>\n
Step 1<\/strong> – Import your dataset as well as the relevant modules<\/p>\n <\/p>\n Step 2<\/strong> – Extract the features and predictor<\/p>\n <\/p>\n <\/p>\n Step 3<\/strong> – Select Best 5 Features <\/p>\n Step 4<\/strong> – View the Results <\/p>\n This is another feature selection technique that works by removing attributes recursively and then building the model with the remaining attributes or features. The RFE module of the sklearn library can be used to achieve RFE.<\/p>\n Step 1<\/strong> – Import your dataset and extract the features and predictor. Just modify the code for univariate model selection above.<\/p>\n Step 2<\/strong> – Import the\u00a0 linear_model library<\/p>\n <\/p>\n Step 3<\/strong> – Create and fit a regression object. In this example, we want to select 3 features. But feel free to increase to a different number.<\/p>\n <\/p>\n Step 4<\/strong> – Display the results<\/p>\n The three selected features are assigned rank 1 in the rankings array<\/p>\n See the video to learn about the the zip function used for iterating two lists at the same time.<\/p>\n Since this topic is quite involving, I am making a different class for it. But find some of my lessons below. However, the next class, we would review PCA using a simple demo<\/p>\n <\/p>\n Feature Importance is a technique for assigning scores to input features based on the how used they are for predicting the target variable. Simply put, feature importance helps use select the most important features.<\/p>\n Some types of feature importance includes:<\/p>\n In this tutorial, we would use the ExtraTreesClassifier module from the sklearn library. As before we would follow the steps:<\/p>\n Step 1<\/strong> – Import the wine dataset and split into wineX and wineY. You already know how to do this!<\/p>\n Step 2<\/strong> – Import the sklearn.ensemble module as the ExtraTreesClassifier is available there.<\/p>\n <\/p>\n Step 3<\/strong> – Create the model, fit the model<\/p>\n <\/p>\n Step 4<\/strong> – Display the Feature importances<\/p>\n <\/p>\n You can now see that the importance value for each feature is displayed.\u00a0 In the next class we would cover Principal Component Analysis (PCA)<\/p>\n# import the neccessary modules as well as your dataset<\/span>\r\nimport<\/span> pandas<\/span> as<\/span> pd<\/span>\r\nimport<\/span> numpy<\/span> as<\/span> np<\/span>\r\nimport<\/span> sklearn.feature_selection<\/span> as<\/span> fs<\/span>\r\nfrom<\/span> sklearn.feature_selection<\/span> import<\/span> chi2\r\npath =<\/span> '\/Users\/kindsonmunonye\/Datasets\/wine.csv'<\/span>\r\nwine_df =<\/span> pd.<\/span>read_csv(path)\r\n<\/pre>\n# Extract the features and the predictor as arrays<\/span>\r\nwineY =<\/span> wine_df.<\/span>iloc[0<\/span>:,0<\/span>:1<\/span>].<\/span>values\r\nwineX =<\/span> wine_df.<\/span>iloc[0<\/span>:,1<\/span>:].<\/span>values\r\n<\/pre>\n
\n<\/p>\n# Select Best 5 features<\/span>\r\nselector =<\/span> fs.<\/span>SelectKBest(score_func=<\/span>chi2, k=<\/span>5<\/span>)\r\nresult =<\/span> selector.<\/span>fit(wineX, wineY)\r\nbest_features =<\/span> result.<\/span>transform(wineX)\r\n<\/pre>\n
\n<\/p>\n# Display the output<\/span>\r\nnp.<\/span>set_printoptions(precision=<\/span>2<\/span>)\r\nprint<\/span>(result.<\/span>scores_)\r\nbest_features.<\/span>shape\r\n<\/pre>\n3. Recursive Feature Elimination(RFE)<\/strong><\/h4>\n
# Import the neccessary modules<\/span>\r\nimport<\/span> sklearn.linear_model<\/span> as<\/span> lm<\/span>\r\n<\/pre>\nreg_model =<\/span> lm.<\/span>LogisticRegression(max_iter=<\/span>10000<\/span>)\r\nrfe =<\/span> RFE(reg_model, n_features_to_select=<\/span>3<\/span>)\r\nfit =<\/span> rfe.<\/span>fit(wineX, wineY.<\/span>ravel())\r\n<\/pre>\n# View the Results<\/span>\r\n# Selected features are assigned rank 1<\/span>\r\nranks =<\/span> rfe.<\/span>ranking_\r\nfeatures =<\/span> wine_df.<\/span>iloc[0<\/span>:,1<\/span>:].<\/span>columns\r\n\r\n# Display feature with rank 1<\/span>\r\nfor<\/span> a, b in<\/span> zip<\/span>(ranks, features):\r\n if<\/span> a ==<\/span> 1<\/span>:\r\n print<\/span>(f'{a}: {b}'<\/span>)\r\n<\/pre>\n4. Dimensionality Reduction<\/strong><\/h4>\n
\n
5. Feature Importance<\/strong><\/h4>\n
\n
# Import the module<\/span>\r\nimport<\/span> sklearn.ensemble<\/span> as<\/span> se<\/span>\r\n<\/pre>\n# Create and fit and display<\/span>\r\nmodel =<\/span> se.<\/span>ExtraTreesClassifier()\r\nmodel.<\/span>fit(wineX, wineY.<\/span>ravel())\r\n<\/pre>\nimportances =<\/span> model.<\/span>feature_importances_\r\nfeatures =<\/span> wine_df.<\/span>iloc[0<\/span>:1<\/span>,1<\/span>:].<\/span>columns\r\nfor<\/span> a, b in<\/span> zip<\/span>(features, importances):\r\n print<\/span>(f'{a} : {b}'<\/span>)\r\n<\/pre>\n