{"id":253,"date":"2021-09-19T18:06:37","date_gmt":"2021-09-19T18:06:37","guid":{"rendered":"https:\/\/www.kindsonthegenius.com\/data-science\/?p=253"},"modified":"2021-09-25T13:49:56","modified_gmt":"2021-09-25T13:49:56","slug":"class-4-introduction-to-data-preprocessing-and-data-cleaning-part-2","status":"publish","type":"post","link":"https:\/\/www.kindsonthegenius.com\/data-science\/class-4-introduction-to-data-preprocessing-and-data-cleaning-part-2\/","title":{"rendered":"Class 4 – Introduction to Data Preprocessing and Data Cleaning – Part 2"},"content":{"rendered":"

This would be Class 4 of\u00a0 of our Data Science Series and in this class we would complete the remaining topics on Data Preprocessing and Data Cleaning. You can get Part 1 here<\/a>.<\/p>\n

 <\/p>\n

The following are covered here<\/p>\n

    \n
  1. Numerical and Categorical Values Conversion<\/a><\/li>\n
  2. Data Binarization<\/a><\/li>\n
  3. Data Standardization<\/a><\/li>\n
  4. Data Labelling and Encoding<\/a><\/li>\n
  5. Data Splitting \u2013 Feature and Class; Train & Test<\/a><\/li>\n<\/ol>\n

    Preprocessing Video Part 2<\/a><\/p>\n

    1. Numerical and Categorical Values Conversion<\/strong><\/h4>\n

    When some values are given as strings or characters, we may have to convert them to numeric values. For example,\u00a0 if you load up the dataset and take a look at the Sex column, you see that it has the values ‘Male’ and ‘Female’. We would need to change this to values 0 and 1.<\/p>\n

    In this case, we should have Male = 1 and Female 0.<\/p>\n

    The format to do this conversion is<\/p>\n

    df.<\/span>loc[df['colname'<\/span>] ==<\/span> <<\/span>value><\/span>, 'colname'<\/span>] =<\/span> <<\/span>new_value><\/span>\r\n<\/pre>\n
    The code above will replace the <value> in the column ‘colname’ with <new_value><\/p>\n
    Therefore the code below would convert the ‘Male’ and ‘Female’ values in out Titanic data<\/p>\n
    titanic_df.<\/span>loc[titanic_df['sex'<\/span>]==<\/span>'male'<\/span>, 'sex'<\/span>] =<\/span> 1<\/span>\r\ntitanic_df.<\/span>loc[titanic_df['sex'<\/span>]==<\/span>'female'<\/span>, 'sex'<\/span>] =<\/span> 0<\/span>\r\n<\/pre>\n
     <\/p>\n
    Add or Remove Column<\/strong><\/p>\n
    Us the code below to add a column and remove a column<\/p>\n
    df['new_col'<\/span>] =<\/span> None<\/span> # Add a column<\/span>\r\ndf =<\/span> df.<\/span>drop(columns =<\/span> 'new_col'<\/span>, axis =<\/span> 1<\/span>) # Drop a column<\/span>\r\n<\/pre>\n
     <\/p>\n
    Exercise<\/strong> – Replace the values in the Embarked column with values 1, 2 3<\/p>\n
     <\/p>\n
    2. Data Binarization or Thresholding<\/strong><\/h4>\n
    Just as the name indicates, binarization is a preprocessing technique used to change values in a dataset to binary (0 and 1). This may be achieved by setting a threshold. The values below the threshold are set to 0 while the values below the threshold are set to 1.<\/p>\n
    Binarization may be needed when logistic regression would have to be performed<\/p>\n
    For example, in our Titanic dataset, we can binarize the fare field following the steps below:<\/p>\n
    I have added it as a screen shot for clarity:<\/p>\n
    \"Binarization<\/a>
    Binarization on the fare column of the Titanic Dataset<\/figcaption><\/figure>\n
     <\/p>\n
    3. Data Standardization<\/strong><\/h4>\n
    This is also called ‘mean removal and variance scaling’ is a technique used to transform the dataset so that the feature look more like a standard normally distributed data. That is data with a mean of 0 and standard deviation of 1.<\/p>\n
    Standardization is used in Machine learning estimators such as linear regression and logistic regression where better results are achieved with a normally distributed data. In Python we use the StandardScaler from sklearn.<\/p>\n
    This work the same as MinMaxScaler from Part 1.<\/p>\n
    Exercise<\/strong> – Import the Wine dataset. Perform Standardization on the Proline column of the wine dataset.<\/p>\n
     <\/p>\n
    4. Data Labelling and Encoding<\/strong><\/h4>\n
    This is an automated way of performing Numerical and Categorical values conversion<\/a> which we covered in section 1<\/a> above.\u00a0 This means that instead of using string labels for the data (like ‘Male’ and ‘Female’), we encode the values into numeric values or number labels. So if theres are n labels in the dataset, they would be encoded in 0 to n-1. The drawback of this method is that you don’t have control over the labels that are assigned.<\/p>\n
    Sklearn provides a module LabelEncoder() which we can use to perform data labelling.<\/p>\n
    The code below extracts the home.dest column, encodes it and adds it as a new column in the dataset. See the video for more explanation<\/p>\n
    encoder =<\/span> pp.<\/span>LabelEncoder()\r\nhome_array =<\/span> titanic_df[['home.dest'<\/span>]].<\/span>values\r\nencoder.<\/span>fit(home_array.<\/span>ravel()) # ravel() converts 1d vector to array<\/span>\r\nhome_array_encoded =<\/span> encoder.<\/span>transform(home_array)\r\ntitanic_df['home.dest.encoded'<\/span>] =<\/span> home_array_encoded\r\n<\/pre>\n
     <\/p>\n
    5. Data Splitting \u2013 Features and Class; Train & Test<\/strong><\/h4>\n
    This is use mostly when we apply supervised learning algorithms to our dataset. Supervised learning is just a ‘fancy word’ for classification and regression!<\/p>\n
    Splitting into Features and Class<\/strong><\/p>\n
    Dataset used for classification is normally has two parts: the features (X) and the class (Y). The idea is that the class can be deduced based on the feature. In the case of the Titanic Dataset,\u00a0 we would like to determine who survived. Therefore, the survived column is the class while every other columns are the features. The code below would split the data in X (features) and Y(class).<\/p>\n
    Y =<\/span> titanic_df[['survived'<\/span>]]\r\nX =<\/span> titanic_df.<\/span>drop(columns =<\/span> 'survived'<\/span>, axis=<\/span>1<\/span>)\r\n<\/pre>\n
     <\/p>\n
    Splitting into Train and Test<\/strong><\/p>\n
    In supervised learning, you will have to split your data into two parts: training dataset and test dataset. The training dataset is used to train the model while the test dataset is used to test the performance of the model on new data.<\/p>\n
    In the code below, I generate a dataset of 1000 records, then\u00a0 perform train-test split to split the data into 70% training set and 30% test dataset.<\/p>\n
    # split a dataset into train and test sets<\/span>\r\nfrom<\/span> sklearn.datasets<\/span> import<\/span> make_blobs\r\nfrom<\/span> sklearn.model_selection<\/span> import<\/span> train_test_split\r\n# Generate dataset<\/span>\r\nX, y =<\/span> make_blobs(n_samples=<\/span>1000<\/span>)\r\n# split into train and test sets<\/span>\r\nX_train, X_test, y_train, y_test =<\/span> train_test_split(X, y, test_size=<\/span>0.30<\/span>)\r\nprint<\/span>(X_train.<\/span>shape, X_test.<\/span>shape, y_train.<\/span>shape, y_test.<\/span>shape)\r\n<\/pre>\n
     <\/p>\n","protected":false},"excerpt":{"rendered":"
    This would be Class 4 of\u00a0 of our Data Science Series and in this class we would complete the remaining topics on Data Preprocessing and … <\/p>\n","protected":false},"author":1,"featured_media":258,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44,43],"tags":[58,60,59,56],"class_list":["post-253","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science","category-python","tag-binarization","tag-data-labelling","tag-label-encoding","tag-preprocessing"],"jetpack_featured_media_url":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-content\/uploads\/sites\/12\/2021\/09\/A-second-class-in-preprocessing.jpg","_links":{"self":[{"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/posts\/253","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/comments?post=253"}],"version-history":[{"count":5,"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/posts\/253\/revisions"}],"predecessor-version":[{"id":280,"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/posts\/253\/revisions\/280"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/media\/258"}],"wp:attachment":[{"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/media?parent=253"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/categories?post=253"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kindsonthegenius.com\/data-science\/wp-json\/wp\/v2\/tags?post=253"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}