This is Class three of our practical Science Course for Data Science Beginners. In this class would be perform data preprocessing and data cleaning (or data cleansing). We would also discuss some of the theoretical concepts.<\/p>\n
We would be using the Titanic Dataset. Get the Titanic Dataset here for free<\/a>.<\/p>\n The following are covered:<\/p>\n Class 3 Video on Preprocessing<\/a><\/p>\n After obtaining your dataset and doing basic visualization, the next step is to perform preprocessing on your dataset. Data preprocessing refers to the operations you perform on your data to ensure it works well with Machine Learning algorithms. Data preprocessing also ensure better performance on analytics process. It includes data cleaning, outlier detection, data wrangling, normalization, data editing, unreliable data removal, data conversion etc.<\/p>\n In this class we would perform most of them on the Titanic Dataset.<\/p>\n <\/p>\n Data scaling is a technique that ensures that the attributes of the dataset are on the same scale. Most times, we need to rescale to a scale of 0 to 1 as required by Machine Learning algorithms like k-Nearest Neighbor and Gradient Descent.<\/p>\n Python provides a library called the MinMaxScalar for performing scaling. This library is available in sklearn module<\/p>\n Take the four steps below to scale the data in the fare column of the Titanic dataset<\/p>\n Step 1<\/strong> – Create the MinMaxScaler object<\/p>\n <\/p>\n <\/p>\n Step 2<\/strong> – Extract the fare column <\/p>\n Step 3<\/strong> – Perform the scaling <\/p>\n Step 4<\/strong> – Replace the original column<\/p>\n <\/p>\n Dropping and interpolating are data cleansing technique used to handle missing values in a dataset. We can decided to drop a column if it does not contribute anything to the data analysis process. For example the name and the ticket columns.<\/p>\n Drop Columns with Missing Values<\/strong><\/p>\n Another reason we may drop a column is when there are multiple missing values. An example is the body, boat and cabin columns of the Titanic dataset.<\/p>\n To drop these columns, use the code below:<\/p>\n The axis = 1 indicates we are dropping columns<\/p>\n Interpolating Missing Values<\/strong><\/p>\n If you have a column with very few missing values, you can just choose to interpolate them using existing values. Interpolation is simply a way to create new data based on existing data.\u00a0 For example if you have a range 2, 4, ?, 8, 10. Then here, by interpolation, the missing value will be 6 by interpolation. That is (4+8)\/2.<\/p>\n Let’s interpolate the age column of the Titanic dataset using the code below<\/p>\n <\/p>\n Drop rows with missing Values<\/strong><\/p>\n To drop all rows with missing values, we can use the code below. Here, we don’t specify the axis.<\/p>\n <\/p>\n Normalization is used when certain features have broad range of values. For example some feature have values of 0 or close to zero while some other feature have very high values of say, in 100s or 1000s.\u00a0 In this case normalization would scale each recored to have a range length of say, 1.<\/p>\n There are two types of normalization: L1 Normalization and L2 Normalization<\/p>\n L1 Normalization<\/strong> – Also known as Manhattan normalization. Here, for each row of the dataset, the sum of the absolution values will always equal 1<\/p>\n L2 Normalization<\/strong> – Also known as Euclidean normalization. Here, for each row of data, the root of the sum of the square of the values will always equal 1.<\/p>\n To perform normalization we simply create a normalizer object and proceed similar to how we performed scaling. Code snippet is given below. See video for full explanation<\/p>\n <\/p>\n Exercise<\/strong>: Perform L2 normalization on the Ash column of the wine dataset. (try it, then see video for procedure and explanation)<\/p>\n The remaining 5 points are covered in the next Class Part 2<\/a><\/strong><\/p>\n\n
1. What is Data Preprocessing?<\/strong><\/h4>\n
2. Data Scaling or Rescaling<\/strong><\/h4>\n
# Create a MinMaxScaler object<\/span>\r\ndata_scaler =<\/span> pp.<\/span>MinMaxScaler(feature_range=<\/span>(0<\/span>,1<\/span>))\r\n<\/pre>\n
\n<\/p>\n# Extract the fare column<\/span>\r\nfare_array =<\/span> titanic_df[['fare'<\/span>]]\r\n<\/pre>\n
\n<\/p>\n# Perform the scaling of the extracted column<\/span>\r\nfare_array_scaled =<\/span> data_scaler.<\/span>fit_transform(fare_array)\r\n<\/pre>\n# Now replace the original column with the scaled column<\/span>\r\ntitanic_df['fare'<\/span>] =<\/span> fare_array_scaled\r\n<\/pre>\n3. Dropping and Interpolating Missing Data<\/strong><\/h4>\n
# Drop Columns<\/span>\r\ncols_to_drop =<\/span> ['body'<\/span>, 'boat'<\/span>, 'name'<\/span>, 'ticket'<\/span>, 'cabin'<\/span>]\r\ntitanic_df =<\/span> titanic_df.<\/span>drop(cols_to_drop, axis=<\/span>1<\/span>)\r\n<\/pre>\n# To replace missing values with interpolated values, for example Age<\/span>\r\ndf['Age'<\/span>] =<\/span> df['Age'<\/span>].<\/span>interpolate()\r\n<\/pre>\n# Drop all rows with missin data<\/span>\r\ndf =<\/span> df.<\/span>dropna()\r\n<\/pre>\n4. Data Normalisation<\/strong><\/h4>\n
# Perform Normaliztion on the parch column<\/span>\r\nnormalizer =<\/span> pp.<\/span>Normalizer(norm=<\/span>'l1'<\/span>) # use l2 for L2 Normalization<\/span>\r\nparch_array =<\/span> titanic_df[['parch'<\/span>]]\r\nparch_array_normalized =<\/span> normalizer.<\/span>transform(parch_array)\r\ntitanic_df['parch'<\/span>] =<\/span> parch_array_normalized\r\n<\/pre>\n