Notebook 1

Exercise 1¶

Exploring Class Separability in 2D¶

Understanding how data is distributed is the first step before designing a network architecture. In this exercise, you will generate and visualize a two-dimensional dataset to explore how data distribution affects the complexity of the decision boundaries a neural network would need to learn.

Instructions¶

1. Generate the Data: Create a synthetic dataset with a total of 400 samples, divided equally among 4 classes (100 samples each). Use a Gaussian distribution to generate the points for each class based on the following parameters:
   • Class 0: Mean = [2, 3], Standard Deviation = [0.8, 2.5]
   • Class 1: Mean = [5, 6], Standard Deviation = [1.2, 1.9]
   • Class 2: Mean = [8, 1], Standard Deviation = [0.9, 0.9]
   • Class 3: Mean = [15, 4], Standard Deviation = [0.5, 2.0]

2. Plot the Data: Create a 2D scatter plot showing all the data points. Use a different color for each class to make them distinguishable.

3. Analyze and Draw Boundaries:

   1. Examine the scatter plot carefully. Describe the distribution and overlap of the four classes.
      • Class 0 and 1 have the most overlap, being pretty ditinguishable from the other two classes (numbers 2 and 3). The most segregated one is class 3, being at the far end of the X1 axis of the plot.
   2. Based on your visual inspection, could a simple, linear boundary separate all classes?
      • I would argue that a line can be made to separate classes from each other, but the line would also put different classes on the same side. This would mean we would need at minimum a second line to properly separate all classes.
   3. On your plot, sketch the decision boundaries that you think a trained neural network might learn to separate these classes.

   

Exercise 2¶

Non-Linearity in Higher Dimensions¶

Simple neural networks (like a Perceptron) can only learn linear boundaries. Deep networks excel when data is not linearly separable. This exercise challenges you to create and visualize such a dataset.

Instructions¶

1. Generate the Data: Create a dataset with 500 samples for Class A and 500 samples for Class B. Use a multivariate normal distribution with the following parameters:

   • Class A:

     Mean vector:

     $$\mu_A = [0, 0, 0, 0, 0]$$

     Covariance matrix:

     $$ \Sigma_A = \begin{pmatrix} 1.0 & 0.8 & 0.1 & 0.0 & 0.0 \\ 0.8 & 1.0 & 0.3 & 0.0 & 0.0 \\ 0.1 & 0.3 & 1.0 & 0.5 & 0.0 \\ 0.0 & 0.0 & 0.5 & 1.0 & 0.2 \\ 0.0 & 0.0 & 0.0 & 0.2 & 1.0 \end{pmatrix} $$

   • Class B:

     Mean vector:

     $$\mu_B = [1.5, 1.5, 1.5, 1.5, 1.5]$$

     Covariance matrix:

     $$ \Sigma_B = \begin{pmatrix} 1.5 & -0.7 & 0.2 & 0.0 & 0.0 \\ -0.7 & 1.5 & 0.4 & 0.0 & 0.0 \\ 0.2 & 0.4 & 1.5 & 0.6 & 0.0 \\ 0.0 & 0.0 & 0.6 & 1.5 & 0.3 \\ 0.0 & 0.0 & 0.0 & 0.3 & 1.5 \end{pmatrix} $$

1. Visualize the Data: Since you cannot directly plot a 5D graph, you must reduce its dimensionality.
   • Use a technique like Principal Component Analysis (PCA) to project the 5D data down to 2 dimensions.
   • Create a scatter plot of this 2D representation, coloring the points by their class (A or B).
2. Analyze the Plots:

   1. Based on your 2D projection, describe the relationship between the two classes.

      • Very intertwined, making it extremely difficult to separate them clearly. They do tend to different sides of the plot (in the x-axis), but they do have noticeable overlap.

   2. Discuss the linear separability of the data. Explain why this type of data structure poses a challenge for simple linear models and would likely require a multi-layer neural network with non-linear activation functions to be classified accurately.

      • Creating a single line that segregates these two classes is not possible, given that they do have overlap. If you were to trace a line between them, it would end up inevitably classifying class A as class B and vice-versa.

Exercise 3¶

Preparing Real-World Data for a Neural Network¶

This exercise uses a real dataset from Kaggle. Your task is to perform the necessary preprocessing to make it suitable for a neural network that uses the hyperbolic tangent (tanh) activation function in its hidden layers.

Instructions¶

1. Get the Data: Download the Spaceship Titanic dataset from Kaggle.

2. Describe the Data:

   • Briefly describe the dataset's objective (i.e., what does the Transported column represent?).

     • We're trying to predict if a passenger was transported to another dimension during the crash of the Spaceship Titanic, the value that represents whether or not the passenger was transported is in the Transported column

   • List the features and identify which are numerical (e.g., Age, RoomService) and which are categorical (e.g., HomePlanet, Destination).

   • Investigate the dataset for missing values. Which columns have them, and how many?

Categorical Columns -

• HomePlanet - The planet the passenger departed from, typically their planet of permanent residence.
• Cabin - The cabin number where the passenger is staying. Takes the form deck/num/side, where side can be either P for Port or S for Starboard.
• Destination - The planet the passenger will be debarking to.
• Name - The first and last names of the passenger.
• PassengerId - A unique Id for each passenger. Each Id takes the form gggg_pp where gggg indicates a group the passenger is travelling with and pp is their number within the group. People in a group are often family members, but not always.

Boolean Columns -

• Transported - Whether the passenger was transported to another dimension. This is the target, the column you are trying to predict.
• VIP - Whether the passenger has paid for special VIP service during the voyage.
• CryoSleep - Indicates whether the passenger elected to be put into suspended animation for the duration of the voyage. Passengers in cryosleep are confined to their cabins.

Numerical Columns -

• Age - The age of the passenger.
• RoomService, FoodCourt, ShoppingMall, Spa, VRDeck - Amount the passenger has billed at each of the Spaceship Titanic's many luxury amenities.

For these missing values, we must treat numerical and categorical features separately.

For numerical features, we use a Simple Imputer to fill null values. In this instance, I chose the Median value to be used as filler. After that, we use a Standard Scaler to put all values centered at 0.

For categorical features, we use the most frequent class as a fill-in for missing values, and pass it to a OneHotEncoder afterwards so our categorical values become boolean features. In this case, this gave us 26 features compared to our initial 14

3. Preprocess the Data: Your goal is to clean and transform the data so it can be fed into a neural network. The tanh activation function produces outputs in the range [-1, 1], so your input data should be scaled appropriately for stable training.
   • Handle Missing Data: Devise and implement a strategy to handle the missing values in all the affected columns. Justify your choices.
   • Encode Categorical Features: Convert categorical columns like HomePlanet, CryoSleep, and Destination into a numerical format. One-hot encoding is a good choice.
   • Normalize/Standardize Numerical Features: Scale the numerical columns (e.g., Age, RoomService, etc.). Since the tanh activation function is centered at zero and outputs values in [-1, 1], Standardization (to mean 0, std 1) or Normalization to a [-1, 1] range are excellent choices. Implement one and explain why it is a good practice for training neural networks with this activation function.
4. Visualize the Results:
   • Create histograms for one or two numerical features (like FoodCourt or Age) before and after scaling to show the effect of your transformation.

Notice that the shape of the graphs is maintained, but the values in the x-axis are centered to 0.