Online Purchase Intent

Online Purchase Intent

In this blog we will explore a classification problem of judging whether an online buyer will end up buying an item based on his online activity. The dataset can be found here. There are 17 features of which 10 are continuous and 7 categorical. There is one target variable with binary classification of True or False depending on whether any revenue was generated.

Exploring the Dataset

Let us begin by exploring the data set. We have the following features and their description.
# Administrative Administrative Value
# Administrative_Duration Duration in Administrative Page
# Informational Informational Value
#Informational_Duration Duration in Informational Page
#Product_related Product Related Value
#ProductRelated_Duration Duration in Product Related Page
#BounceRates Bounce Rates of a web page
#ExitRates Exit rate of a web page
#PageValues Page values of each web page
#SpecialDay Special days like valentine etc
#Month Month of the year
#OperatingSystem Operating system used
#Browser Browser used
#Region Region of the user
#TrafficType Traffic Type
#VisitorType Types of Visitor
#Weekend Weekend or not


For the purposes of visualization and for Naive Bayesian Analysis I consider features with < 30 unique entries as categorical variables and the remaining as continuous variables. I also log transform the categorical variables to make them less skewed. Below I have plotted the distributions of the continuous variables. As can be seen there can be significant differences in the distribution of variables. 
Probability distributions for continuous feature. The continuous variables have been log transformed
We can also visualize the probability distributions of the target variable as categorical variables by plotting the relative fractions in a bar plot. We have done this below

Target variable as a function of the categorical variables
We also need to keep in mind that the target variable has class imbalance i.e ~ 85% of the values are False and only 15% are true. So even if we predict all false entries we will get an accuracy score of ~ 85%. Any model we build will need to improve on this accuracy score. We will begin by doing a naive Bayesian analysis.

Naive Bayesian analysis

Naive bayesian analysis assumes that all the variables are independent of each other. In Naive bayesian analysis we can write



In order to do a Naive Bayesian analysis we need to find out the Likelihoods i.e. P(xi|True,False). We treat continuous and categorical variables separately. To find the Likelihood of the continuous variables we bin the data for each target value and count the entries per bin which gives us our Likelihood probabilities.  For categorical  variables we bin it according to the number of bins in any given category. The priors P(True) and P(False) are about 0.15 and 0.85 respectively since that's the probability of the target variables in the underlying dataset. We split the data into train and test. We then find whether a given test data print has a larger probability to be True or False to classify it.

 We first train the naive Bayes classifier on the continuous features. Below we have plotted the precision, recall, accuracy, and F1 score for each individual feature. As we can see the Page Values is the best predictor whereas the others don't seem to affect the predictions individually. Now we combine all the continuous features into a single Bayesian classifier. As can be seen the predictability of the model in fact goes down. This is most likely due to the fact that our iid assumption in not correct. Training on categorical variables we find that none of the categorical variables are predictable by themselves, so we ignore them. Since Page Values is a better predictor than the combined we use Naive Bayesian on Page Values for our baseline accuracy score. Any model we use should do better than that.
Scores for Naive Bayes with continuous features. As we can see the Page Values classifies better than combined features
Scores for Naive Bayes with categorical features. As we can see the categorical variables cannot classify well

One would assume that if we do PCA on our continuous features and rerun Naive Bayesian Analysis we would get better results because our features are now iid. However that does not turn out to be the case. In fact the predictions are no better predicting all False.

Logistic Regression

One of the simplest classifier models is Logistic Regression. For Logistic Regression we consider the last 7 features as categoorical. One hot encoding them with get_dummies we get a total of 73 features.  We also add another boolean feature which is true when Administrative_Duration = -1. 

 We run a GridSearchCV on Logistic Regression with the following parameters
['fit_intercept': [True,False], 'solver':['liblinear'], 'C': np.logspace(0,4,5), 'penalty': ['l2'],
    'class_weight': [{0:1,1:1},{0:1.2,1:1},{0:3,1:1},{0:6,1:1}]]

 The best parameters are 
Optimal Parameters: {'C': 1000.0, 'class_weight': {0: 3, 1: 1}, 'fit_intercept': False, 'penalty': 'l2', 'solver': 'liblinear'}

 Below we have plotted the ROC curve. The test and train AUC are extremely close to each other indicating that we are not overfitting. The accuracy we get with Logistic Regression is 0.897

Random Forests

The second classifier we use is Random Forests. A Random Forest is an ensemble method for a Decision Tree .  We also use a GridSearchCV for our Random Forest Classifier.  The parameters that we used for GridSearchCV are
{'n_estimators': [10, 30, 100,300], 'criterion': ['gini', 'entropy'], 'max_depth': [None, 2, 6, 10],
'min_samples_split': [5, 10,15], 'min_samples_leaf': [1,3, 6]}

 The optimal parameters of our search are
Optimal Parameters: {'criterion': 'entropy', 'max_depth': None, 'min_samples_leaf': 1, 'min_samples_split': 15, 'n_estimators': 300}

with an accuracy of 91.4% .

We have plotted below the top 10 important features in our random forest classification. As expected the PageValues is the most important feature.

Stacking and Conclusion

We tried 3 different classification methods and our Random Forest Algorithm achieved a maximum accuracy of 91.4% . We can not stack all the three predictors together and see if we get a better result. The idea is that each of the predictor fits better to different features of the data. We stack our predictors by giving one vote to each predictor. As can be seen from the summary of the final results below the stacked predictor does not do better than our best predictor, the Random Forest.


Comments

Post a Comment

Popular posts from this blog

ETF Predictions

Image
Introduction An exchange traded fund ( ETF ) is a fund that owns multiple underlying securities. A popular example is the SPY ETF which tracks the S&P 500 index  which itself tracks the performance of the 500 largest companies in the United States. In the long term the stock market always goes up and investing in ETF's such as SPY is a lucrative passive investment that leverages this fact. Diversification is an important tool for an investor to minimize risks of owning individual stocks and ETFs accomplish this by owning multiple underlying securities. If fact in the 30 year period, ETF's such as SPY and VTSAX  outperform 99% of portfolio managers  . Thus predicting how ETF's will change over a given time horizon will improve our returns, possibly far more than the average 6%-10% annual return generated by the popular ETFs. However that's easier said than done. The efficient market hypothesis ( EMH ) says that it is impossible to consistently generate profits ( Fam...
Read more

MNIST Digit Recognition

Image
MNIST Digit Recognition without Neural Networks  Introduction The MNIST dataset is a set of 70,000 handwritten digits. Of these 42,000 are used for training while 28,000 are used for testing our trained models. The dataset can be found here on Kaggle. The MNIST is an excellent standard benchmark dataset and can be used an a playground to to test and learn more about standard classification algorithms. Using Neural Networks it's reasonably easy to achieve an accuracy greater than 98% as is evidenced by the submitted notebooks on Kaggle. In this blog we will try to see what's the best accuracy we can get without using deep neural networks. We will use Random Forest, Gradient Boost, SVM, stacking and combinations of the above to perform our analysis. We will try different tricks and feature engineering, not all of which will give us better accuracy compared to out of box classification methods . However since the goal of this blog is to explore and learn we will report  The anal...
Read more

Zillow Data Time Series Analysis

Image
Zillow Data Time Series Analysis In this bog post we will deal with the time series analysis of of the Zillow data set which details the average house prices in every zipcode between April 1996 to April 2018. Our analysis can be found in this github repo . Of course not all zipcodes have data starting from  April 1996. There are overall 14723 Zipcodes which is a lot of data.  The question that we address in this project is the following. Imagine a real estate company has contracted us to analyze the data. The question that they want answered is What are the 5 best zipcodes to invest in? Of course the important question in what does one mean by best . That is of course why we are being paid the big bucks. We will have several definitions on best and analyze the data accordingly. Let us begin by describing the data set and gain some intuition from it. The dataset looks like the following, when stored in the 'wide' format with one column for the average price for ea...
Read more