It’s all about Assumptions, Pros & Cons

This blog is all about the assumptions made by the popular ML Algorithms, their pros & cons.

source: https://media.geeksforgeeks.org/wp-content/cdn-uploads/machineLearning3.png

Getting started with this blog let me first tell the reason for coming up with this blog.

There are tons of Data Science enthusiasts who are either looking for the job in Data Science or switching jobs for better opportunities. Given that every individual has to go through some strict hiring process containing several rounds of interview.

And there are several basic questions which a recruiter/interviewer expect to be answered by us. And knowing about the assumptions of the popular Machine Learning Algorithms along with their Pros & Cons is one of them.

Going ahead in the blog, I will first introduce you with the assumptions made by that specific algorithm followed by its pros & cons. So it's just gonna take 5 minutes from your precious time, and at the end of the blog, you would surely get something to learn.

I am gonna follow the below sequence to introduce assumption, pros & cons.

1. K-NN(K-Nearest Neighbours)
2. Logistic Regression
3. Linear Regression
4. Support Vector Machines
5. Decision Trees
6. Naive Bayes
7. Random Forest(Bagging Algorithm)
8. XGBoost(Boosting Algorithm)

1. K-NN

source: https://www.analyticsvidhya.com/wp-content/uploads/2014/10/scenario2.png

Assumptions:

1. The data is in feature space, which means data in feature space can be measured by distance metrics such as Manhattan, Euclidean etc.
2. Each of the training data points consists of a set of vectors and class label associated with each vector.
3. Wishes to have ‘K’ as an odd number in case of 2 class classification.

Pros:

1. Easy to understand, implement and explain.
2. Is a non-parametric algorithm so don’t have strict assumptions.
3. No training steps required. It uses training data at run time to make predictions making it faster than all those algorithms which need to be trained.
4. Since it doesn’t need training on train data so data points can be easily added.

Cons:

1. Inefficient and slow when the dataset is large. As the cost of the calculating, the distance between the new point and train points is large.
2. Don’t work well in with high dimensional data. As it becomes harder to find the distance in higher dimensions.
3. Sensitive to outliers. Get easily affected by outliers.
4. Can’t work when data is missing. So data needs to be manually imputed to make it work.
5. Needs feature scaling/normalization.

2. Logistic Regression

source: https://miro.medium.com/max/2400/1*RqXFpiNGwdiKBWyLJc_E7g.png

Assumptions:

1. It assumes that there is minimal or no multicollinearity among the independent variables.
2. It usually requires a large sample size to predict properly.
3. It assumes the observations to be independent of each other.

Pros:

1. Easy to interpret, implement and train. Doesn’t require too much computational power.
2. Makes no assumption of Class-Distribution.
3. Fast in classifying unknown records.
4. Can easily accommodate new data points.
5. Is very efficient when features are linearly separable.

Cons:

1. Tries to predict precise probabilistic outcomes. Which leads to overfitting in high dimensions.
2. Since has linear decision surface. So, can’t solve non-linear problems.
3. Tough to obtain complex relations other than linear relations.
4. Requires very less or no multicollinearity.
5. Need large dataset and also sufficient training examples for all the categories to make correct predictions.

3. Linear Regression

source: https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Linear_regression.svg/1200px-Linear_regression.svg.png

Assumptions:

1. There should be a linear relationship.
2. There should be no or less multicollinearity.
3. Homoscedasticity: The variance of residual should be the same for any value of X.

Pros:

1. Performs very well when there is a linear relationship between the independent and dependent variable.
2. If overfits, overfitting can be reduced easily by L1 or L2 Norms.

Cons:

1. Its assumption of data independence.
2. Assumption of linear separability.
3. Sensitive to outliers.

4. Support Vector Machines

source: https://www.researchgate.net/publication/304611323/figure/fig8/AS:668377215406089@1536364954428/Classification-of-data-by-support-vector-machine-SVM.png

Assumptions:

1. It assumes data is independent and identically distributed.

Pros:

1. Works really well on high dimensional data.
2. Memory efficient.
3. Effective in cases where the number of dimensions is greater than the number of samples.

Cons:

1. Not suitable for large datasets.
2. Don’t work well when the dataset has noise i.e target classes are overlapping.
3. Slow to train.
4. No probabilistic explanation for classification.

5. Decision Trees

source: https://miro.medium.com/max/3840/1*jojTznh4HOX_8cGw_04ODA.png

Assumptions:

1. Initially, whole training data is considered as root.
2. Records are distributed recursively on the basis of the attribute value.

Pros:

1. Compared to other algorithms data preparation requires less time.
2. Don’t require data to be normalized.
3. Missing values till and extent don’t affect its performs much.
4. Is very intuitive as can be explained as if-else conditions.

Cons:

1. Need high time to train the model.
2. A small change in data can cause a considerably large change in the Decision Tree structure.
3. Comparatively expensive to train.
4. Not good for regression tasks.

6. Naive Bayes

source: https://www.kdnuggets.com/wp-content/uploads/bayes-nagesh-0.jpg

Assumptions:

1. The biggest and only assumption is the assumption of conditional independence.

Pros:

1. Gives high performance when the conditional independence assumption is satisfied.
2. Easy to implement as only probabilities need to be calculated.
3. Works well with high dimensional data. Such as text.
4. Fast in real-time predictions.

Cons:

1. If conditional independence does not hold perform poorly.
2. The problem of Numerical Stability or Numerical Underflow because of the multiplication of several small digits.

7. Random Forest

source: https://www.researchgate.net/publication/316982197/figure/fig5/AS:559887665303554@1510499029585/The-structure-of-random-forest-algorithm-The-random-forest-is-composed-of-the-generated.png

Assumptions:

1. Assumption of no formal distributions. Being a non-parametric model can handle skewed and multi-modal data.

Pros:

1. Robust to outliers.
2. Works well for non-linear data.
3. Low risk of overfitting.
4. Runs efficiently on large datasets.

Cons:

1. Slow training.
2. Biased when dealing with categorical variables.

8. XGBoost

source: https://miro.medium.com/max/4000/1*IWBGb4PC7F2q0fszK-Iplw.png

Assumptions:

1. It may have an assumption that encoded integer value for each variable has ordinal relation.

Pros:

1. Can work parallelly.
2. Can handle missing values.
3. No need for scaling or normalizing data.
4. Fast to interpret.
5. Great execution speed.

Cons:

1. Can easily overfit if parameters are not tuned properly.
2. Hard to tune.

These are the pros, cons & assumptions of all the above Machine Learning Algorithm. You can always find a better explanation of things on GeeksforGeeks, Stackexchange, Quora, Stackoverflow etc.

But my aim to write this blog is to help anyone preparing for an interview to get everything in one place as some quick notes.

I hope you like my work. If you find any problems in the above details please comment it out so that I can improve them. Thanks