SVM
Support Vector Machine Tutorial
1- Overview
Support Vector Machine is a highly customizable machine learning algorithm that works well with kernels. It can do both classification and regression fairly well. Due to its customizable nature and lots of parameters to optimize there was once a strong hype about the use of SVMs.
Although still utilized commonly Support Vector Machines face serious competition.
SVMs can be used with several kernels:
- linear
- rfb
Why Support Vector Machine Algorithm?
2- Support Vector Machine Benefits
- Customizable.
- Tackles non-linear data very well.
- Plenty of settings to adapt to different problems that may be unapproachable by other algorithms.
- Capable of regression as well as classification (even clustering in some applications)
- Relatively easy pre-processing procedures.
- Can handle missing data
- Doesn’t require normalization
SVM Pros
- Easy data pre-processing
- Lots of custom options
- Very versatile
SVM Cons
- Difficult to optimize
- Doesn’t always scale well
Application Areas
3- Key Industries
Support Vector Machines have been utilized in many key industries ranging from: Engineering, Material Sciences, Scientific Research, Medical Research, Business Administration, Finance, Marketing, Technology and many others.
Like many other Machine Learning Algorithms, SVMs have tremendous use cases in the field and their utilization is extremely on the rise. Many experts in Machine Learning suggest combining Machine Learning technology with domain knowledge to come up with useful solutions and insights.
Luckily there are almost no limits to the application field of Machine Learning Algorithms. Support Vector Machine is no exception. Support Vector Machine thrives wherever data exists. Vaguely speaking this can be in the form of text, image, video, facial expression, signal, audio, geo-location coordinates, satellite imagery, medical screening, X-rays, surveys, CT scans, road signals, cloud shapes, solar values, wind forecast, customer use, transaction data, financial reports, asset prices etc. etc.
So many different type of data are being used as input in Machine Learning combined with Support Vector Machine and/or other models to gain competitive edge across many fields and disciplines.
- Finance
- Medicine
- Cyber Security
- Retail
- E-commerce
Who Found Support Vector Machine?
4- Support Vector Machine History
Support Vector Machine Algorithm was introduced in 1992 in a paper published by Vladimir N. Vapnik, Bernhard E Boser and Isabelle M Guyon named “A training algorithm for optimal margin classifiers“.
After its initial discovery SVM has seen incredible research popularity and it has been advanced and extended by numerous research work especially in late 90s and early 2000s.
For a more detailed history of Support Vector Machines you can read the article below:
Are Support Vector Machines Fast?
5- Support Vector Machine Complexity
Support Vector Machine is not known to be a very fast machine learning algorithm and it doesn’t scale very well. That being said there are number of options that can be tried for faster implementations such as Linear SVM models.
If your problem is linear then you can use LinearSVC or LinearSVR and save huge amount of computation time. These classes are optimized for linear datasets and they are more efficient than SVC or SVR even if you use linear kernels with them. LinearSVC or LinearSVR will allow you to work with big data including datasets with millions of rows in reasonable computing time ranges (usually minutes to a couple of hours).
That being said working on large datasets won’t be nearly as feasible with SVC or SVR since they don’t scale as well with quadratic (or cubic) time complexity.
In short, Linear SVM models (LinearSVC and LinearSVR) are expected to have much more favorable computational complexity with O(N) due to solver Liblinear being used while SVC and SVR will have much higher complexity between O(N^2) and O(N^3) due to kernelized structure and usage of libsvm solver which works with kernel methods.
You can read time complexity of Support Vector Machine in this article:
Beyond complexity we have carried out a few speed tests that offer more insight regarding the runtime duration of different implementations with Support Vector Machine algorithm.
LinearSVC (75K): 2 seconds
LinearSVC (150K): 8 seconds
LinearSVC (750K): 85.5 seconds
LinearSVC (1M): 125 seconds
SVC (75K): ~1 minute
SVC (150K): ~3 minutes
SVC (300K): ~12 minutes
Please note : tests were done with non-dimensional data (only 2 features) using a fairly fast laptop (i7 8th Gen processor, 16GB RAM)
How to Use SVMs?
6- Scikit-Learn Support Vector Machine Implementation
Support Vector Machine algorithms can be used for both classification and regression problems. This means one needs to construct different models that’s suitable for the data and problem in hand.
Scikit-learn library offers extremely practical resources to tackle real world problems with Support Vector Machine algorithms.
- svm.SVC is a class named Support Vector Classifier that can be used for Support Vector Classification using Scikit-learn and Python.
- LinearSVC: Furthermore, there is a high performance SVM class which can be used to tackle linear classification problems more efficiently and that one is called LinearSVC.
- Classification with Support Vector Machines
- svm.SVR is a class named Support Vector Regressor that can be used for Support Vector Regression using Scikit-learn and Python.
- LinearSVR: Similar to classification usage, LinearSVR can be used to solve Linear regression problems and performs very efficiently.
We have created a special page for Classification with SVMs where you can read about SVR in more detail.
- Regression with Support Vector Machines
One of the reasons why LinearSVR and LinearSVC are much faster than SVR and SVC is the utilization of solvers. Liblinear is a solver used in both LinearSVC and LinearSVR and it helps them perform much faster than kernelized options SVC and SVR which use the slower Libsvm solver.
In Machine Learning solvers are used to find the optimum parameters to minimize the cost function and Support Vector Machine is no different.
Liblinear and Libsvm are open-source technologies contributed by National Taiwan University, specifically, C.J. Lin et. al.
You can read more about Liblinear and Libsvm from their official webpage and documentations.
Support Vector Machine Clustering is theoretically and practically possible but it is not a popular application for clustering in Machine Learning field. It was proposed by Ben Hur et al. in 2001 and if you are interested you can access the original paper from the clustering title above.
Creating the ideal SVM model
Usage of Support Vector Machine is also a lot about building the right model, using the correct settings, making the right adjustments and having a good command of parameter tuning. This is more so the case particularly with SVM.
You can read our SVM Optimization section which consists of detailed SVM parameter explanations, tips and tricks about optimizing Support Vector Machine algorithms as well as hyper-parameter fine tuning and more.
How can I improve Support Vector Machine?
7- Support Vector Machine Optimization
Support Vector Machine algorithm comes with with tons of optimization opportunities. It can be an experience to advance in understanding data relations and kerneling.
The more you tune a Support Vector Machine the more you will understand its art like implementation. Also more often than not, Support Vector Machines require rigorous optimization of its parameters based on dataset and project requirements.
So, we have prepared a tutorial to explain some of the most commonly adjusted SVM parameters and their ideal use cases. Please see the article below:
Is there a Support Vector Machine Implementation Example?
8- Support Vector Machine Example
Most machine learning algorithms can be mastered best via practice and examples. Support Vector Machine is no different. We have prepared a Support Vector Machine example where you can see the intricacies of an SVM implementation.
This example can be used to learn and/or teach training an SVM model, predicting with an SVM model, visualizing an SVM model and evaluating the prediction results of an SVM model.
How Do SVM Kernels Compare?
Linear vs Polynomial vs RBF vs Sigmoid
Support Vector Machines are used with kernels in fact that’s the whole point of Support Vector Machines since they use kernelling to separate data.
You can’t really master Support Vector Machine algorithm without mastering the kernels. Kernel trick lies at the heart of SVM implementations. For many different reasons, it’s actually quite important to know how to utilize kernels with SVMs, which kernels to use, how kernels perform and other nuances about different kernel choices.
Most commonly used Support Vector Machine kernels are as following:
- linear: Useful for linear data
- polynomial: Useful for polynomial data
- rbf: Useful for data with complex relations
- sigmoid: Useful for complex Neural Network like applications
For more details on right kernel selection when using Support Vector Machine classifier and regressor models you can check out our SVM Kernel page.
These kernels can easily be implemented in Scikit-learn using svm class for Support Vector Machine models. For more details on right kernel selection when using Support Vector Machine classifier and regressor models you can check out our SVM Kernel page.
- In the same kernel tutorial you can also find out answers to questions such as “How to identify data linearity?”, “How to apply kernels to Support Vector Machines”, “How to choose the right kernel for SVMs?” and “Which kernels perform the best?”. Tutorial here: Support Vector Machine Kernels Explained
What is SVM's status in Practice?
Real World SVM Implementations
Since its foundation in early 90s Support Vector Machines have seen lots of hype around them. After three decades Support Vector Machine continues to be a famous Machine Learning Algorithm and find many usage opportunity in different cases.
As SVM remains a popular Machine Learning Algorithm both in theoretical studies and practical applications we have gathered some of the exciting work that’s being done in the field for you. Here are some curated publications that can inspire you for SVM algorithm implementations in a specific domain.
SVM in Medical Research:
SVM in Finance:
SVM in Business:
SVM in Agirculture:
SVM in Energy:
SVM in Science and Engineering:
