On this page

Algorithms & Hyperparameters

 

ALGORITHMS AND PARAMETERS

 

Scaling

Scaling before applying ML algorithms is very important. The main advantage of scaling is to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges and to avoid numerical difficulties during the calculation.

we perform linear scaling that is scale range [-1,1].

Tune Param

Proper choice of C and gamma is critical to the SVM’s performance. The user is advised to select Tune Param .

On selecting Tune Param – you are performing cross-validation and Grid Search. You can consider speed as step length between two consecutive values C and gamma.

CLASSIFICATION

 

Parameters

values

Definition

Tips

SVM_Type

0 – C-SVC

C-SVC, NuSVC and One-Class SVM performes binary and multi-class classification on a dataset. C-SVM and NuSVM are similar methods, but accept slightly different sets of parameters and have different mathematical formulations.

One-class SVM algorithms, learns a decision function for novelty detection: classifying new data as similar or different to the training set.

1 – Nu-SVC

2 – ONE-CLASS SVM

Kernel Type

0 – Linear

linear: u’*v

Radial Basis Function is a general purpose kernel, used when there is no prior knowledge about the data because

1. The linear kernel is a special case of RBF since the linear kernel with a penalty parameter C has the same performance as the RBF kernel with some parameters (C, gamma)

2. The second reason is the number of hyperparameters which influences the complexity of model selection. The polynomial kernel has more hyperparameters than

the RBF kernel.

There are some situations where the RBF kernel is not suitable. In particular, when the number of features is very large, one may just use the linear kernel.

1 – Polynomial

polynomial: (gamma*u’*v + coef0)^degree

2 – RBF

radial basis function: exp(-gamma*|u-v|^2)

This kernel nonlinearly maps samples into a higher dimensional space so it, unlike the linear kernel, can handle the case when the relation between class labels and attributes is nonlinear.

3 – Sigmoid

sigmoid: tanh(gamma*u’*v + coef0)

Gamma

[0.000122,8]

gamma defines how much influence a single training example has. The larger gamma is, the closer other examples must be to be affected.

Degree

Degree of the polynomial kernel function. Ignored by all other kernels.

Coef0

Independent term in kernel function. It is only significant in ‘polynomial’ and ‘sigmoid’.

Cost (C)

[0.031250,8192]

The parameter C, trades off misclassification of training examples against simplicity of the decision surface. A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly. As C increases, tendency to misclassification decreases on train data( may lead to overfitting).

C is 1 by default and it’s a reasonable default choice. If you have a lot of noisy observations you should decrease it: decreasing C corresponds to more regularization.

NU

(0,1]

It’s a hyperparameter for nu-SVC, one-class SVM and nu-SVR. It is similar to C. nu is upper bound on the fraction of errors and lower bound on the fraction of number of support vectors( number of support vectors determine the run time).

Example: if we want error to be less than 1% then nu is 0.01 and the number of supported vectors will be more than 1% of the total records.

Nu approximates value = the fraction of training errors and support vectors.

Cachesize

For C-SVC, SVR, NuSVC and NuSVR, the size of the kernel cache has a strong impact on run times for larger problems.

If you have enough RAM available, it is recommended to set cache size to a higher value than the default of 200(MB), such as 500(MB) or 1000(MB).

Termination Criterion

Tolerance for stopping criterion. The stopping tolerance affects the number of iterations used when optimizing the model.

Shrinking

The shrinking are there to save the training time.They sometimes help, and sometimes they do not. It’s a matter of runtime, rather than convergence. If the number of iterations is large, then shrinking can shorten the training time.

We found that if the number of iterations is large, then shrinking can shorten the training time.

Probability_Estimates

Whether to enable probability estimates.

nr_weight

nr_weight is the number of elements in the array weight_label and weight. Each weight[i] corresponds to weight_label[i], meaning that the penalty of class weight_label[i] is scaled by a factor of weight[i].

REGRESSION

 

Parameters

values

Definition

Tips

SVM_Type

3 – Epsilon-SVR

The Nu parameter in Nu-SVM can be used to control the amount of support vectors in the resulting model. However, in ϵ-SVR you have no control on how many data vectors from the dataset become support vectors, it could be a few, it could be many. Nonetheless, you will have total control of how much error you will allow your model to have, and anything beyond the specified ϵ will be penalized in proportion to C, which is the regularization parameter.

4 – Nu-SVR

Kernel Type

0 – Linear

linear: u’*v

Radial Basis Function is a general purpose kernel, used when there is no prior knowledge about the data because

1. The linear kernel is a special case of RBF since the linear kernel with a penalty parameter C has the same performance as the RBF kernel with some parameters (C, gamma)

2. The second reason is the number of hyperparameters which influences the complexity of model selection. The polynomial kernel has more hyperparameters than

the RBF kernel.

There are some situations where the RBF kernel is not suitable. In particular, when the number of features is very large, one may just use the linear kernel.

1 – Polynomial

polynomial: (gamma*u’*v + coef0)^degree

2 – RBF

radial basis function: exp(-gamma*|u-v|^2)

This kernel nonlinearly maps samples into a higher dimensional space so it, unlike the linear kernel, can handle the case when the relation between class labels and attributes is nonlinear.

3 – Sigmoid

sigmoid: tanh(gamma*u’*v + coef0)

Gamma

[0.000122,8]

gamma defines how much influence a single training example has. The larger gamma is, the closer other examples must be to be affected.

Degree

Degree of the polynomial kernel function. Ignored by all other kernels.

Coef0

Independent term in kernel function. It is only significant in ‘polynomial’ and ‘sigmoid’.

Cost (C)

[0.031250,8192]

Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive. The penalty is a squared l2 penalty.

C is 1 by default and it’s a reasonable default choice. If you have a lot of noisy observations you should decrease it: decreasing C corresponds to more regularization.

NU

(0,1]

It’s a hyperparameter for nu-SVC, one-class SVM and nu-SVR. It is similar to C. nu is upper bound on the fraction of errors and lower bound on the fraction of number of support vectors( number of support vectors determine the run time).

Example: if we want error to be less than 1% then nu is 0.01 and the number of supported vectors will be more than 1% of the total records.

Nu approximates value = the fraction of training errors and support vectors.

Epsilon_SVR (P)

Epsilon in the epsilon-SVR model. It specifies the epsilon-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual value.

Cachesize

For C-SVC, Epsilon-SVR, NuSVC and NuSVR, the size of the kernel cache has a strong impact on run times for larger problems.

If you have enough RAM available, it is recommended to set cache size to a higher value than the default of 200(MB), such as 500(MB) or 1000(MB).

Termination Criterion

Tolerance for stopping criterion. The stopping tolerance affects the number of iterations used when optimizing the model.

Shrinking

The shrinking are there to save the training time.They sometimes help, and sometimes they do not. It’s a matter of runtime, rather than convergence. If the number of iterations is large, then shrinking can shorten the training time.

We found that if the number of iterations is large, then shrinking can shorten the training time.

Probability_Estimates

Whether to enable probability estimates.

LINEAR REGRESSION

 

Parameters

values

Definition

Solver

11 – L2-regularized L2-loss SVR primal

We have 3 linear Regression solvers, by combining several types of loss functions and regularization schemes. The regularization

can be L1 or L2, and the losses can be the regular L2-loss for SVM (hinge loss), or L1-loss for SVM. The default value for type is 11

12 – L2-regularized L2-loss SVR dual

13 – L2-regularized L1-loss SVR dual

Cost (C)

The parameter C, trades off misclassification of training examples against simplicity of the decision surface. A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly. As C increases, tendency to misclassification decreases on train data( may lead to overfitting).

Epsilon_SVR (P)

Epsilon in the epsilon-SVR model. It specifies the epsilon-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual value.

Termination Criterion

Tolerance for stopping criterion. The stopping tolerance affects the number of iterations used when optimizing the model.

Folds

V-fold for Cross Validation. In v-fold cross-validation, we first divide the training set into v subsets of equal size. Sequentially one subset is tested using the classifier trained on the remaining v − 1 subsets. Thus, each instance of the whole training set is predicted once so the cross-validation accuracy is the percentage of data which are correctly classified

LINEAR CLASSIFICATION

 

Parameters

values

Definition

Solver

0 – L2-regularized logistic regression primal,

1 – L2-regularized L2-loss SVC dual,

2 – L2-regularized L2-loss SVC primal ,

3 – L2-regularized L1-loss SVC dual,

We have 8 linear Classification solvers, by combining several types of loss functions and regularization schemes. The regularization

can be L1 or L2, and the losses can be the regular L2-loss for SVM (hinge loss), L1-loss for SVM, or the logistic loss for logistic regression. The default value for type is 0

4 – Support Vector Classification by Crammer and Singer

5 – L1-regularized L2-loss SVC

6 – L1-regularized Logistic regression ,

7 – L2-regularized Logistic regression dual

Cost (C)

Regularization parameter. The strength of the regularization is inversely proportional to C. Must be strictly positive. The penalty is a squared l2 penalty.

Bias

Conside : w_1 * x_1 + w_2 * x_2 + w_3 * x_3 + … + w_bias * x_bias = 0, Here x are the feature values and w are the trained “weights”. The additional feature x_bias is a constant, whose value is equal to the bias.

Termination Criterion

Tolerance for stopping criterion. The stopping tolerance affects the number of iterations used when optimizing the model.

Folds

V-fold for Cross Validation. In v-fold cross-validation, we first divide the training set into v subsets of equal size. Sequentially one subset is tested using the classifier trained on the remaining v − 1 subsets. Thus, each instance of the whole training set is predicted once so the cross-validation accuracy is the percentage of data which are correctly classified.

nr_weight

nr_weight is the number of elements in the array weight_label and weight. Each weight[i] corresponds to weight_label[i], meaning that the penalty of class weight_label[i] is scaled by a factor of weight[i].

Weight (wi)

set the parameter C of class i to weight*C, for C-SVC.

Weight_Label

These weights are used to change the penalty for specific labels (classes). If the weight for a label is not changed, it is set to 1.0.

K-MEANS

 

Parameters

values

Definition

Kernel Type

0. LINEAR

linear: u’*v

1. POLYNOMIAL

polynomial: (gamma*u’*v + coef0)^degree

2. RBF

radial basis function: exp(-gamma*|u-v|^2)

This kernel nonlinearly maps samples into a higher dimensional space so it, unlike the linear kernel, can handle the case when the relation between class labels and attributes is nonlinear.

3. SIGMOID

sigmoid: tanh(gamma*u’*v + coef0)

Gamma

gamma defines how much influence a single training example has. The larger gamma is, the closer other examples must be to be affected.

Coef0

Degree of the polynomial kernel function. Ignored by all other kernels.

Degree

Independent term in kernel function. It is only significant in ‘polynomial’ and ‘sigmoid’.

Dimension (Number of Attributes)

Number of input attributes / columns in the training data set

Number of Centers

Number of clusters

Stopping Criteria

Tolerance for stopping criterion. The stopping tolerance affects the number of iterations used when optimizing the model.

Number of Rows

Total number of records / rows in the training data

 

Was this article helpful to you? Yes No