K-Fold Cross Validation is a method to estimate a classifier’s performance (accuracy, speed etc.) by measuring its performance on a large amount of training data and then repeating that process on a smaller set of data. So, how does this method work? Well, it takes a dataset, splits it into a training set and a test set, then trains a classifier and then predicts similar instances from the test set to the training set.
K-Fold cross validation is a popular feature in many machine learning models. It splits a problem into K folds (where K is a hyperparameter) and trains on each fold separately. The training of each fold is not independent and each one provides an approximation of the generalization error. This is also the reason why K-Fold cross validation is called a Cross Validation.
In this blog post, we will be learning how we can use the advanced features of PyTorch and the cross validation technique to develop a K-Fold cross validation for NLP and text classification.
Every machine learning model has a plethora of hyperparameters that must be specified. When it comes to performance optimization, hyperparameter tuning allows us to control the behavior of machine learning algorithms. Finding the right hyperparameter tuning for performance optimization is an art in and of itself, and there are no hard-and-fast rules that guarantee the best results on a given dataset.
We split our dataset into two pieces using the holdout method: a training and a test dataset. We evaluated the model after the machine learning algorithm fitted it to the training set on the independent test set that we withheld from the model fitting process. In our learning algorithms, we used fixed hyperparameter parameters such as learning rate and batch size. Hyperparameters are the parameters of the learning algorithm that must be specified a priori — prior to model fitting.
Choosing a model based on the performance of the test set appears to be a reasonable strategy. Multiple uses of the test set would result in unduly optimistic generalization performance estimations.
We’ll focus on K-Fold cross-validation for model evaluation in this lesson. In machine learning, K-fold cross-validation is the most often used technique for model evaluation and model selection.
The key concept underlying K-Fold cross-validation is that each sample in our dataset has a chance to be tested. It is a type of cross-validation in which we iterate k times across a dataset collection. Each round, we divide the dataset into k parts: one is used for validation, and the remaining k-1 parts are combined into a training subset for model evaluation, as indicated in the diagram below:
In most cases, the final score is the average of all the scores acquired across all k-folds.
Model created with PyTorch
For the MNIST dataset, let’s define a simple convolutional neural network.
def init (self): super MNISTNet(nn.Module) class MNISTNet(nn.Module) class MNISTNet(nn.Module) class MNISTNet(nn.Modul __init__ (MNISTNet, self) () self. conv1 = nnnnnnnnnnnnnnnnnn self Conv2d(1, 32, 3, 1) conv2 = nnnnnnnnnnnnnnnnnn self Conv2d(32, 64, 3, 1) dropout1 = nn nn nn nn nn nn Self-dropout (0.25). dropout2 = nnnnnnnnnnnnnnnnnn Dropout(0.5) nn = self.fc1. a straight line (9216, 128) nn = self.fc2. forward(self, x) def linear(128, 10): x is the same as self. x = F.relu(x) x = self conv1(x) x = F.relu(x) x = F.max pool2d conv2(x) x = F.max pool2d (x, 2) x is the same as self. dropout1(x) x = torch.flatten(x, 1) x = self.fc1(x) x = F.relu(x) x = self.fc1(x) x = self.fc1(x) x = self.fc1(x) x = self.fc1(x) x = self.fc1(x) self.fc2 = dropout2(x) (x) F.log softmax output (x, dim=1) (x, dim=1) (x, dim=1) ( produce a result
Model Weights Should Be Reset
You must reset the model’s weights so that each cross-validation fold begins with a random beginning state rather than learning from prior folds. All child modules could be reset using reset parameters().
function reset weights(m): m.reset parameters if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear): ()
It will be used to reset the model’s settings during the folds. This prevents weight leaking by ensuring that the model is trained with weights that are randomly initialized.
Dataset Concatenation
The train and test halves of the MNIST dataset, which we’ll use to train the model, must be concatenated. You don’t want PyTorch to do the splitting for you, thus K-fold means you produce the splits yourself.
dataset1 = datasets.MNIST(‘../data’, train=True, download=True, transform=transform); dataset2 = datasets.MNIST(‘../data’, train=True, download=True, transform=transform); dataset3 = datasets.MNIST dataset2 = datasets.MNIST(‘../data’, train=False,transform=transform), dataset2 = datasets.MNIST(‘../data’, train=False,transform=transform) dataset=ConcatDataset([dataset1,dataset2])
You may now create the fold and start training your model. You can achieve this by creating a loop that iterates across the fold, providing the fold and the training’s list of identifiers, and testing samples for that fold. These can be utilized to carry out the training procedure.
in enumerate(kfold.split(dataset)) for fold,(train idx,test idx): print(‘——————fold no————-{} ———————————’.format(fold)) torch.utils.data = train subsampler test subsampler = torch.utils.data SubsetRandomSampler(train idx) trainloader = torch.utils.data SubsetRandomSampler(test idx) testloader = torch.utils.data DataLoader( dataset, batch size=batch size, sampler=train subsampler) Model with DataLoader( dataset, batch size=batch size, sampler=test subsampler). train(fold, model, device, trainloader, optimizer, epoch) test(reset weights) for epoch in range(1, epochs + 1) (fold,model, device, testloader)
The current fold is printed first within the for loop. The training procedure is then carried out. Using a SubsetRandomSampler to sample the real elements from the train idx and test idx. Within a DataLoader, a sampler can be used to use only specific samples. The SubsetRandomSampler takes a random sample of elements from a list with no replacements. To put it another way, you make two subsamplers that follow the fold given in the forloop.
We’ll actually sample these samples from the entire dataset using the data loaders. We run functions to train and test the neural network after preparing the dataset for this fold.
Because you run the model numerous times on different dataset folds, K-fold cross-validation is quite expensive.
Use Google colab to run this code.
Deep learning excels at learning from data, however extracting the relevant knowledge from the huge amount of data is a challenging and tedious task. The traditional solution is to design models that are able to learn from the data and then use these models to classify the data. One such model, K-fold cross-validation is one approach for developing these models. User can easily implement this approach using dataloader and sklearn.. Read more about pytorch lightning cross validation and let us know what you think.
Related Tags:
skorch cross validationk fold cross validation sklearnk fold cross validation python codestratified k fold cross validationpytorch lightning cross validationk fold cross validation sagemaker,People also search for,Privacy settings,How Search works,skorch cross validation,k fold cross validation sklearn,k fold cross validation python code,stratified k fold cross validation,pytorch lightning cross validation,k fold cross validation python code without sklearn,k fold cross validation sagemaker,how to implement cross validation in pytorch
