Organize product data to your taxonomy with Amazon SageMaker

This leads to the second part in which I use another algorithm, XGBoost, to do the classification of our data, specifically to predict the class or genre of the book. This requires supervised learning, that is, a dataset that already has a target column, also known as the label. Therefore, there are some prerequisites to start labeling enough data to be able to train a classifier, because XGBoost is a supervised learning algorithm.

With machine learning (ML) problems, it all starts with the data. Ideally, your supervised dataset is balanced, that is, it has an evenly distributed number of rows per class. This improves the performance of the model training. If not, you can use some procedures to improve that situation. An example procedure for this is to use over-sampling or under-sampling. To do this, you could duplicate rows of the dataset from the minority class or remove rows from the majority class, respectively, to reach a more balanced dataset.

Dataset

I use a public dataset; the following table contains a subset of this dataset, including the features that we use to create our model.

This particular dataset is largely imbalanced. The main problem with this is that when training the model, the model favors the majority class. You might find yourself with a model that always predicts well for the majority class, but doesn’t perform as well for the non-majority classes. To this point, when analyzing the performance of the model, you want to check different metrics like precision, recall, and F1 score. These reveal that although the accuracy metric of the model is high, it isn’t good at predicting the minority class. In this case, you need to go back to the data and resolve the problem there.

In the following chart, on the x-axis we have the class label, where we have converted the category of the book to a number. The y-axis shows the count of each class label. The chart shows the class distribution of the data. You can see that categories 4.000 and 17.000 far outnumber the other classes. Even with the best algorithm, you will struggle to get good performance because of the large imbalance.

Methodology

SageMaker offers many built-in ML algorithms. A good candidate algorithm for a problem like this is to use BlazingText to do the text classification. However, in this post, I use GenSim’s implementation of FastText, a powerful NLP toolkit. FastText is useful because it can create word vectors from words that weren’t in the original training vocabulary. This functionality provides an advantage over Word2vec, which is part of the BlazingText implementation. This functionality also allows the solution to work reliably for the new data that is flowing in regularly.

I use the framework container for XGBoost and the scikit-learn implementation for XGBoost classification, which allows me to pass in a sample_weight parameter. This parameter isn’t available with the built-in XGBoost algorithm. By setting this parameter, I can inform the algorithm of weights per class, and the model performs better for the minority classes.

SageMaker notebook

I have published a SageMaker notebook that takes you through the preprocessing and training of the models.

You can launch a copy of the notebook within the SageMaker console and run the notebook cells to go through the entire workflow of inferring a product category from its product description and title.

You’re taken through a process of cleansing the data first, where I transform the case of the words. It also removes stopwords, that is, the common words that appear often and don’t help capture the semantics of the sentences. Then I tokenize the sentences so that they’re in a format ready to be consumed by the algorithm.

The following code snippets illustrate how you can generate vectors for the sentences in each row after you train your model.

The following code is an example sentence:

description_str[1]
'recreates the world from the second to the fourth century ad when the graecoroman gods lost their dominion and christianity with the conversion of constantine triumphed in the mediterranean world'

The following is an example of the sentence after feature engineering and tokenizing:

data_subset["description_str_token"][1]
['recreates',
 'world',
 'second',
 'fourth',
 'century',
 'ad',
 'graecoroman',
 'gods',
 'lost',
 'dominion',
 'christianity',
 'conversion',
 'constantine',
 'triumphed',
 'mediterranean',
 'world']

The next cell shows an example of the vector representation of the preceding sentence after our training. When training the model, I selected a vector size of 50, so the model has mapped this sentence to a vector in this space. From this vector, you could measure the cosine distance between this vector and another, and from that measurement, you could assess whether the vectors are similar in semantics.

vector_description_str[1]
array([[-0.01796775, -0.32192793,  0.4427467 ,  1.01180661, -0.34540969,
         0.19013038,  0.27239057,  0.34442747, -0.87200141,  0.39073905,
        -1.11305904,  0.15815021, -0.28595585, -0.67114389, -0.29362822,
         0.52312618, -0.48540148, -0.23456573,  0.16478145, -1.10297   ,
        -0.11184762, -0.36203459,  0.52202058, -0.24631335,  0.07348673,
         0.43191677, -0.02977397,  0.2740055 , -1.282076  , -0.1733315 ,
         0.46160921, -0.36126369,  0.36206457,  0.39822555,  0.03307921,
        -0.3936322 ,  0.82833076,  0.55542183, -0.37598661, -0.68967819,
         0.15875748, -0.17870745, -0.38433194, -0.41593724,  0.00131598,
        -0.13523583,  0.79293609,  0.46883106,  1.0133872 ,  0.27109554]])

One benefit of using SageMaker is that from your notebook, with a few lines of code, you can launch your training job on the Amazon Elastic Compute Cloud (Amazon EC2) instance of your choice. SageMaker orchestrates the sync of data to the training instance, deploys and runs the Docker image on the instance, and runs your training script or serves your inference. It also shuts down the infrastructure when the training is finished so you’re only charged for the seconds that your training took to complete. The EC2 instance and Docker container are spun up automatically. There is no need for the ML engineer to configure it, except for a single line of code to specify the size of the instances and the number of instances you require.

Results

When we train both models using different versions of XGBoost, the weighted version gives a slightly better F1 score than the built-in method. However, we could still do much more to the data to improve this metric. For instance, by using a larger sample of the same dataset, this metric starts to improve and the gap between the algorithms closes. So while building models, you sometimes need to test different methods, run experiments, and compare the metrics that are important to you. The first step to improve the performance is always to return to the data.

In the following code snippets, the F1 scores are largely the same, with the FastText score slightly higher. Out of the XGBoost’s F1 score, the version where you can supply the class weights performs slightly better. Keep in mind that we’re dealing with 32 classes with heavily imbalanced data. These are factors that you can control with your data before you start training a model for inference.

The following is the F1 score when we used XGBoost and supplied the weights of the classes:

from sklearn.metrics import f1_score
from sklearn.metrics import classification_report

predictions_test_xgb_weighted = [ float(xgb_predictor_gensim.predict(x)) for x in X_test]  
score = f1_score(y_test,predictions_test_xgb_weighted,labels=np.unique(y),average='micro')

print('F1 Score(micro): %.1f' % (score * 100.0))
F1 Score(micro): 60.1

The following is the F1 score when we used the built-in container image of XGBoost from SageMaker:

xgb_predictor.serializer = csv_serializer

predictions_test = [ float(xgb_predictor.predict(x).decode('utf-8')) for x in X_test] 
score = f1_score(y_test,predictions_test,labels=np.unique(y),average='micro')

print('F1 Score(micro): %.1f' % (score * 100.0))
F1 Score(micro): 56.8

The following is the F1 score when we use FastText’s text classification library:

FastText_Precision_Recall = model_native.test(valid_file_name, k=1)
print(FastText_Precision_Recall)
(11833, 0.6220738612355278, 0.6220738612355278)

f1_score = 2*((FastText_Precision_Recall[1]*FastText_Precision_Recall[2])/(FastText_Precision_Recall[1]+FastText_Precision_Recall[2]))
print('F1 Score(micro): %.1f' % (f1_score * 100.0))
F1 Score(micro): 62.2

Furthermore, in my notebook, I share an example of how you can create your FastText model locally. Then you can store this model binary in Amazon Simple Storage Service (Amazon S3). You can then deploy your model to a real-time SageMaker inference endpoint using the BlazingText container. This is compatible with FastText, so you don’t need to create your own container image to serve your predictions.

The following cell shows an example of invoking a prediction locally with FastText:

model_native.predict(fasttext_sample_validation[1], k=5)
(('__label__27', '__label__17', '__label__19', '__label__30', '__label__4'),
 array([0.63667679, 0.18908226, 0.08489674, 0.06093778, 0.00397073]))

The following cell shows an example of invoking a prediction in SageMaker, in the cloud with the same model, but using the BlazingText code to serve the inference:

predictions = predictor.predict(payload)
print(predictions)
[{'label': ['__label__27'], 'prob': [0.6366767287254333]}]

Although FastText returns multiple labels with a score for each, we can see label 27 comes in at 63% for both as the top result:

Data_subset_2[‘fastText_label’][1]
‘__label__27’

Clean up

To avoid incurring future charges, delete the SageMaker endpoints and stop the SageMaker notebook.

Conclusion

In this post, I explored how you can extract the semantics from sentences and use that to predict what class that sentence belongs to. You can use this solution to scale out a taxonomy workload that may otherwise be manual or require constant updating to work efficiently. Because you can deploy these models to a real-time inference endpoint, you can classify your data as soon as it lands in your data lake.

As a next step, try to modify the SageMaker notebook to download your dataset and experiment building your own classifier.

About the Author

Mark Watkins is a Solutions Architect within the Media and Entertainment team, supporting his customers solve many data and ML problems. Away from professional life, he loves spending time with his family and watching his two little ones growing up.

View Original Source (aws.amazon.com) Here.