Practical Guidelines for Getting Started with Machine Learning

Wed 5 Sep 2018 Gregory J Stein
Part 3 of AI Perspectives

As the use of machine learning systems grows beyond the academic sphere, one of the more worrying features I have witnessed is a lack of understanding of how machine learning systems should be trained and applied. The lessons the AI community has learned over the last few decades of research are hard-earned, and it should go without saying that those who do not understand the inner workings of a machine learning tool risk having that system fail in often surprising ways.

This advice is not limited to AI. Using any stochastic system without an understanding of when or how it is likely to fail comes with inherent risk.

However, the potential advantages of AI are many, and using machine learning to accelerate your business, whether empowering employees or improving your product, may outweigh potential pitfalls. If you are looking to use machine learning tools, here are a few guidelines you should keep in mind:

  • Establish clear metrics for success.
  • Start with the simplest approach.
  • Ask yourself if machine learning is even necessary.
  • Use both a test and a validation dataset.
  • Understand and mitigate data overfitting.
  • Be wary of bias in your data.

Even before you get started, it is important to establish clear metrics for success. How is performance measured? Is it measured as accuracy on a particular task? Is it improvement in user feedback or retention? You should also ask yourself how much of an improvement in performance you would require before you would put it into practice: is the hassle of maintaining the machine learning system worth the benefits it provides?

Having a clear idea of what success would look like is good advice in general.

In addition to establishing a performance metric, you should also have a performance baseline. Are you comparing the system to human performance or is there another automated system that you might be comparing against? With this in mind, I always recommend that one start with the simplest approach. This is a topic I've written about in the past, but it certainly bears repeating. Many modern machine learning tools are complex and opaque, and sometimes a simpler approach is enough to get excellent performance. Even if the simpler approaches do not reach the performance goal you might be looking for, they may serve as performance baselines of their own, against which the more complex techniques can be compared.

K-nearest neighbors and Support Vector Machine are two relatively easy-to-implement machine learning techniques that each make for a good starting point when tackling a new task. These two apply to a large number of problems and do not require very much parameter tuning.

You might also ask yourself if machine learning is even necessary to solve the problem you are looking to overcome. Nearly a decade ago, I very nearly automated myself out of a job working as a paralegal. Many of the tasks I had been assigned were various types of data entry and were rather straightforward to automate with conventional scripting in Visual Basic: no machine learning necessary. This example is rather clear-cut, but I hear about scenarios like this one in different industries rather frequently. A programmer may be better suited than a data scientist to solve your problem.

I recommend taking a look at this article in which a database programmer in the e-commerce industry describes how a combination of asking the right questions and writing a few clever SQL queries vastly improved the open rate for the company's marketing emails.

The way in which the system is trained is also important to keep in mind. One important guideline is to use both a test and a validation dataset. For outsiders to the machine learning community, it is usually clear that the data used to tune the parameters of the algorithm, often known as the training set, should not be used to evaluate the performance of the algorithm. A separate test set should be provided to evaluate the performance once the algorithm is trained. However, one might want to tune the parameters of the algorithm in response to the results on the test set. This is bad practice: instead, there should be another data set, known as the validation set, on which the parameters of the algorithm can be tuned. Only once the parameters of the algorithm are fixed should the final performance statistics be generated on the test set.

The primary reason you should want more data than just your training dataset is so that you can understand and mitigate data overfitting. With only a single dataset, it's easy to be fooled into thinking performance is very good. Convolutional Neural Networks, a popular tool for tasks like image classification and object detection, are so powerful that they are capable of correctly classifying a training dataset of random noise with very high accuracy. However, a machine learning algorithm trained in such a way expectedly performs poorly on the test and validation datasets, since there is no correlation between them and the data upon which the learning algorithm was trained. Some overfitting to your training data is expected, but a large performance gap between the training and test sets may indicate that more data is necessary or that a different machine learning technique should be used.

Finally, you should always be wary of bias in your data. This is another issue I have written about at length. Bias manifests itself in subtle ways and can influence the performance of your system in practice. In particular, if your machine learning system will directly interact with users, be sure that the training, test, and validation data you use are collected so that they matche the sort of data that your system will see in production: this is a common failure mode for machine learning systems. In particular, this effect has been known to exacerbate social biases, particularly when there is a disparity between the composition of those who train the system and the end users of the system.

For those curious to read more about dataset bias, I would recommend the fantastic research article Unbiased Look at Dataset Bias by Antonio Torralba and Alexei Efros, two influential researchers in the field of computer vision.

These are just a handful of the things one should keep in mind when using machine learning to make decisions in your business. A slightly more technical guide, Google's Best Practices for ML Engineering is worth taking a look at: it lists over 40 guidelines one should consider when working with machine learning systems in production and is curated by some of the most talented machine learning engineers in the world. In short, be cautious when using machine learning; despite the incredible promise of these techniques, you should take care to ensure that these systems are actually performing well.

Liked this post? Subscribe to our RSS feed or add your email to our newsletter:

all posts in series

AI Perspectives

Reflections on the progress, promise, and impact of AI.
  • 1 Bias in AI Happens When We Optimize the Wrong Thing
    Bias is a pervasive problem in AI. Only by discouraging machine learning systems from exploiting a certain bias can we expect such a system to avoid doing so.
  • 2 For AI, translation is about more than language
    Translation is about expressing the same underlying information in different ways, and modern machine learning is making incredibly rapid progress in this space.
  • 3 Practical Guidelines for Getting Started with Machine Learning
    The potential advantages of AI are many, and using machine learning to accelerate your business may outweigh potential pitfalls. If you are looking to use machine learning tools, here are a few guidelines you should keep in mind.
  • 4 DeepMind's AlphaZero and The Real World
    Using DeepMind's AlphaZero AI to solve real problems will require a change in the way computers represent and think about the world. In this post, we discuss how abstract models of the world can be used for better AI decision making and discuss recent work of ours that proposes such a model for the task of navigation.
  • 5 Massive Datasets and Generalization in ML
    Big, publically available datasets are great. Yet many practitioners who seek to use models pretrained on this data need to ask themselves how informative the data is likely to be for their purposes. Dataset bias and task specificity are important factors to keep in mind.
  • 6 Proxy metrics are everywhere in Machine Learning
    Many machine learning systems are optimized using metrics that don't perfectly match the stated goals of the system. These so-called "proxy metrics" are incredibly useful, but must be used with caution.

+ Show Comments From Disqus