Parameters of Random-Forest Algorithm for Classifications

Oracle Analytics includes several Machine Learning algorithms that enable data analysts to identify patterns in their data and make predictions on it. These various algorithms support different type of models : classifications, clustering or numeric predictions. Each algorithm offers several hyper-parameters to control the model training process and that can be manually tuned to improve overall model accuracy. It is important to understand these hyper-parameters to tune your training process in order to quickly get to the most accurate predictive model.

If we look in particular at classification models -both binary and multi-classification, Oracle Analytics offers several algorithms to perform it : 

                                     

Each of these come with a set of hyper-parameters that can be configured. In this blog, let us understand in detail on the Random Forest algorithm and review these parameters, they are actually fairly simple to understand and leverage :

Random Forest for model training

Random Forest is a classification algorithm that builds several Decision Tree models on the data and predicts using all these trees, called 'a forest of trees'.This algorithm builds each individual decision tree is by choosing a random sample data from the data set as the input. At each node of a tree, only a random sample of predictors is chosen for computing the split point. This introduces variation in the data used by the different trees in the forest.

Following is the list of hyperparameters available for this algorithm (for both binary classifications and multi-classifications): 

    

Let us understand these parameters in detail: 

• Target: This is the column with actual values which we wish to learn the model on. In case of binary classifications, the target must have two possible values (Yes/No, 0/1 and so on). In case of multi classifications, it can have more than two outcomes ( Low/Medium/High, North/East/South/West and so on).

• Positive class in Target: This allows the user to specify a value of the Target column that is interpreted as a positive class in the target. In order to classify the possible output values, the algorithm needs Positive and Negative classes. ‘Yes’ is the default option.

• Number of Trees in the forest: This hyper-parameter is considered to build the ensemble or forest of the model. It accepts a number between 2 and 100, with a default value of 10. A general guidance is, the more trees in the forest, the better the results. A higher number of trees means means more samples created from the initial training data-set. That reduces the biased-ness in the data used for training and can achieve more realistic results. However, increasing number of trees value will have performance implications for large datasets. 10 is the default value.

• Sample Size for a Tree: this determines the size of the sample data that is used for any single tree to predict the outcome. You can control the degree of randomness with the sample size parameter as increasing the sample size results in less 'variation' (randomness) in data for the individual trees in the forest. The best sample size is actually equal in size to the size of the original dataset : some rows are not selected while others are selected more than once. This typically provides near-optimal performance. However, in real-world applications, you may find that adjusting the sample size iteratively can lead to improved performance. This accepts a number between 500 and 10,000 with 500 as the default value.

• Number of Features for a Tree: This accepts a number between 2 and 10. Increasing the number of features generally improves the performance of the model as there are higher number of options to be considered at each node. However, this can decrease the diversity of an individual tree which is the significant aspect of random forest. Also, by increasing the features, the speed of algorithm decreases. Hence, we must find the right balance and choose the optimal value. 3 is the default value.

• Minimum Node Size: This is the minimum size of each leaf nodes in the decision tree and accepts any value between 10 and 100. This will constrain the tree into not being able to have a terminal node with less records than this number. When the limit is set high, it causes decision trees to be smaller and hence take less time to compute the samples. Setting it low leads to trees with a larger depth which means that more splits are performed until the terminal nodes, which can provide deeper insights on the data. Lower values are believed to generally provide good results, but performance can potentially be improved by tuning it. 50 is the default value.

• Maximum Depth: This represents the depth of each tree in the ensemble or the forest. The depth is the number of levels in the tree 'splits' hierarchy : how many levels from the first split to the furthest leaf level. The deeper the tree, the more the tree captures variance information about the data. The objective should be to expand each tree until every leaf is as pure as possible. A pure leaf is one where all the data comes from exactly the same class.  This parameter combines wiht Node Size as well, and has a direct impact on training performance : more levels require more processing time to train a model. Maximum Depth can be set up to 10 with 5 being the default.

• Maximum Confidence: This represents the confidence interval the model can take and can be used to rank the rules and hence the predictions. It is a measure of how sure the model is that the true value lies within a confidence interval computed by the model. This determines the likelihood of the predicted outcome, given that the rule has been satisfied. For example, consider a list of 1000 patients. Out of all the patients, 100 satisfy a given rule say, high fasting blood sugar level. Of these 100, 75 are likely to have heart disease, and 25 are not likely to have. The confidence of the prediction (likely to have) for the cases that satisfy the rule is 75/100 (75%). Maximum confidence parameter accepts a value between 1 and 100 with 80 being the default.

• Train Partition Percent: This is used to specify the percentage of training data that should be used to build model and the remaining will be used for internal testing purpose of the model. During the model creation process, the input data set is split into two parts to train and test the model based on the Train Partition Percent parameter. The model uses the test portion of the data set to test the accuracy of the model that is built. Default value is 80 which means 80% of the data is used to train the model and 20% to test model accuracy.

• Balancing method: This is used when there is an imbalance classification of the given samples : a skewed number of records have a given value (vast majority of 'No' and very few 'Yes').... In such situations, a method is applied in the algorithm to re-balance the set for better training : Under Sample, Over Sample and None are the options available. When the dataset is imbalanced, with majority of the records having a target attribute value of ‘No’, with under-sampling, the set of records where the target attribute is ‘No’ will be under-sampled to provide the same number of records as those with a value of ‘Yes’. When the dataset is small, an oversampling method can be used, and its effect is the opposite of under-sampling. Over-sampling will look at records having a target value of ‘Yes’ and over-sample these until it has the same number of records as those of the other target values. Default is Under Sample.

Case Study

To explore more about Random Forest algorithm and the effect of its parameters, let us consider a customer demographic data set with 1500 records. This dataset contains customer attributes like cust_gender, cust_marital_status, education, occupation, household_size and so on and we would like to train a model that will predict the customer response to an affinity card program with two possible outcomes Yes/No.

Here’s the sample dataset:

 

First, upload this dataset in Oracle Analytics. Next, create a dataflow with this input dataset and choose Train Binary Classifier step. Choose Random Forest for Model Training as the script.

A list of hyperparameters to train the model with default values as described in the previous section is displayed. Let us understand the effect of these hyperparameters on the model quality in different iterations.

Iteration 1: Let us leave these hyperparameters with default values and execute the dataflow. Successful dataflow execution creates a Machine Model which can be viewed from the Machine Learning tab. Let us right click on the model and inspect it to know more about its quality. With default parameter values, following is the model’s quality. 

Model accuracy is 76% which is computed as (True Positive + True Negative)/Total = (65+163)/300

Precision is 54% which is computed as (True Positive)/(True Positive+False Positive) = 65/(65+55)

False Positive Rate is 25% which is computed as False Positive/Actual Number of Negatives = 55/218

Iteration 2 : Let us increase the Sample Size of the tree from default 300 to 1500( as we have 1500 records in our input dataset). Let’s rebuild the model by executing the dataflow. Upon inspecting the model quality, we notice that while accuracy remains at 76%, precision has improved from 54% to 55%.

Iteration 3: Let us increase the number of trees from default 10 to 30 and Maximum depth from default 5 to 7.  Rebuild the model and inspect model details. 

We notice that the model accuracy has improved to 77%, precision has improved to 56% and False Positive Rate has dropped from 25% to 24%.

Conclusion:

Every Machine Learning algorithm in Oracle Analytics comes with a set of model training hyperparameters that can be tuned to improve overall accuracy of the model created. Given that the model building process via Dataflows is a fairly simple and intuitive process, it becomes easy to iteratively change the model hyperparameter values, inspect the model quality and subsequently arrive at a model of desired accuracy in a reasonable amount of time. 

Maps - Custom Image backgrounds Part II

The Voyageurs Wolf Project was an initiative taken by University of Minnesota in collaboration with the Voyageurs National Park to study the ecology of wolves and their prey(moose, dear, beavers etc) during the summer in Great Voyageurs ecosystem. Specifically this research was aimed at studying the movement pattern of the wolves and its reproductive ecology.

To track the movements of wolves and their packs, the researchers targeted multiple wolves of 4 different wolf packs and attached a GPS tracking device to the animals. The location of these wolves were tracked every hour for 3 alternate days. The results are fascinating and present a great use case for data analysis. Researchers have a specific image covering the habitat of Voyageurs National Park and they have a set of Lat/Longs representing the positions of the wolves over time, they need to visualize these positions on top of the habitat image. Oracle Analytics provides a very simple way to easily achieve this. Users have the ability to upload any images as map backgrounds and to create map layers on top of these images. In case you missed it, here is a blog entry which covers this feature in detail.

An Oracle Analytics user can upload the image of the National Park, and could take the approach of manually creating points (as a layer), representing possible positions of the wolves on top of the image. But with the amount of distinct Lat/Longs positions in the data, this process would be too hard or too long. The number of points is going to be huge and building a pre-set layer process will turn out to be a real pain.

Great news : that there is a much simpler and elegant way to achieve this, where we wipe away the manual and error-prone step of creating of point layers on top of the image. Oracle Analytics allows to directly plot Lat/Longs data (or any coordinates) on top of any image used as a map background. No need to create any layers !

Here are the simple steps to be followed to achieve the result

1) Upload the image of the national park as a map backgrounds

Image of Voyageur National Park

2) Upload the dataset having the position of the wolves and their corresponding positions.

Dataset used for the experiment

3) Find the National Park image background from the Maps Console and choose the Inspect option. By default the coordinates are set to Auto

Now choose Custom(Lat,Long) from the drop down. This prompts to enter the bounding Lat/Longs for the image

The bounding box of the lat/long covered by the Voyageur National Park can be easily obtained through a simple web search and we will be using that information to fill the above fields.

After entering the values for Lat/Long, click on the Save button to save the changes made to the image background.

4) Create a project using the dataset, drag and drop the Lat and Long columns on to a canvas and select map viz. The Map Viz will default with a world map background, and plot the points on the default map background.

5) Go to the viz properties and change the background map to Image background depicting the Voyageurs National Park : click on map viz properties (bottom left), click on the Map icon, and select the Voyageur National Park Map backgrdound we just uploaded

and you should see the wolf positions on the image background.

The color represents the different wolf packs that were involved in the research. It is a pretty visual point that distinct wolf packs seem to avoid each other’s range. They generally avoid being around each other unless they are fighting for food that may be in short supply. When that occurs, they may engage in battles with other packs in order to continue have their claim on a given location as well as the food found within it. 

So Oracle Analytics not only allows user to upload any images as backgrounds for map visuals, but provides the ability to either define layers on it, or directly use coordinate systems to plot any amount of points data on the image directly. That allows pretty visual representations of the data. This blog entry leveraged direct Lat/Long information for, this, in an upcoming blog entry we will highlight how this can be used with different use case, such as shop-floor traffic.