Predictive modeling

modeling
Predictive modeling comparison in R using caret and tidymodels for preprocessing, resampling, training, and model evaluation.
Published

June 5, 2022

Overview

This post is dedicated to make a comparison between Caret and TidyModels R packages. Data modelling with R pass through data preprocessing and parameters assessments to predicting an outcome. Both set of packages can be used to achieve same results, with the purpose of finding the best predictive performance for data specific models.

Predictive modeling - TreeMap

The Caret package is the starting point for understanding how to manage models and produce unbiases predictions with R. As well as TidyModels meta package, it gives the opportunity to contruct a multivariate model syntax to manage several models to be applied on same set of data. TidyModels allows the use of a set of concatenated functions in partership with the TidyVerse grammar to build a structural model base which blends different models as one global model.

The following is an attempt to a comparison between the two predictive model structures.

Caret package

The most important functions for this package, grouped by steps to modeling, are:

  1. Preprocessing (data cleaning/wrangling)

    • preProcess()

  2. Data splitting and resampling

    • createDataPartition()

    • createResample()

    • createTimeSlices()

  3. Model fit and prediction

    • train()
    • predict()

  4. Model comparison

    • confusionMatrix()

TidyModels meta package

This β€œmeta package” is made of a set of packages for modeling, with the support of other well known packages for data manipulation and visualization such as broom, dplyr, ggplot2, purrr, infer, modeldata, and tibble; it includes:

  • recipes (a preprocessor)
  • rsample (for resampling)
  • parsnip (model syntax)
  • tune and dials (optimization of hyperparameters)
  • workflows and workflowsets (combine pre-processing steps and models)
  • yardstick (for evaluating models)

The most important functions for this meta package, grouped by steps to modeling, are:

  1. Preprocessing (data cleaning/wrangling)

    • recipes::recipe()

    • recipes::step_()

  2. Data splitting and resampling

    • rsample::initial_split()

    • rsample::training()

    • rsample::testing()

    • rsample::bootstraps()

    • rsample::vfold_cv()

    • tune::control_resamples()

  3. Model fit and prediction

    • parsnip::() %>% set_mode() %>% set_engine()

    • parsnip::extract_fit_engine()

    • parsnip::extract_fit_parsnip()

    • parsnip::fit() stats::predict()

    • tune::fit_resamples()

  4. Model workflow

    • workflows::workflow() %>% add_model()

    • workflows::add_formula()

    • workflows::add_recipe()

    • parsnip::fit()

    • stats::predict()

    • workflows::update_formula()

    • workflows::add_variables() / remove_variables()

    • workflowsets::workflow_set()

    • workflowsets::workflow_map()

    • workflowsets/tune::extract_workflow() / extract_recipe() / extract_fit_parsnip()

    • tune::last_fit()

    • workflowsets/tune::collect_metrics()

    • workflowsets/tune::collect_predictions()

  5. Model comparison

    • yardstick::conf_mat()

    • yardstick::accuracy()

    • yardstick::metric_set()

    • yardstick::roc_curve()

    • yardstick::roc_auc()

    • yardstick::sensitivity()

Machine learning algorithms in R

  • Linear discriminant analysis
  • Regression
  • Naive Bayes
  • Support vector machines
  • Classification and regression trees
  • Random forests
  • Boosting
  • etc.

Resource: Practical Machine Learning

Caret or TidyModels?

Caret Tidymodels

Caret Example with SPAM Data

library(caret); library(kernlab); data(spam)
Loading required package: ggplot2
Loading required package: lattice

Attaching package: 'kernlab'
The following object is masked from 'package:ggplot2':

    alpha
inTrain <- createDataPartition(y=spam$type,
                              p=0.75, list=FALSE)
training <- spam[inTrain,]
testing <- spam[-inTrain,]
# dim(training)

set.seed(32343)
modelFit <- train(type ~.,data=training, method="glm")
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
# modelFit

predictions <- predict(modelFit,newdata=testing)
# predictions

cm <- confusionMatrix(predictions,testing$type)
cm
Confusion Matrix and Statistics

          Reference
Prediction nonspam spam
   nonspam     659   52
   spam         38  401
                                          
               Accuracy : 0.9217          
                 95% CI : (0.9047, 0.9366)
    No Information Rate : 0.6061          
    P-Value [Acc > NIR] : <2e-16          
                                          
                  Kappa : 0.8352          
                                          
 Mcnemar's Test P-Value : 0.1706          
                                          
            Sensitivity : 0.9455          
            Specificity : 0.8852          
         Pos Pred Value : 0.9269          
         Neg Pred Value : 0.9134          
             Prevalence : 0.6061          
         Detection Rate : 0.5730          
   Detection Prevalence : 0.6183          
      Balanced Accuracy : 0.9153          
                                          
       'Positive' Class : nonspam         
                                          
plot(cm$table,main="Table")

TidyModels Example with SPAM Data

── Attaching packages ────────────────────────────────────── tidymodels 1.4.1 ──
βœ” broom        1.0.10     βœ” rsample      1.3.1 
βœ” dials        1.4.2      βœ” tailor       0.1.0 
βœ” dplyr        1.1.4      βœ” tidyr        1.3.1 
βœ” infer        1.0.9      βœ” tune         2.0.1 
βœ” modeldata    1.5.1      βœ” workflows    1.3.0 
βœ” parsnip      1.3.3      βœ” workflowsets 1.1.1 
βœ” purrr        1.2.1      βœ” yardstick    1.3.2 
βœ” recipes      1.3.1      
── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
βœ– scales::alpha()          masks kernlab::alpha(), ggplot2::alpha()
βœ– dials::buffer()          masks kernlab::buffer()
βœ– rsample::calibration()   masks caret::calibration()
βœ– purrr::cross()           masks kernlab::cross()
βœ– purrr::discard()         masks scales::discard()
βœ– dplyr::filter()          masks stats::filter()
βœ– dplyr::lag()             masks stats::lag()
βœ– purrr::lift()            masks caret::lift()
βœ– yardstick::precision()   masks caret::precision()
βœ– yardstick::recall()      masks caret::recall()
βœ– yardstick::sensitivity() masks caret::sensitivity()
βœ– yardstick::specificity() masks caret::specificity()
βœ– recipes::step()          masks stats::step()
tidymodels_prefer()
set.seed(123)
split <- initial_split(spam,0.75,strata=type)
training <- training(split)
testing <- testing(split)

modelFit <- logistic_reg() %>% 
  set_engine("glm") %>%
  fit(type~.,data=spam)
Warning: glm.fit: fitted probabilities numerically 0 or 1 occurred
# tidy(modelFit)

predictions <- predict(modelFit,new_data=testing)
# predictions

testing$pred <- predictions$.pred_class
cm <- yardstick::conf_mat(data = testing, truth = type, estimate = pred)
cm
          Truth
Prediction nonspam spam
   nonspam     668   50
   spam         29  404

Back to top