Package: performanceEstimation 1.1.1

Luis Torgo

performanceEstimation: An Infra-Structure for Performance Estimation of Predictive Models

An infra-structure for estimating the predictive performance of predictive models. In this context, it can also be used to compare and/or select among different alternative ways of solving one or more predictive tasks. The main goal of the package is to provide a generic infra-structure to estimate the values of different metrics of predictive performance using different estimation procedures. These estimation tasks can be applied to any solutions (workflows) to the predictive tasks. The package provides easy to use standard workflows that allow the usage of any available R modeling algorithm together with some pre-defined data pre-processing steps and also prediction post- processing methods. It also provides means for addressing issues related with the statistical significance of the observed differences.

Authors:Luis Torgo [aut, cre]

performanceEstimation_1.1.1.tar.gz
performanceEstimation_1.1.1.zip(r-4.5)performanceEstimation_1.1.1.zip(r-4.4)performanceEstimation_1.1.1.zip(r-4.3)
performanceEstimation_1.1.1.tgz(r-4.4-any)performanceEstimation_1.1.1.tgz(r-4.3-any)
performanceEstimation_1.1.1.tar.gz(r-4.5-noble)performanceEstimation_1.1.1.tar.gz(r-4.4-noble)
performanceEstimation_1.1.1.tgz(r-4.4-emscripten)performanceEstimation_1.1.1.tgz(r-4.3-emscripten)
performanceEstimation.pdf |performanceEstimation.html✨
performanceEstimation/json (API)

# Install 'performanceEstimation' in R:
install.packages('performanceEstimation', repos = c('https://ltorgo.r-universe.dev', 'https://cloud.r-project.org'))

Peer review:

Bug tracker:https://github.com/ltorgo/performanceestimation/issues

On CRAN:

51 exports 16 stars 2.07 score 41 dependencies 1 dependents 175 scripts 430 downloads

Last updated 7 years agofrom:6262e570e7. Checks:OK: 1 NOTE: 6. Indexed: yes.

TargetResultDate
Doc / VignettesOKSep 16 2024
R-4.5-winNOTESep 16 2024
R-4.5-linuxNOTESep 16 2024
R-4.4-winNOTESep 16 2024
R-4.4-macNOTESep 16 2024
R-4.3-winNOTESep 16 2024
R-4.3-macNOTESep 16 2024

Exports:bootEstimatesBootstrapCDdiagram.BDCDdiagram.NemenyiclassificationMetricsComparisonResultsCVcvEstimatesEstimationResultsestimationSummaryEstimationTaskgetIterationsInfogetIterationsPredsgetScoresgetWorkflowhldEstimatesHoldoutis.classificationis.regressionknnImpLOOCVloocvEstimatesmcEstimatesmergeEstimationResmetricNamesmetricsSummaryMonteCarlopairedComparisonsperformanceEstimationplotPredTaskrankWorkflowsregressionMetricsresponseValuesresults2tablerunWorkflowshowsignifDiffssmotestandardPOSTstandardPREstandardWFsubsetsummarytaskNamestimeseriesWFtopPerformertopPerformersWorkflowworkflowNamesworkflowVariants

Dependencies:backportsBBmisccheckmateclicolorspacecpp11data.tabledplyrfansifarvergenericsggplot2gluegtableisobandlabelinglatticelifecyclemagrittrMASSMatrixmgcvmunsellnlmeparallelMappillarpkgconfigpurrrR6RColorBrewerrlangscalesstringistringrtibbletidyrtidyselectutf8vctrsviridisLitewithr

Readme and manuals

Help Manual

Help pageTopics
Performance estimation using (e0 or .632) bootstrapbootEstimates
Class "Bootstrap"Bootstrap Bootstrap-class show,Bootstrap-method
CD diagrams for the post-hoc Boferroni-Dunn testCDdiagram.BD
CD diagrams for the post-hoc Nemenyi testCDdiagram.Nemenyi
Calculate some standard classification evaluation metrics of predictive performanceclassificationMetrics
Class "ComparisonResults"ComparisonResults ComparisonResults-class plot,ComparisonResults-method show,ComparisonResults-method summary,ComparisonResults-method
Class "CV"CV CV-class show,CV-method
Performance estimation using cross validationcvEstimates
Class '"EstCommon"'EstCommon-class
Class "EstimationMethod"EstimationMethod EstimationMethod-class
Class "EstimationResults"EstimationResults EstimationResults-class plot,EstimationResults-method show,EstimationResults-method summary,EstimationResults-method
Obtain a set of descriptive statistics of the scores of a workflow on a taskestimationSummary
Class '"EstimationTask"'EstimationTask EstimationTask-class show,EstimationTask-method
Obtaining the information returned by a workflow when applied to a task, on a particular iteration of the estimation process or on all iterationsgetIterationsInfo
Obtaining the predictions returned by a workflow when applied to a task, on a particular iteration of the estimation process, or on all iterationsgetIterationsPreds
Obtaining the metric scores on the different iterations for a workflow / task combinationgetScores
Obtain the workflow object corresponding to an IDgetWorkflow
Performance estimation using holdout and random resamplinghldEstimates
Class "Holdout"Holdout Holdout-class show,Holdout-method
Check if a certain predictive task is a classification problemis.classification
Check if a certain predictive task is a regression problemis.regression
Fill in NA values with the values of the nearest neighboursknnImp
Class "LOOCV"LOOCV LOOCV-class show,LOOCV-method
Performance estimation using Leave One Out Cross ValidationloocvEstimates
Performance estimation for time series prediction tasks using Monte CarlomcEstimates
Merging several 'ComparisonResults' class objectsmergeEstimationRes
The evaluation metrics estimated in an experimentmetricNames
Obtains a summary of the individual metric scores obtained by each workflow on a set of tasks.metricsSummary
Class "MonteCarlo"MonteCarlo MonteCarlo-class show,MonteCarlo-method
Statistical hypothesis testing on the observed paired differences in estimated performance.pairedComparisons
Estimate the predictive performance of modeling alternatives on different predictive tasksperformanceEstimation
Class "PredTask"PredTask PredTask-class show,PredTask-method
Provide a ranking of workflows involved in an estimation process.rankWorkflows
Calculate some standard regression evaluation metrics of predictive performanceregressionMetrics
Obtain the target variable values of a prediction taskresponseValues
Obtains a dplyr data frame table object containing all the results of an experimentresults2table
Run a workflow on a predictive taskrunWorkflow
Obtains a list with the set of paired differences that are statistically significant according to a p-value thresholdsignifDiffs
SMOTE algorithm for unbalanced classification problemssmote
A function for applying post-processing steps to the predictions of a modelstandardPOST
A function for applying data pre-processing stepsstandardPRE
A function implementing a standard workflow for prediction tasksstandardWF
Methods for Function 'subset' in Package 'performanceEstimation'subset,ComparisonResults-method subset-methods
The prediction tasks involved in an estimation experimenttaskNames
A function implementing sliding and growing window standard workflows for time series forecasting taskstimeseriesWF
Obtain the workflow that best performed in terms of a metric on a tasktopPerformer
Obtain the best scores from a performance estimation experimenttopPerformers
Class "Workflow"show,Workflow-method summary,Workflow-method Workflow Workflow-class
The IDs of the workflows involved in an estimation experimentworkflowNames
Generate (parameter) variants of a workflowworkflowVariants