Best practice

1
Package MCRestimate Maintainer Markus
Ruschhaupt
Misclassification error estimation with
cross-validation Crossvalidation is an
appropiate instrument to estimate the
misclassification rate (MCR) of a classification
algorithm. The function MCRestimate first
combines a classification algorithm, e.g. PAM,
and preprocessing stategies (variable
selection/aggregation) and then performes two
nested crossvalidation loops. The outer loop is
used to estimate the misclassification rate and
the inner loop is used to select optimal
parameters for this combination of preprocessing
method and classification algorithm. The whole
procedure is repeated several times with
different block constellations. The package
MCRestimate includes the function MCRestimate, a
plotting method for the resuls, various
preprocessing functions, and wrappers for some
common classification algorithms. However, it is
quite simple to write wrappers for other
classification algorithms.
Use tunedparameters
Use tunedparameters
Use tunedparameters
PAM.wrap lt- function(x,y,threshold,...)
require(pamr) trainlistlt- list(xt(x),yy)
model lt- pamr.train(trainlist,thresholdthreshold,
...) predict.function lt- function(testmatrix)
return(pamr.predict(model,t(testmatrix),threshold
threshold)) return(predict.function)
Outer cross-validation Estimate
misclassification rate
Inner cross-validation Tune parameters
Trainings set
Trainings set

Wrapper for PAM (Hastie T, Tibshirani R,
Narasimhan B, Chu G. pamr Pam - prediction
analysis for microarrays.
Result of MCRestimate For each sample we see how
often it was predicted correctly
Test set
Test set