Apply V-Fold Cross-Validation to a List of Datasets

Source: R/split_cv.R

split_cv creates (optionally repeated and stratified) v-fold cross-validation splits for every dataset in a list. It is implemented with base R and data.table only and returns, per dataset, a data.table of fold identifiers, row indices and (optionally) the training / validation subsets.

split_cv(
  data,
  v = 10L,
  repeats = 1L,
  strata = NULL,
  breaks = 4L,
  pool = 0.1,
  seed = NULL,
  materialize = TRUE,
  out_type = "dt"
)

Arguments

data

A list whose every element is a data.frame or data.table. Must be non-empty. A single data.frame is not accepted (wrap it in list()).

v

Number of folds. A single integer >= 2. Default 10.

repeats

Number of repeats. A single integer >= 1. Default 1.

strata

NULL (default) or a single column name present in every dataset. Rows are shuffled within each stratum so that every fold receives a proportional share of each stratum.

breaks

Number of quantile bins used when strata is numeric with more than breaks distinct values. Default 4.

pool

Strata holding less than this proportion of the rows are merged with the next-smallest stratum. A number in [0, 0.5). Default 0.1.

seed

NULL (default) or a single number for reproducible folds. The caller's random-number stream is restored on exit.

materialize

Logical. If TRUE (default), train / validate list-columns holding the subsets are added. FALSE keeps only the row indices, which needs far less memory for large data.

out_type

Class of the train / validate subsets: "dt" (data.table, default) or "df" (plain data.frame, e.g. for 'ASReml-R' or other modelling functions that expect a data.frame). Factor levels are kept in full in every subset. Only used when materialize = TRUE; the returned fold table itself is always a data.table.

Value

A list of data.table objects (one per input dataset, names of data preserved), each with one row per fold and the columns:

  • id — fold label (Fold1, ...), or repeat label (Repeat1, ...) when repeats > 1.

  • id2 — fold label, only when repeats > 1.

  • train_idx — list-column of training row indices.

  • validate_idx — list-column of validation row indices.

  • train, validate — list-columns of subsets, data.table or data.frame according to out_type (only when materialize = TRUE).

Details

Fold sizes differ by at most one row. With stratification, rows are shuffled within strata, the strata are laid out one after another and fold labels are dealt out cyclically, so each stratum is spread evenly across folds. The stratification rules (breaks, pool) follow the same idea as rsample::vfold_cv(), but the random fold assignments are not identical to rsample's.

Training and validation subsets can always be rebuilt from the indices: data[[1]][res[[1]]$train_idx[[k]], ].

For mixed-model cross-validation (e.g. genomic prediction with 'ASReml-R') a common alternative to subsetting is masking: keep all records, set the phenotypes of validate_idx to NA, fit the model on the full data and correlate the predicted breeding values of the masked individuals with their observations. materialize = FALSE is enough for this, since only the indices are needed.

See also

nest_cv() for the nested data.table variant.

Examples

# Prepare example data: Convert first 3 columns of iris dataset to long format and split
dt_split <- w2l_split(data = iris, cols = 1:3)
# dt_split is now a list containing 3 data tables for Sepal.Length, Sepal.Width, and Petal.Length

# Example 1: Single cross-validation (no repeats)
split_cv(
  data = dt_split,      # Input list of split data
  v = 3,                # Set 3-fold cross-validation
  repeats = 1,          # Perform cross-validation once (no repeats)
  seed = 123            # Reproducible folds
)
#> $Sepal.Length
#>        id                  train_idx              validate_idx
#>    <char>                     <list>                    <list>
#> 1:  Fold1  1, 2, 5, 7, 9,10,...[100]  3, 4, 6, 8,15,19,...[50]
#> 2:  Fold2       3,4,5,6,7,8,...[100]  1, 2, 9,10,11,14,...[50]
#> 3:  Fold3       1,2,3,4,6,8,...[100]  5, 7,12,13,16,17,...[50]
#>                  train           validate
#>                 <list>             <list>
#> 1: <data.table[100x3]> <data.table[50x3]>
#> 2: <data.table[100x3]> <data.table[50x3]>
#> 3: <data.table[100x3]> <data.table[50x3]>
#> 
#> $Sepal.Width
#>        id                  train_idx              validate_idx
#>    <char>                     <list>                    <list>
#> 1:  Fold1  2, 4, 5, 6, 7,13,...[100]  1, 3, 8, 9,10,11,...[50]
#> 2:  Fold2       1,2,3,4,6,8,...[100]  5, 7,13,14,17,21,...[50]
#> 3:  Fold3       1,3,5,7,8,9,...[100]  2, 4, 6,15,18,22,...[50]
#>                  train           validate
#>                 <list>             <list>
#> 1: <data.table[100x3]> <data.table[50x3]>
#> 2: <data.table[100x3]> <data.table[50x3]>
#> 3: <data.table[100x3]> <data.table[50x3]>
#> 
#> $Petal.Length
#>        id                  train_idx              validate_idx
#>    <char>                     <list>                    <list>
#> 1:  Fold1  1, 2, 8, 9,10,11,...[100]  3, 4, 5, 6, 7,12,...[50]
#> 2:  Fold2       2,3,4,5,6,7,...[100]  1,10,11,17,18,19,...[50]
#> 3:  Fold3       1,3,4,5,6,7,...[100]  2, 8, 9,14,15,20,...[50]
#>                  train           validate
#>                 <list>             <list>
#> 1: <data.table[100x3]> <data.table[50x3]>
#> 2: <data.table[100x3]> <data.table[50x3]>
#> 3: <data.table[100x3]> <data.table[50x3]>
#> 
# Returns a list where each element contains:
# - id: fold labels (Fold1, Fold2, Fold3)
# - train_idx / validate_idx: row indices of each fold
# - train / validate: training and validation subsets

# Example 2: Repeated cross-validation
split_cv(
  data = dt_split,      # Input list of split data
  v = 3,                # Set 3-fold cross-validation
  repeats = 2,          # Perform cross-validation twice
  seed = 123
)
#> $Sepal.Length
#>         id    id2                  train_idx              validate_idx
#>     <char> <char>                     <list>                    <list>
#> 1: Repeat1  Fold1  1, 2, 5, 7, 9,10,...[100]  3, 4, 6, 8,15,19,...[50]
#> 2: Repeat1  Fold2       3,4,5,6,7,8,...[100]  1, 2, 9,10,11,14,...[50]
#> 3: Repeat1  Fold3       1,2,3,4,6,8,...[100]  5, 7,12,13,16,17,...[50]
#> 4: Repeat2  Fold1  2, 4, 5, 6, 7,13,...[100]  1, 3, 8, 9,10,11,...[50]
#> 5: Repeat2  Fold2       1,2,3,4,6,8,...[100]  5, 7,13,14,17,21,...[50]
#> 6: Repeat2  Fold3       1,3,5,7,8,9,...[100]  2, 4, 6,15,18,22,...[50]
#>                  train           validate
#>                 <list>             <list>
#> 1: <data.table[100x3]> <data.table[50x3]>
#> 2: <data.table[100x3]> <data.table[50x3]>
#> 3: <data.table[100x3]> <data.table[50x3]>
#> 4: <data.table[100x3]> <data.table[50x3]>
#> 5: <data.table[100x3]> <data.table[50x3]>
#> 6: <data.table[100x3]> <data.table[50x3]>
#> 
#> $Sepal.Width
#>         id    id2                  train_idx              validate_idx
#>     <char> <char>                     <list>                    <list>
#> 1: Repeat1  Fold1  1, 2, 8, 9,10,11,...[100]  3, 4, 5, 6, 7,12,...[50]
#> 2: Repeat1  Fold2       2,3,4,5,6,7,...[100]  1,10,11,17,18,19,...[50]
#> 3: Repeat1  Fold3       1,3,4,5,6,7,...[100]  2, 8, 9,14,15,20,...[50]
#> 4: Repeat2  Fold1  1, 4, 5, 7, 9,10,...[100]  2, 3, 6, 8,14,20,...[50]
#> 5: Repeat2  Fold2       2,3,6,7,8,9,...[100]  1, 4, 5,12,17,21,...[50]
#> 6: Repeat2  Fold3       1,2,3,4,5,6,...[100]  7, 9,10,11,13,15,...[50]
#>                  train           validate
#>                 <list>             <list>
#> 1: <data.table[100x3]> <data.table[50x3]>
#> 2: <data.table[100x3]> <data.table[50x3]>
#> 3: <data.table[100x3]> <data.table[50x3]>
#> 4: <data.table[100x3]> <data.table[50x3]>
#> 5: <data.table[100x3]> <data.table[50x3]>
#> 6: <data.table[100x3]> <data.table[50x3]>
#> 
#> $Petal.Length
#>         id    id2                  train_idx              validate_idx
#>     <char> <char>                     <list>                    <list>
#> 1: Repeat1  Fold1       1,2,4,5,6,8,...[100]  3, 7, 9,15,22,23,...[50]
#> 2: Repeat1  Fold2       1,2,3,4,5,7,...[100]  6, 8,12,14,16,17,...[50]
#> 3: Repeat1  Fold3  3, 6, 7, 8, 9,12,...[100]  1, 2, 4, 5,10,11,...[50]
#> 4: Repeat2  Fold1  1, 2, 5, 8,10,12,...[100]  3, 4, 6, 7, 9,11,...[50]
#> 5: Repeat2  Fold2       3,4,5,6,7,8,...[100]  1, 2,12,14,20,21,...[50]
#> 6: Repeat2  Fold3       1,2,3,4,6,7,...[100]  5, 8,10,15,16,25,...[50]
#>                  train           validate
#>                 <list>             <list>
#> 1: <data.table[100x3]> <data.table[50x3]>
#> 2: <data.table[100x3]> <data.table[50x3]>
#> 3: <data.table[100x3]> <data.table[50x3]>
#> 4: <data.table[100x3]> <data.table[50x3]>
#> 5: <data.table[100x3]> <data.table[50x3]>
#> 6: <data.table[100x3]> <data.table[50x3]>
#> 
# Returns a list where each element contains:
# - id: repeat labels (Repeat1, Repeat2)
# - id2: fold labels (Fold1, Fold2, Fold3)
# - train_idx / validate_idx, train / validate

# Example 3: Stratified CV, indices only (memory friendly)
res <- split_cv(dt_split, v = 5, strata = "Species", seed = 1,
                materialize = FALSE)
# Rebuild the training set of fold 1 of the first dataset when needed
head(dt_split[[1]][res[[1]]$train_idx[[1]], ])
#>    Petal.Width Species value
#>          <num>  <fctr> <num>
#> 1:         0.2  setosa   5.1
#> 2:         0.2  setosa   4.9
#> 3:         0.2  setosa   4.7
#> 4:         0.2  setosa   4.6
#> 5:         0.2  setosa   5.0
#> 6:         0.4  setosa   5.4