Export Nested Data Structures with Hierarchical Directory Organization

Exports list-columns containing data.frame or data.table objects from a data.frame/data.table to txt or csv files, automatically constructing a hierarchical directory structure from non-nested columns. Exportable nested columns (those holding data.frame/data.table elements) are distinguished from non-exportable custom-object columns (e.g. fitted model objects); only the former are written to disk by default.

export_nest(
  data,
  by = NULL,
  cols = NULL,
  path = tempdir(),
  file_type = "txt",
  na = "NA",
  quote = FALSE,
  ...
)

Arguments

data

A data.frame or data.table containing at least one nested list-column. Must have one or more rows.

by

Optional character vector of column names used to build the hierarchical output directory structure. When NULL (default), all non-nested columns are used automatically.

cols

Optional character vector of nested column names to export. When NULL (default), all columns whose elements are data.frame/data.table objects are exported automatically; custom-object list-columns are reported and skipped. Specifying a non-data.frame column triggers a warning and that column is skipped.

path

Single character string specifying the root export directory. Defaults to tempdir(). Created recursively if it does not exist.

file_type

Either "txt" (tab-separated, default) or "csv" (comma-separated). Case-insensitive.

na

Single string written for missing values. Default "NA" (fwrite's own default, an empty field, breaks whitespace-delimited readers). Use e.g. "-9999" for DMU.

quote

Passed to fwrite. Default FALSE: with a non-empty na, fwrite's "auto" would quote every header and character field, which command-line breeding programs (HIBLUP, DMU, ...) cannot parse. Use "auto" if values may contain the separator.

...

Further arguments passed to fwrite (e.g. col.names = FALSE for software that expects no header).

Value

An invisible character vector of the files written (length 0 when nothing was written).

Details

Nested column classification (mutually exclusive):

• Exportable — every element inherits from data.frame or data.table.

• Non-exportable — empty lists or elements of any other class (e.g. model objects). Reported to the console; never written.

Directory layout: path / <group1_value> / <group2_value> / <nest_col_name>.<file_type>

Group values are sanitised (characters that are invalid in file names, such as slashes, colons or asterisks, become underscores), so they can never create extra directory levels. The grouping columns must identify every row uniquely; otherwise rows would overwrite each other's files and the function stops with an error listing the duplicated targets.

Performance notes:

• Row data is accessed via .subset2() (zero-copy column access) rather than data[i], eliminating per-row data.table allocation in the hot loop.

• All output directory paths are pre-computed in a single vectorised do.call(file.path, ...) call; only the unique directories are created.

• The field separator and output filenames are computed once before the loop.

See also

fwrite

Examples

# Example: Basic nested data export workflow
# A dedicated sub-folder of tempdir() keeps the clean-up safe
out_dir <- file.path(tempdir(), "mintyr_export_nest")

# Step 1: Create nested data structure
dt_nest <- w2l_nest(
  data = iris,              # Input iris dataset
  cols = 1:2,               # Columns to be nested
  by = "Species"            # Grouping variable
)

# Step 2: Export nested data to files
files <- export_nest(
  data = dt_nest,                    # Input nested data.table
  cols = "data",                     # Column containing nested data
  by = c("name", "Species"),         # Columns to create directory structure
  path = out_dir
)
#> [ export_nest ] Export complete. 6 file(s) written to: /tmp/RtmpBFC2vg/mintyr_export_nest
# Returns (invisibly) the paths of the written files
# Directory structure: out_dir/<name>/<Species>/data.txt
files
#> [1] "/tmp/RtmpBFC2vg/mintyr_export_nest/Sepal.Length/setosa/data.txt"    
#> [2] "/tmp/RtmpBFC2vg/mintyr_export_nest/Sepal.Length/versicolor/data.txt"
#> [3] "/tmp/RtmpBFC2vg/mintyr_export_nest/Sepal.Length/virginica/data.txt" 
#> [4] "/tmp/RtmpBFC2vg/mintyr_export_nest/Sepal.Width/setosa/data.txt"     
#> [5] "/tmp/RtmpBFC2vg/mintyr_export_nest/Sepal.Width/versicolor/data.txt" 
#> [6] "/tmp/RtmpBFC2vg/mintyr_export_nest/Sepal.Width/virginica/data.txt"  

# Clean up
unlink(out_dir, recursive = TRUE)