NEWS.md
validate_adg(): cv_rmse and cv_mae are now the median over pigs of each pig’s cross-validation error (they were the mean). A few pigs with an absurd extrapolation over a left-out block, mostly with poly3, could make the mean meaningless (on one large batch, tens of kg against a median of 2.5 kg, from four pigs with errors of thousands of kg). New column cv_share_over: the share of pigs with an error above the new argument cv_limit (default 10 kg), to see this risk separately.validate_outliers(), validate_adg(), validate_weight_fill(), validate_adfi() and validate_truncation() start with one line about the data and the design (pigs, methods, scenarios, repetitions), then show numbered phases ([1/3], …). The long phases, such as cleaning the remaining records again for every fold, truncation or mask, now have a progress bar with the count done and the ETA from the start; before they ran silently. Each phase ends with one line and its run time, and the function ends with the total time.*, e.g. the ADFI error per method and scenario.write_report(x, path) writes the result of run_pipeline() as a folder with index.html and the plots, to browse like a book: an overview page (pigs, mean ADG / ADFI / FCR, QC settings, pigs to check, stations) and one page per station with its growth and feed intake plots and a table of its pigs (ADG, ADFI, FCR, status, reason). Arrow keys turn the pages; a pig ID in the search box jumps to its station and highlights the pig. Works offline; printing gives one page per station.run_pipeline() numbers its steps ([2/6]) and shows how long each took; it starts with a rule and ends with a summary box (pigs, mean +/- SD of ADG, ADFI and FCR, pigs to check). print() of the result shows the same box. With verbose >= 1 the result is returned invisibly, as the box was already shown.verbose = 2 only, and counts of records or pigs lost are highlighted.All four single-step validators now hide days before cleaning, so the hidden data no longer decide what happens to the rest. Their numbers change, mostly at the stage ends; the methods of the pipeline are unchanged.
validate_adg(): cross-validation uses blocks of block_days consecutive days (new argument, default 7) as folds, instead of single random days whose neighbours stayed in the training data (block_days = 1 gives the old folds). In both cross-validation and truncation, all weight records of the hidden days are removed and the remaining records are cleaned again with clean_weight().validate_weight_fill(): the remaining records are cleaned again after the hidden days are removed.validate_adfi() gains weight (the result of clean_weight()): the weight records of the hidden days are hidden too, and the daily body weight used for the imputation is computed again from the remaining records. Without weight it works as before, with a warning.validate_outliers() gains reference = c("consensus", "qc"): by default the clean records are those that no compared method flags, so the method of qc is no longer favoured; the reference ADG comes from the clean records without any outlier method. reference = "qc" gives the old behaviour. $settings$n_clean gives the number of clean records.impute_method = "loess_pop": the days after the last recorded day are now filled by the pig’s own regression of daily intake on body weight. The population curve flattens at high weights and underestimated the late intake of pigs that leave early (by 43-129 g/d when 20-30 days were missing, in a check on 457 NEDAP pigs), which made them look more efficient. Long gaps and the days before the first recorded day keep the population curve.curve_extrap = "start", the new default: the growth curve is continued by the tangent line only before the first record, by the curve itself after the last. In the same check, the tangent at the start lowered the ADG error with 20 days missing at the start from 81 to 66 g/d and raised the FCR rank correlation from 0.82 to 0.86; at the end it gave better ADG but a worse FCR ranking. "curve" (before 1.4.0) and "tangent" remain available.validate_adg() and validate_weight_fill() compare the extrapolation of the settings by default (extrap = NULL).validate_truncation(visits, settings, scenarios, reps) checks the whole pipeline without information leaking back from hidden data: it removes all visits (weights and feed) of half of the pigs over the first ("late<d>"), last ("early<d>") or middle ("gap<d>") days of the stage, reruns clean_weight() to calc_fcr() for every QC setting, and compares ADG, ADFI and FCR with the full-data result: bias, rmse, mae_pct, pearson, spearman, slope and top_agree (agreement on the best 20%), with their SD over repetitions.validate_adfi() keeps the body weight of hidden feed days; weight records are cleaned with all data before hiding) and are optimistic at the stage ends; the documentation now says so, and the claim of 1.3.0 that "loess_pop" halved the error at both stage ends is corrected: it held at the start, not at the end.impute_method = "loess_pop": long gaps (more than long_gap_days) and the days before the first / after the last recorded day are filled from the population curve of daily intake on body weight, scaled to the intake level of the pig: dfi = r * f(bw). f is a quadratic regression over the recorded stage days of all pigs (of each breed with at least 10 pigs when there is a breed column), held constant beyond the weights it was fitted on; r is the pig’s recorded intake divided by f over its recorded days, shrunk towards 1 with weight n / (n + 10). Short gaps still use LOESS.
impute_method = "loess_lm"; "linear" is unchanged.calc_adfi() returns the curves in $pop_curve.validate_adfi() compares "loess_pop", "loess_lm" and "linear" by default; the population curve of each pig is fitted without that pig (leave-one-out).curve_extrap (default "curve", unchanged results): with "tangent", the growth curve is continued beyond the first / last record by a straight line with the robust slope of the records of the first / last 14 days (limited to 0.2-1.5 kg/d), instead of the polynomial itself. curve_used then reads e.g. "poly2 + tangent".validate_adg() and validate_weight_fill() gain extrap to compare both, e.g. validate_adg(w, extrap = c("curve", "tangent")). On the example data the tangent lowered the truncation error of poly2 in most scenarios (30 days hidden at the start: 53 to 38 g/d NEDAP, 344 to 173 g/d FIRE) but not in all (FIRE, 20 days at the start: 123 to 129 g/d).misread_max_visits (new, default 2) visits inside the visits at another station is a transponder misread. Misreads are kept as unidentified visits (identified = FALSE) instead of being deleted, so the feeder error checks and the unidentified feed of the feeder stay right."101>102>101").<id>_<station>. max_gap_days now flags pigs whose longest gap is longer (clean_weight(): low confidence, “test interrupted”).max_jump_kg now also flags the pig as low confidence.max_overlap_days is removed (qc_config() says what to use instead). The "stations" attribute has new columns n_transfers, misreads and longest_gap_days; split and misreads_removed are gone.calc_adg(): a pig whose growth curve never enters stage_range (all weights above or below it) gets adg_type = "none" instead of a “partial” ADG over its whole test. The end of the stage is the first time the curve rises through the upper limit, so an extrapolated curve that starts above it no longer hides the real crossing.calc_fcr(): days are put into weight classes, and the class gain is computed, from the growth curve. With the noisy daily weights, days near a class limit fell into the neighbouring class and inflated the gain (inner classes now 8.8-11.1 kg of 10 kg, before 4.5-11.4 kg).validate_outliers(cores > 1) no longer runs parallel fits inside parallel jobs (4 cores gave 16 processes), and matches records by unique keys.prep_visits(): a missing duration is computed from exit_time row by row (with the midnight correction), not only when the whole column is missing.cores > 1 on Windows stops with a clear message when the package is loaded with devtools::load_all() or the workers cannot load it.read_station() now returns a standard table: the same field names, types and units for NEDAP and FIRE (responder, location, entry_time, exit_time, duration, feed_intake and trough_* in g, weight in kg, topup, breed, source_file), ready for prep_visits().format argument and station_format(): a station format maps the columns of an export (by name, alternatives with |, or by position) to the standard fields, with units (kg/g/auto, s/min) and time formats. Date and clock time in two columns are joined (Date,Entry). station_format("fire", file = ) writes the built-in format as a tab-separated template to edit for other exports..xlsx, .xls) are read with readxl (in Suggests); the default pattern now matches CSV, TXT and Excel files.prep_visits() gains format for raw data already in R (e.g. from a database). Raw data such as the example datasets nedap and fire is mapped with the same built-in formats, so both routes give the same visits.read_station() keeps only the mapped fields, not all columns of the export, and fill is removed: optional fields missing in a file (e.g. Topup Amount) are filled with NA, required fields must be present in every file.A simpler, single method from the weight records to FCR. Breaking changes are marked with (breaking).
"casey", "national" and "legacy" are removed: qc_config() takes only named parameters (qc_config("national") gives an error that says what to do instead). Removed parameters: method, day_weight_min, daily_weight, rel_dev_max, daily_drop_min, daily_r2_min, max_feed_missing, curve_r2_min, max_daily_jump_kg, adg_method, curve_margin_kg, bin_min_gain_kg, cfi_r2_min; curve_model = "loess" is removed.weight_outlier defaults to c("rlm", "loess"): after the tracking from the end of the test, the remaining records are checked by quadratic robust regression (bisquare weight < 0.5) and by a robust LOESS smooth (more than 3 robust SDs away). Tracked records are no longer exempt from these methods.smooth_r2_min (default 0.99): a pig whose valid weights fit their LOESS smooth with a lower R² is flagged as low confidence. per_pig$r2, entry_weight and exit_weight now come from this smooth. r2_daily, feed_missing and max_daily_jump are removed.daily$weight_filled): the observed daily median; linear interpolation between recorded days; the growth curve (curve_model: "poly2", "poly1", "poly3" or "gompertz") before the first and after the last recorded day. daily$source is "observed", "interpolated" or "curve".r_squared. New columns n_observed_days, n_interpolated_days, n_curve_days; adg_curve stays for comparison. adg_source, lm_slope, lm_weight_from, lm_weight_to, curve_start and curve_end are removed.plot_adg() colours the daily weights by source and draws the ADG regression line.impute_method is "loess_lm" (default) or "linear". "loess_lm": gaps of up to long_gap_days (now 3) days between recorded days from a robust LOESS smooth, longer gaps and the stage ends from a per-pig regression of daily intake on body weight. "linear": linear interpolation inside, the same regression at the ends. The body weight of the regression, of the correction model and of fiv_max_by_weight is the filled daily weight.$cfi and plot_cfi() are removed.validate_weight_fill(): hides recorded daily weights of the stage (random days, blocks, start, end), fills them with linear interpolation or a growth curve and compares filled weights and ADG with the recorded ones.validate_outliers() compares method sets with or without the tracking ("track" in a set). validate_adg() measures the ADG of the filled daily weights. validate_adfi() compares "loess_lm" and "linear".plot_cfi() plots the cumulative feed intake of the national method against body weight, one plot per station: kept stage days, removed weight classes shaded, and the line whose slope is the FCR (dashed where extended by the regression). The national calc_fcr() result gains $cfi (every stage day with its class, whether it is kept and the cumulative intake).impute_method = "national", imputed days could get the wrong state (iii / elrp); the imputed values were right.qc_config("national") now implements the national FCR procedure (chapter 6 “FCR raw data processing”) as a whole pipeline (method = "national"), replacing the earlier, unverified LOESS-based preset. Body weights: range 20-140 kg; rlm (bisquare) weights <= 0.25 removed (rlm_cutoff); a day whose records all have rlm weights < 0.90 is removed (day_weight_min, qc_flag = "day_weight"); daily weight = mean weighted by the rlm weights (daily_weight = "weighted_mean"); pigs with a daily-weight R² (quadratic) < 0.995 (daily_r2_min), a test shorter than 40 days or more than 10% of days without feed (max_feed_missing) are removed. Feed errors and their correction are those of the package; missing days are imputed by LOESS for gaps of up to 3 days and by a regression of intake on body weight for longer gaps (impute_method = "national"). calc_fcr() then removes 10 kg classes with a gain below bin_min_gain_kg (6 kg) or an FCR outside table 5 (bin_fcr_limits), removes pigs with a removed class between kept ones, regresses cumulative feed intake on body weight (pigs with R² <= cfi_r2_min, 0.95, removed), extends it to the stage limits and computes FCR = (CFI at the upper limit - CFI at the lower limit) / stage width. New per_pig columns cfi_start, cfi_end, cfi_r2, cfi_slope, weight_first, weight_last, extrap_kg_start, extrap_kg_end, n_classes, n_classes_removed; by_bin$removed.adg_method: "lm" (new default) uses the slope of a linear regression of all valid weights between the stage start and end times (with its R² in r_squared); "curve" uses the stage weight difference / days of the growth curve (the earlier default). Both are reported (lm_slope, adg_curve); adg_source tells which one adg is; lm_weight_from / lm_weight_to give the weight range covered by the regression.adapt_days (default 7): weights of the first 7 days after a pig’s first visit are not used (qc_flag = "adaptation"). Their feed intake still counts when these days lie inside stage_range, and not when the pig is below it.r2_daily and feed_missing; records keep their rlm weight in rlm_w.clean_weight() now tracks the weight of each pig back from the end of the test (weight_track = TRUE, default in all three presets). The scale only counts a pig standing on it with all four feet, so readings are rarely too light; early in the test, when pigs are small, several of them fit into the station and readings are too heavy. At the end only one pig fits, so the last days are the most reliable. Starting from the median of the last 3 recorded days and the gain of the last 14 days, the weight is tracked back day by day; records more than track_tol (default 8%, at least 2 kg) from the expected weight are removed (qc_flag = "track"). The gain is updated from the kept weights of the next 10 days, so slow growth after entry is followed. Kept records are not overruled by the outlier methods, which are still used for pigs that are not tracked (less than 7 recorded days or a test shorter than 14 days) and with weight_track = FALSE.track_flag_share (default 40%) of the records are flagged as low confidence: a possible weighing fault such as a faulty load cell. New per_pig column track_removed.double_layer_gap, double_layer_min_share, double_layer_flag_weeks, entry_noise_days, entry_noise_iqr_kg, entry_noise_tol, entry_noise_flag_days; removed per_pig columns n_weeks, double_layer_weeks, entry_noise_days, entry_noise_kept.validate_outliers() compares "track" with the other methods (which run without tracking).curve_margin_kg (default NULL = all records): fit the growth curve only to the records from the last day below stage_range[1] - curve_margin_kg to the first day above stage_range[2] + curve_margin_kg (cut by time). Slow growth for weeks after entry then no longer bends the curve and pulls the stage start earlier. With 5 kg, the mean difference between ADG and lm_slope drops from 29 to 19 g/d on the NEDAP example. New per_pig columns curve_start and curve_end.clean_weight() detects double-layer weights before the outlier methods: within each week, records are split into two groups; when both are tight bands that differ by at least double_layer_gap (default 8%) of the upper weight and the smaller one holds at least double_layer_min_share (default 15%) of the records, the layer that does not continue the pig’s growth in the other weeks is removed (qc_flag = "double_layer"). Pigs with two layers in at least double_layer_flag_weeks (default 3) weeks are flagged as low confidence (possibly two pigs on one transponder). Before, when both layers were about equally large, the curve ran between them and the daily median jumped from one layer to the other. New per_pig columns n_weeks and double_layer_weeks. Set double_layer_gap = NULL to switch it off.
clean_weight() also handles unstable weights after entry: new pigs often show a wide smear of readings for days (pen mates’ weights mixed in while the pig itself hardly eats and may lose weight). Within entry_noise_days (default 21) after the first record, days whose interquartile range exceeds entry_noise_iqr_kg (default 3 kg) and 4 times the pig’s usual range are unstable; the period runs from entry to the last such day as long as at least half of the days are unstable. From the stable days after it, the weight is tracked back day by day and only records within entry_noise_tol (default 8%, at least 2 kg) of the next day’s weight are kept (qc_flag = "entry_noise"); the curve methods do not overrule the kept records. Pigs with a period longer than entry_noise_flag_days (default 14) or less than 30% of its records kept are flagged as low confidence. New per_pig columns entry_noise_days and entry_noise_kept.
lm_slope is now the regression on all valid records between the stage start and end times, instead of on the records whose weight lies inside stage_range. A dip below 30 kg after the start no longer leaves a few isolated early records in the regression.On the example data ADG changes by 2 g/d on average; pig 15961 (wide smear of readings for two weeks after entry) changes most, from 1211 to 1320 g/d.
calc_adg() failed with a cryptic data.table error when clean_weight() had removed every pig (e.g. a test period that is too short); it now says so and lists the most common reasons.calc_fcr() printed R warnings (no non-missing arguments to min) when no body weight record fell inside stage_range.cores > 1 on Linux/macOS, stopping a run (Esc / Ctrl+C, or an error) left the worker processes running in the background; they are now stopped.validate_outliers() adds artificial errors to clean body weight records and reports, for each outlier method, the share of errors found (sensitivity), the share of good records removed, precision and the effect on ADG. direction = "down" checks weights that are too low only.validate_adg() compares growth curve models by k-fold cross-validation on recorded days, and by how much ADG changes when the first or last days are hidden (late entry, early exit).validate_adfi() hides recorded feed intake days (random days, blocks, start or end of the stage), imputes them again and compares ADFI with the recorded value.seed, max_pigs (random sample of pigs for large data) and cores, show a progress bar, and return $summary plus per-pig details. New Chinese article “方法验证”.clean_weight(), calc_adg(), calc_adfi() and save_plots() is now also shown with cores > 1. On Linux/macOS one process per core is started and the bar moves while it works; on Windows the bar moves after each of 10 chunks.cores > 1, the function sent to the workers no longer carries the data of all pigs (faster start-up, less memory).The feed-intake correction model is fitted by a built-in REML routine for the random-intercept model (profiled likelihood over the variance ratio, computed from cross-products). Estimates, standard errors and p values are the same as nlme::lme() (difference < 0.001 g), but on 1.75 million visits calc_adfi() takes 9 s instead of 66 s. nlme is no longer a dependency.
daily_error_covariates() (used by calc_adfi()) is vectorised: 39 s to 1.6 s on the same data. Polynomial growth curves are evaluated directly instead of through predict(), and outlier reasons are built column-wise: clean_weight() 27 s to 18 s, calc_adg() 18 s to 12 s. Results are unchanged.
Plots are built from their own station’s data only. Before, every plot kept a reference to all pigs’ records (about 177 MB per plot when serialised on 1.75 million visits, now about 6 MB), which made parallel export and saveRDS() of plots slow.
save_plots() gains cores to save stations in parallel (fork on Linux/macOS, a PSOCK cluster on Windows) and shows a progress bar. Drawing takes about 1–1.5 s per station of 15 pigs.
verbose = 0 to hide them, or suppressMessages().add_breed() merges a breed (line) table from your own records into the visit table. prep_visits() keeps a breed column of the raw data and otherwise reminds you to add one. With breed, calc_adfi() adds it as a fixed effect to the correction model and calc_fcr() summarises traits by breed. run_pipeline() gains breeds.fiv_max_by_weight: the FIV-hi limit can depend on body weight (and breed) instead of the fixed 2000 g. The limit used for each visit is kept in column fiv_max. Example table: fiv_max_example().max_daily_jump_kg (off by default): flags pigs whose daily median weight changes too fast; clean_weight() always reports max_daily_jump.calc_fcr() returns $by_bin: feed, gain and FCR per body weight class (bin_width, default 10 kg). Inner classes covered by less than min_bin_coverage_kg (default 6 kg) of recorded weights, and segment FCR outside the optional bin_fcr_limits (example: bin_fcr_example()), flag the pig as low confidence.write_results() writes all result tables to CSV files named <farm>_<table>_<date>.csv (UTF-8 with BOM for Excel).clean_weight(), calc_adg(), calc_adfi() and run_pipeline() gain cores for parallel per-pig computation (base package parallel).calc_fcr(): the numbers in fcr_note (coverage, extrapolated days) could belong to another pig when several pigs were flagged. The flags themselves were right.import_nedap(), preprocess_data(), clean_weight_get(), adg_get(), adfi_get(), fcr_get(), adg_plot() and adfi_plot(). Use read_station(), prep_visits(), clean_weight(), calc_adg(), calc_adfi(), calc_fcr(), plot_adg() and plot_adfi(). Version 0.1.0 stays installable with pak::pak("tony2015116/tidypptsda@v0.1.0").qc_config() parameter and a guide to the three QC presets.dev/flat_tidypptsda.Rmd.read_station() -> prep_visits() -> clean_weight() -> calc_adg() -> flag_feed_errors() / calc_adfi() -> calc_fcr() -> plot_adg(), plot_adfi(), save_plots(); or all at once with run_pipeline().qc_config() (presets "casey", "national", "legacy").?tidypptsda-deprecated).adg_type (observed / extrapolated / partial / none).corrected_adg was removed).low_confidence instead of removing them.low_confidence when the growth curve or the feed intake is extrapolated over more than max_edge_extrap_days (default 15) at either end of the stage, instead of whenever any extrapolation happens.long_gap_days (default 7) are imputed with robust LOESS; impute_method = "cumulative_loess" and impute_r2_min were removed (the R2 of a cumulative curve is about 0.9 even for a constant intake, so it does not check anything). Use impute_method = "loess" instead.removed table with the step and the reason.