FINN 0.2.0

Changes that alter results

These change the numbers a fit or a simulation produces. Everything else in this release is new behaviour behind a new argument, a rename that keeps the old name, or documentation. Each row names the setting that restores the 0.1.0 result.

change new default restore 0.1.0
Offset of the trees likelihood at a simulated zero count fit(trees_offset = 0.01) trees_offset = NULL
Count gradients taken through the expected deaths and recruits, not through the draw fit(rb_gradient = TRUE) rb_gradient = FALSE
Exact binomial death draw instead of the moment-matched Gamma createProcess(mortality_draw = "binomial") mortality_draw = "gamma"
Exact recruit draw below lambda = 2 instead of the Gaussian approximation createProcess(regeneration_sampler = "refined") regeneration_sampler = "gaussian"
Limits of the mortality probability, per simulation step createProcess(mortality_floor = 0.001, mortality_ceiling = 0.999) 0.01 / 0.99
Recruitment capped by a Beverton-Holt term, with a prior on the cap finn()’s default regeneration process is regeneration_capped(cap = "all", prior = "max_obs_recruits", prior_lambda = 0.1) FINN::regeneration
Lower bound on the recruit dispersion, in the draw and the likelihood alike createProcess(theta_floor = 0.5) theta_floor = 0 or NULL
Growth enters mortality() in %/yr instead of as the raw rate createProcess(growth_scale = 0.01) growth_scale = 1
Binned competition takes its height quantiles over the live cohorts only createProcess(light_mask = TRUE) light_mask = FALSE
Additive recruitment floor createProcess(reg_floor = 1e-3) reg_floor = 0.2
A numeric loss_weights vector multiplies the baseline-scaled objective instead of replacing it fit(weight_scale = "auto") weight_scale = FALSE
predict() and simulateForest() leave requires_grad as they found it instead of switching every parameter on (#55) always -
resolveSiteIDs() numbers sites, patches and species by radix order instead of the collation locale’s always reproduces 0.1.0 where LC_COLLATE was already C
A hybrid growth process reads the growth predictors and a hybrid mortality process the mortality predictors (#72, B2) always - (0.1.0 and 0.2.0-rc fed each the other’s; identical where the two formulas are the same)
predict() / simulateForest() from bare ground without patches use the patch count of the fit instead of 100 patches not given: model$patches patches = 100

The sections The stem-count gradient, The death draw, The recruit draw, The mortality limits and The recruitment cap under The release in detail carry the measurements behind the first six rows.

The interface of finn() and fit()

Agreed for the release so that the two central functions read for an ecologist who does not know modelling terms. Every renamed or moved argument is still accepted under its old name, with a warning that names the new form, and gives the same result.

Documentation and data preparation

The release in detail

This release adds the model features developed for the FIA (US Forest Inventory and Analysis) and BWI (German Bundeswaldinventur) calibration projects. Every new argument defaults to the previous behaviour, with three exceptions that do not reproduce 0.1.0 and are marked in the table below: mortality(growth_scale = 0.01), createProcess(light_mask = TRUE) and createProcess(reg_floor = 1e-3). Two further exceptions repair the stem-count gradient and are on by default: fit(trees_offset = 0.01) and fit(rb_gradient = TRUE), both described under The stem-count gradient below. One further change reproduces 0.1.0 only on a machine whose LC_COLLATE was already C: resolveSiteIDs() now numbers sites, patches and species by radix order instead of the collation locale’s.

The two tables are the overview; the sections below them carry the detail.

The stem-count gradient

Two defects in how the trees term reached the parameters, both repaired here and both on by default (FINNverse/FINN#53).

The count likelihood added a constant to the predicted stem count so that log_prob stays finite where a species’ simulated count is exactly zero and the observed one is not. That constant was 1e-5 for "poisson" and 0.001 for "nbinom", which leaves the value finite and the gradient at y/offset – four orders of magnitude above every other term in the objective. Where the gradients are clipped per group, one such cell then rescales the whole mechanistic block. fit(trees_offset = ) exposes the constant and defaults to 0.01, on the scale of the predicted count. NULL restores the two historical values.

The simulator draws deaths and recruits and rounds them, and the gradient ran through both the reparameterised draw and the rounding’s straight-through surrogate, so the sampler’s noise reached the parameters. On the German inventory 97 % of the mortality gradient was draw noise (signal-to-noise 0.52 against 43 without it), which made a fit diverge or not depending on the host’s float kernels. fit(rb_gradient = TRUE) keeps the drawn, rounded count in the forward pass – a simulation is unchanged – and takes the derivative through n_at_risk * m and the expected recruitment instead. Against a 200-draw reference gradient the repaired direction agrees to a cosine of 1.000 in every parameter block at update_step = 1, where the unrepaired objective’s cosine is negative. FALSE restores the previous path.

Only fits that score trees are affected; a model scoring growth and mortality alone has no such term.

The death draw

mortality_draw gains "binomial", which is the new default, and "bernoulli".

"binomial" is the exact binomial, and it resolves by whether the draw has to carry a gradient. With rb_gradient = TRUE (also the new default) the drawn count is detached and the derivative comes from the expectation, so the draw is torch’s own binomial: exact in both moments at constant memory, and non-differentiable, which no longer matters. With rb_gradient = FALSE the gradient reaches the mortality parameters through the state – deaths enter trees, trees enters ba – so the draw must be differentiable, and it resolves to the moment-matched Gamma instead. Measured on that state path, the Gamma gives -39.36 against the expectation’s -39.32, while a straight-through Bernoulli gives -22.59 and the capped Poisson gives exactly 0.

Consequence worth knowing: under "binomial", switching rb_gradient changes the sampler as well as the gradient, so the two no longer simulate the same stand. Hold mortality_draw fixed to compare them.

"poisson" (a Bernoulli for one-tree cohorts, otherwise a capped Poisson) stays available. It is integer-valued and exact in expectation for small n*p, but its variance is n*p rather than n*p*(1-p) – measured 12.044 against 8.400 at n = 40, p = 0.3 – and it has no pathwise gradient. Combining it with rb_gradient = FALSE leaves the mortality parameters with exactly zero gradient, so that combination is now refused rather than silently fitted.

"bernoulli" is binomial_from_bernoulli(), repaired in this release (FINNverse/FINN#62): it returned exactly zero for every input, through an inverted trial mask and a reparameterised draw that was passed to a second torch_bernoulli(). It is exact in both moments and keeps a gradient path, but expands each cell to the largest cohort in the batch, so its memory grows with the largest trees value in the data.

The recruit draw

createProcess(regeneration_sampler = "refined") becomes the default. It samples exactly below lambda = 2 and drops the +0.001 offset and the 1e-4 clamp, which together imposed a recruitment floor of 0.0011 per cell-year – at a 0.0168 ha patch, 0.066 recruits per hectare and year per species. regeneration_sampler = "gaussian" restores the previous draw.

The mortality limits

The simulation clamps the per-cohort mortality probability before the death draw. That interval was hard-coded to [0.01, 0.99]; it is now createProcess(mortality_floor =, mortality_ceiling =) and defaults to [0.001, 0.999].

The bound is per simulation step, not per year, and the old floor is why a simulated low-mortality stand – a mature unthinned stand, a slow-growing species, an equilibrium run – lost stems the observations did not: at a 1 % floor every cohort is at risk at 1 % per step however low the fitted rate says it should be. The per-tree mortality loss was never bounded by it, so a fit could be correct while the trajectory it simulated was not.

The floor cannot simply be dropped: binomial_from_gamma()’s shape n*p/(1-p) degenerates as p -> 0. Measured with that draw it stays finite with an unbiased mean gradient down to 1e-4, but single-draw gradients grow heavy-tailed as n*p shrinks – the largest per-tree gradient is about ten times the old default’s at 1e-3 and about a hundred times at 1e-4 – so 1e-3 is the lowest tested value and is the new default.

The recruitment cap

regeneration_capped() replaces regeneration_saturation() (kept as a deprecated alias for one release) and becomes the default regeneration process. FINN::regeneration, unbounded as before, stays available and is now the opt-in.

Two things change beyond the name. The cap parameter reg_logK is created by FINN, so nothing has to be declared through custom_parameters any more; createProcess(cap = ) sets its resolution, "all" for one cap shared by every species and "species" for one per species.

And the cap carries a prior, because nothing else in the objective constrains it: a cap fitted freely runs to whatever the calibration window supports, which on a national inventory is far above anything observed. The penalty is prior_lambda * (log K - log prior)^2 per cap, the same form as the L2 pull of a staged calibration. prior = "max_obs_recruits" (the default) resolves it at the start of the first fit() from the largest observed recruitment rate per timestep, max(reg / period_length) – reg is the sum over an observation’s interval while the cap bounds one timestep’s mean – which is also the cap’s starting value; a later fit() continues from the trained cap rather than resetting it; a number sets the prior directly, on the same per-timestep scale; prior_lambda = 0 switches it off. With cap = "species" the prior is hierarchical – each species’ log cap is pulled towards the differentiable mean of all of them, so it penalises how far the species spread apart and leaves the overall level to the data.

The penalty is added after the loss_weights = "auto" baseline scaling, so prior_lambda means the same thing on every dataset.

What is new since 0.1.0

what where default
Offset of the trees count likelihood where a simulated count is exactly zero fit(trees_offset = ) 0.01; not 0.1.0 behaviour, NULL restores it
Exact binomial death draw instead of the moment-matched Gamma createProcess(mortality_draw = ), createHybrid() "binomial"; not 0.1.0 behaviour, "gamma" restores it
Exact recruit draw below lambda = 2 instead of the Gaussian approximation createProcess(regeneration_sampler = ), createHybrid() "refined"; not 0.1.0 behaviour, "gaussian" restores it
Lower and upper limit of the mortality probability, per simulation step createProcess(mortality_floor =, mortality_ceiling =), createHybrid() 0.001 / 0.999; not 0.1.0 behaviour, 0.01 / 0.99 restores it
Lower bound on the negative-binomial recruit dispersion, applied to the draw and the likelihood alike createProcess(theta_floor = ), createHybrid() 0.5; not 0.1.0 behaviour, 0 or NULL restores it
Recruit dispersion kept fixed at dispersion_parameter instead of estimated createProcess(optimizeTheta = ), createHybrid() TRUE (estimated, as in 0.1.0)
Recruitment capped by a Beverton-Holt term, with a prior on the cap finn()’s default regeneration process, createProcess(cap =, prior =, prior_lambda =) regeneration_capped(cap = "all", prior = "max_obs_recruits", prior_lambda = 0.1); not 0.1.0 behaviour, FINN::regeneration restores it
Count gradients taken through the expected deaths and recruits instead of through the draw fit(rb_gradient = ) TRUE; not 0.1.0 behaviour, FALSE restores it
Oregon FIA calibration sample as lazy-loaded datasets fia_obs_dt, fia_env_dt, fia_init_trees, fia_obs_test, fia_env_test, fia_init_test, fia_species_dt —
Per-response likelihood, renamed fit(loss_family = ) as loss had it; loss still accepted, deprecated
How a response is reduced before it is scored: "mean", "sum", "size_classes", "quantiles", "tree" fit(loss_aggregation = ) NULL (per-cell means)
Tree table feeding both per-tree routes fit(tree_data = ) NULL
Score growth and mortality on each simulated cohort that started as an observed tree fit(loss_resolution = ) "site"
Evaluate the observed-tree route on a fitted model predictTrees() new export
Per census and epoch: frac_killed, mean_kill_year, frac_ghost; and the per-tree predictions $tree_deaths, $tree_pred under loss_resolution = "tree"
Multiple shooting: rebuild the stand from the observed tree list at every anchor fit(reanchor = , anchor_cohorts = , anchor_mode = , anchor_min_dbh = ) FALSE, NULL, "replace", NULL
Reset the stand to the observation every year fit(teacher_forcing = ) FALSE
Drive the stand with the observed recruitment (reg / period_length per ha and year in every year of its interval, drawn by the recruit sampler) instead of the regeneration process, which is frozen and unscored for that call: stage 2 of a staged calibration fit(loss_recruit = ); simulateForest(recruits =, obs_recruits =), predict() "simulated", as in 0.1.0
Per-site aggregation windows for mixed remeasurement intervals automatic from period_length; override with model$period_mode "auto"
A non-finite loss term drops that term, or the whole timestep; occurrences counted fit(nonfinite = ), $nonfinite_counts "drop_term"
Set-up only, no training fit(epochs = 0) —
Per-predictor scaling: "auto", "identity", "0to1" or a function fit(env_autoscale = ) TRUE, which z-scores as before; the new strings are opt-in
Recruitment limited by conspecific adults regeneration_adult() new export
Conspecific adult density as an effect on recruitment, with a share f0 recruited without adults regeneration_adult_effect() new export
A process function declares and calibrates its own named parameters createProcess(custom_parameters = ) NULL
Additive recruitment floor, was hard-coded at 0.2. A default that does not reproduce 0.1.0 — see below createProcess(reg_floor = ) on the regeneration process 1e-3 (was 0.2)
Observation operator mapping predicted and observed recruitment per ha into the tallied-count space of the sampling design (applied under the "nbinom" regeneration likelihood only, #65) finn(recruit_obs_weight = ) 1.0
Score growth over the whole remeasurement interval finn(growth_rate = ) "annual"
Binned competition takes its height quantiles over the live cohorts only. A default that does not reproduce 0.1.0 — see below createProcess(light_mask = ) TRUE (was unmasked)
A cohort id identifies the initial cohort it came from, so a simulated cohort traces back to its tree simulateForest(return_cohorts = TRUE), $pred —
A numeric weights vector multiplies the baseline-scaled objective instead of replacing it. A default that does not reproduce 0.1.0 fit(loss_weights = ), fit(weight_scale = ) "auto"; weight_scale = FALSE restores the old meaning
Growth enters mortality() in %/yr. A default that does not reproduce 0.1.0 — existing code changes its answer mortality(growth_scale = ), overridable per model with createProcess(growth_scale = ) 0.01 (was the raw rate); growth_scale = 1 restores 0.1.0

What was broken

Most of these were silent. The last column says whether a result changes for someone who was affected.

defect results change?
fit(epochs = 0) ran two epochs (1:0) yes — it trained when asked not to
createProcess(upper = , lower = ) were overwritten with NULL, so every fit ran on the process defaults yes — a fit asking for tighter bounds ran unconstrained
createProcess(NN = ) was dropped inside create_nn() yes — the supplied network was not used
createProcess(growth_scale = ) was unvalidated: a bare NA divides by 1 and silently restores the 0.1.0 objective, 0 saturates mortality yes, and with nothing to signal it; now an error
A torch_load()ed model’s optimizer was reused although it no longer owned the model’s tensors yes — a crash (optim_ignite_*) or no training at all (optim_adam)
Observed-tree, size-class and quantile tensors of an earlier fit() were still evaluated in forward() yes — a stale term entered the loss
fit() set-up looked site and year up by linear scan, O(rows x sites) no — arrays bit-identical; ~20 min of set-up per fit at 32,686 sites
Observed-tree tensors were sized by patches, not by the patches the trees occupy no — light is per patch; predictTrees(patches = ) is now ignored
Per-tree growth and mortality were evaluated at the interval’s last year only, against interval observations yes, for the per-tree routes (new in 0.2.0)
predictTrees() returned rows in its own order, and failed on a model that had never been run yes — misaligned predictions
A tracked cohort the simulation killed left the scored set mid-window, so the scored sample was a function of the parameters yes, for loss_resolution = "tree" (new in 0.2.0)
A retained dead cohort, and any empty padding slot, moved the height quantiles of the binned competition and so the light of the living yes — masking is now the default; light_mask = FALSE keeps the old behaviour
torch_nanquantile over a patch with no live cohort returned NaN into the likelihood yes — the fit aborted; the quantiles are now 0
A per-tree term was scored under a weight taken from the per-tree null while the cell term ran yes, on data with a constant census period
With sites observed in different years, each response’s loss summed one mean per observation year, so a year with ten trees weighed as much as one with thousands yes, for period_mode = "per_site" — the loss is now one pooled mean per batch
period_mode = "constant" with a per-tree term was silently overridden, with a false reason yes — now an error naming both halves
Cohort ids were enumerated flat over the whole array, losing the tree each cohort came from no — the link was recoverable, just not without a positional formula
resolveSiteIDs() numbered sites, patches and species through LC_COLLATE, so the same tables gave different integer ids on a German and on a C machine yes, off a C locale — the species number indexes par_growth / par_mortality, so a parameter row belonged to another species
$tree_deaths counted an earlier window’s ghosts among this window’s kills, and recorded liveness before the death draw diagnostic only — frac_killed was identically 0 on a one-year window
$tree_pred was not reset by fit(), and $tree_deaths reported NaN for a census with no fate target diagnostic only
A U+039B and a U+00D7 in ?fit, a stray \cr, two dangling \link targets, a @param on the wrong function no — the LaTeX manual and two R CMD check WARNINGs

New features

Behaviour changes

Performance

Bug fixes

Documentation

FINN 0.1.0

First public release.

FINN is a differentiable forest gap model: a cohort-based dynamic vegetation model whose demographic processes (competition, growth, mortality, regeneration) can each be a mechanistic function, a neural network, or a mixture of the two, all calibrated end-to-end by gradient descent.

Highlights of the current interface:

See the vignettes for a guided tour: Introduction to FINN, Plausible succession from a handful of species, Preparing your data for FINN, Fitting FINN to forest inventory data, and Mortality.