Sindbad.Setup Module
Setup
The Setup module provides tools for setting up and configuring SINDBAD experiments and runs. It handles the creation of experiment configurations, model structures, parameters, and output setups, ensuring a streamlined workflow for SINDBAD simulations.
Purpose
This module is designed to produce the SINDBAD info object, which contains all the necessary configurations and metadata for running SINDBAD experiments. It facilitates reading configurations, building model structures, and preparing outputs.
Dependencies
Related (SINDBAD ecosystem)
ErrorMetrics: Metric type construction for cost options.TimeSamplers: Temporal setup helpers.OmniTools: Shared utilities (e.g.Table).
External (third-party)
CSV,JLD2,JSON: Configuration and persistence tooling.ConstructionBase,Dates,NaNStatistics: Core utilities used during setup.Infiltrator: Optional interactive debugging (reexported for convenience).
Internal (within Sindbad)
Sindbad.TypesSindbadTEM
Included Files
defaultOptions.jl: Default configuration knobs for optimization, sensitivity, and machine-learning routines referenced during setup.utilsSetup.jl: Shared setup helpers (validation, convenience utilities).generateCode.jl: Code-generation helpers used by development workflows.getConfiguration.jl: Read JSON/CSV configuration and normalize into internal settings representation.setupSimulationInfo.jl: Build the simulationinfoNamedTuple that downstream modules consume.setupTypes.jl: Instantiate SINDBAD types after parsing settings (time, land, metrics, optimization, etc.).setupPools.jl: Initialize land pools and state variables based on model structure/helpers/constants.updateParameters.jl: Apply metric feedback to parameter values (e.g., during optimization iterations).setupParameters.jl: Load parameter metadata (bounds/timescales/priors) and prepare arrays used by optimization/ML.setupModels.jl: Validate/order selected processes, wiring approaches to the overall run sequence.setupOutput.jl: Configure diagnostic/output arrays, filenames, and write schedules.setupParameterOptimization.jl: Collect optimizer-specific settings (algorithm, stopping criteria, restarts, etc.).setupHybridMachineLearning.jl: Hybrid ML configuration (fold definitions, feature sets, surrogate wiring).setupInfo.jl: Final assembly step integrating all previous pieces into theinfoobject exported to simulations.
Notes
The package re-exports several key packages (
Infiltrator,CSV,JLD2) for convenience, allowing users to access their functionality directly throughSetup.Designed to be modular and extensible, enabling users to customize and expand the setup process for specific use cases.
Functions
backScaleParameters
Sindbad.Setup.backScaleParameters Function
backScaleParameters(parameter_vector_scaled, parameter_table, <: ParameterScaling)Reverts scaling of parameters using a specified scaling strategy.
Arguments
parameter_vector_scaled: Vector of scaled parameters to be converted back to original scaleparameter_table: Table containing parameter information and scaling factorsParameterScaling: Type indicating the scaling strategy to be used::ScaleDefault: Type indicating scaling by initial parameter values::ScaleBounds: Type indicating scaling by parameter bounds::ScaleNone: Type indicating no scaling should be applied (parameters remain unchanged)
Returns
Returns the unscaled/actual parameter vector in original units. The input vector is not changed, so the vector of an optimizer can be passed as it is.
sourceCode
function backScaleParameters end
function backScaleParameters(parameter_vector_scaled, parameter_table, ::ScaleNone)
return parameter_vector_scaled
end
function backScaleParameters(parameter_vector_scaled, parameter_table, ::ScaleNone)
return parameter_vector_scaled
end
function backScaleParameters(parameter_vector_scaled, parameter_table, ::ScaleDefault)
parameter_vector_scaled = abs.(parameter_table.initial) .* parameter_vector_scaled
return parameter_vector_scaled
end
function backScaleParameters(parameter_vector_scaled, parameter_table, ::ScaleBounds)
ub = parameter_table.upper # upper bounds
lb = parameter_table.lower # lower bounds
return lb .+ (ub .- lb) .* parameter_vector_scaled
endcheckIOModelStructure
Sindbad.Setup.checkIOModelStructure Function
checkIOModelStructure(info; model_funcs=(:define, :precompute, :compute))Validates the input/output (I/O) structure of the models selected in info.models.forward, independently of any plotting/visualization backend. Runs getInOutModels internally so the checks below are always performed, whether or not a plot is ever produced.
Arguments
info: ANamedTuplecontaining experiment information, includinginfo.models.forward.model_funcs: The model functions to analyze, given in their execution order (default:(:define, :precompute, :compute)).:updateis intentionally excluded because it is not currently invoked anywhere in the SINDBAD run loop.
Conventions checked (warnings only, via @warn; never errors):
Single-writer convention: an output variable
land.<namespace>.<variable>may be written by more than one distinct model only ifnamespaceis:pools,:states, or:diagnostics. Any other namespace written by more than one model is flagged, since it indicates more than one model approach is competing to produce the same variable.Valid-input convention: a variable used as input must be one of:
a forcing (
:forcingnamespace),a
:pools/:states/:diagnosticsvariable (these are exempt from the ordering check:pools/statespersist across timesteps/iterations and are seeded from initial conditions, whilediagnosticsis commonly written by more than one model),the output of a strictly earlier model in
info.models.forward, orthe output of an earlier function of the same model, following
model_funcsorder.
Any other input is flagged as having no valid earlier producer, which typically indicates either a model-ordering bug or a genuinely undefined/orphaned variable.
Returns
nothing.
Code
function checkIOModelStructure(info; model_funcs=(:define, :precompute, :compute))
models = info.models.forward
model_names = [Symbol(nameof(supertype(typeof(m)))) for m in models]
# `getInOutModel`/`getInOutModels` (SindbadTEM.Utils) unconditionally println progress for
# every model/function combination; that's fine for interactive/ad hoc use but far too noisy
# to run on every experiment setup, so it's silenced here rather than in SindbadTEM itself.
in_out = redirect_stdout(devnull) do
getInOutModels(models, model_funcs)
end
func_rank = Dict(f => i for (i, f) in enumerate(model_funcs))
exempt_namespaces = (:pools, :states, :diagnostics)
# registry: (namespace, variable) => [(model_index, func_rank), ...]
producers = Dict{Tuple{Symbol,Symbol}, Vector{Tuple{Int,Int}}}()
for (m_i, m_name) in enumerate(model_names)
for f in model_funcs
for (ns, var) in in_out[m_name][f][:output]
push!(get!(producers, (ns, var), Tuple{Int,Int}[]), (m_i, func_rank[f]))
end
end
end
# convention 1: single-writer unless pools/states/diagnostics
for ((ns, var), locs) in producers
if ns ∉ exempt_namespaces
distinct_models = unique(first.(locs))
if length(distinct_models) > 1
writer_names = join(unique(model_names[m_i] for m_i in distinct_models), ", ")
@warn "I/O convention violation: `land.$(ns).$(var)` is output by more than one model ($writer_names). Only variables in `land.pools`/`land.states`/`land.diagnostics` may be written by multiple models."
end
end
end
# convention 2: every input must have a valid earlier producer
for (m_i, m_name) in enumerate(model_names)
for f in model_funcs
for (ns, var) in in_out[m_name][f][:input]
(ns === :forcing || ns ∈ exempt_namespaces) && continue
locs = get(producers, (ns, var), Tuple{Int,Int}[])
has_valid_producer = any(locs) do (p_m, p_f)
p_m < m_i || (p_m == m_i && p_f < func_rank[f])
end
if !has_valid_producer
@warn "I/O convention violation: `land.$(ns).$(var)` is used as input in `$(f)` of `$(m_name)` (model #$(m_i) of $(length(model_names))) but has no valid earlier producer. Inputs must be a forcing, a `pools`/`states`/`diagnostics` variable, an output of an earlier model, or an output of an earlier function of the same model (order: $(join(model_funcs, ", ")))."
end
end
end
end
return nothing
endcheckParameterBounds
Sindbad.Setup.checkParameterBounds Function
checkParameterBounds(p_names, parameter_values, lower_bounds, upper_bounds, _sc::ParameterScaling; show_info=false, model_names=nothing)Check and display the parameter bounds information for given parameters.
Arguments
p_names: Names or identifier of the parameters. Vector of strings.parameter_values: Default values of the parameters. Vector of Numbers.lower_bounds: Lower bounds for the parameters. Vector of Numbers.upper_bounds: Upper bounds for the parameters. Vector of Numbers._sc::ParameterScaling: Scaling Type for the parametersshow_info: a flag to display model parameters and their bounds. Boolean.model_names: Names or identifier of the approaches where the parameters are defined.
Returns
Displays a formatted output of parameter bounds information or returns an error when they are violated
sourceCode
function checkParameterBounds(p_names, parameter_values, lower_bounds, upper_bounds, _sc::ParameterScaling; p_units=nothing, show_info=false, model_names=nothing)
if show_info
print_info(checkParameterBounds, @__FILE__, @__LINE__, "checking Parameter Bounds")
if nameof(typeof(_sc)) == :ScaleNone
print_info(nothing, @__FILE__, @__LINE__, "→→→ no scaling applied. The values and bounds are original/input values, while their units, when provided, are scaled to match the model run time steps and may differ from the original units in the model when @timescale of the parameter is different from the model run time step.")
else
print_info(nothing, @__FILE__, @__LINE__, "→→→ $(nameof(typeof(_sc))) scaling applied. The values and bounds are scaled values, while their units, when provided, are scaled to match the model run time steps and may differ from the original units in the model when @timescale of the parameter is different from the model run time step. Check info.models.parameter_table for interpreting parameter values in original/input units.")
end
end
for (i,n) in enumerate(p_names)
in_range = checkInRange(n, parameter_values[i], lower_bounds[i], upper_bounds[i], show_info)
if !in_range
error("$(String(n)) => value=$(parameter_values[i]) [lower_bound=$(lower_bounds[i]), upper_bound=$(upper_bounds[i])] violates the parameter bounds requirement (lower_bound <= value <= upper_bound). Fix the bounds in the given model ($(model_names[i])) or in the parameters input to continue.")
end
if show_info
ps = String(n)
if !isnothing(model_names)
ps = "`$(String(model_names[i]))`.jl: `$(String(n))`"
end
units_str = ""
if !isnothing(p_units)
units_str = p_units[i] == "" ? "unitless" : "$(p_units[i])"
units_str = "(units: $(units_str))"
end
print_info(nothing, @__FILE__, @__LINE__, "$(ps) => $(parameter_values[i]) [$(lower_bounds[i]), $(upper_bounds[i])] $units_str", n_f=6)
end
end
endconvertParametersToNamedTuple
Sindbad.Setup.convertParametersToNamedTuple Function
convertParametersToNamedTuple(parameter_table, name_field::Symbol)
convertParametersToNamedTuple(parameter_table, names::Union{AbstractVector,Tuple})Convert a parameter table into a NamedTuple keyed by parameter name, so that a single row - and any of its columns - can be reached by dot-notation (e.g. nt.Model__a.value) instead of filtering the table. Each row is kept as a NamedTuple (not e.g. a Tuple of Pairs), since only NamedTuple supports this further .column chaining.
Arguments
parameter_table: a table of parameters (one row per parameter; anything supportinglength, integer row indexing, andgetpropertyfor named columns, e.g.Table).name_field: eithera
Symbolgiving the column ofparameter_tablewhose values are used as theNamedTuplekeys, ora
Vector/Tupleof names to use directly as keys, one per row ofparameter_table.
Keys that come out as AbstractStrings (e.g. from a name_full column such as "Model.a") have . replaced with __ so that they form valid Julia identifiers (:Model__a), since . cannot appear in a NamedTuple/dot-access field name.
Returns
A NamedTuple with one entry per row of parameter_table; each value is that row (itself a NamedTuple, keyed by parameter_table's column names), e.g. (name = "a", name_full = "Model.a", value = 1).
Errors
Throws an ArgumentError if the resulting keys are not unique - dot-access requires each key to resolve to exactly one row - naming the duplicated key(s) and the row indices that produced them, so a different name_field/names can be chosen instead.
Examples
julia> using Sindbad
julia> using TypedTables: Table
julia> t = Table(name=["a", "b"], name_full=["Model.a", "Model.b"], value=[1, 2]);
julia> nt = convertParametersToNamedTuple(t, :name_full);
julia> nt.Model__a
(name = "a", name_full = "Model.a", value = 1)
julia> nt.Model__a.value
1convertParametersToNamedTuple(parameter_table, group_field::Symbol, name_field::Symbol)Convert a parameter table into a two-level nested NamedTuple: first keyed by group_field (e.g. :model), then within each group by name_field (e.g. :name), so that a single column of a single parameter can be reached like nt.rootMaximumDepth.constant_frac_max_root_depth.default.
group_field, name_field, and (whichever of these are present) name_full, model_approach, approach_func are dropped from each leaf row: the path to a leaf (nt.<group>.<name>) already says which model/parameter it is, so repeating that (or the model's implementation choice) inside the leaf itself would be pure redundancy.
Arguments
parameter_table: a table of parameters (one row per parameter; anything supportinglength, integer row indexing,getproperty, andpropertynames, e.g.Table).group_field: column used for the outer key, e.g.:model.name_field: column used for the inner key (within each group), e.g.:name.
Keys that come out as AbstractStrings have . replaced with __, same as convertParametersToNamedTuple(parameter_table, name_field).
Within a group, name_field is not always unique on its own - two different approaches in the same group_field (e.g. two process types that both, correctly or by a supertype bug, land in the same :model group) can each declare a same-named parameter such as f_τ. When that happens and parameter_table has a model_approach column, the colliding keys are disambiguated the same way getUniqueVarNames disambiguates output variable names: only the rows that collide are rewritten as <name>__<model_approach>, so unambiguous names in the same group keep their plain form. If a model_approach column isn't available, or names still collide after that rewrite, an ArgumentError is thrown naming the duplicated key(s) and the row indices that produced them.
Returns
A NamedTuple of NamedTuples of NamedTuples: nt.<group>.<name> is that row, minus group_field/name_field/name_full/model_approach/approach_func. Assumes the (group_field, name_field) combination is unique per row (true for (:model, :name) in a SINDBAD parameter table); see above for what happens when it isn't.
Examples
julia> using Sindbad
julia> t = Table(model=["Model", "Model"], name=["a", "b"], value=[1, 2]);
julia> nt = convertParametersToNamedTuple(t, :model, :name);
julia> nt.Model.a
(value = 1,)
julia> nt.Model.a.value
1Code
function convertParametersToNamedTuple(parameter_table, name_field::Symbol)
return convertParametersToNamedTuple(parameter_table, getproperty(parameter_table, name_field))
end
function convertParametersToNamedTuple(parameter_table, names::Union{AbstractVector,Tuple})
n_rows = length(parameter_table)
length(names) == n_rows || throw(ArgumentError("convertParametersToNamedTuple: got $(length(names)) names for $(n_rows) rows in parameter_table; they must match one-to-one."))
keys_sym = _sanitizeNamedTupleKey.(names)
rows_by_key = Dict{Symbol,Vector{Int}}()
for (row_ind, key) in enumerate(keys_sym)
push!(get!(rows_by_key, key, Int[]), row_ind)
end
duplicated_keys = filter(kv -> length(kv.second) > 1, rows_by_key)
if !isempty(duplicated_keys)
dup_msg = join((" :$k <- rows $(row_inds)" for (k, row_inds) in duplicated_keys), "\n")
throw(ArgumentError("convertParametersToNamedTuple: names are not unique, so they cannot become NamedTuple fields (each key must map to exactly one row). Duplicated keys:\n$dup_msg\nPass a different name_field (or names) - e.g. a fully-qualified column such as `name_full` - to disambiguate."))
end
return NamedTuple{Tuple(keys_sym)}(Tuple(_concreteNamedTuple(parameter_table[row_ind]) for row_ind in 1:n_rows))
end
function convertParametersToNamedTuple(parameter_table, group_field::Symbol, name_field::Symbol)
n_rows = length(parameter_table)
group_values = getproperty(parameter_table, group_field)
name_values = getproperty(parameter_table, name_field)
disambiguator_values = hasproperty(parameter_table, :model_approach) ? getproperty(parameter_table, :model_approach) : nothing
exclude_fields = (group_field, name_field)
for redundant_field in (:name_full, :model_approach, :approach_func)
if redundant_field !== group_field && redundant_field !== name_field && hasproperty(parameter_table, redundant_field)
exclude_fields = (exclude_fields..., redundant_field)
end
end
rows_by_group = Dict{Symbol,Vector{Int}}()
for row_ind in 1:n_rows
push!(get!(rows_by_group, _sanitizeNamedTupleKey(group_values[row_ind]), Int[]), row_ind)
end
group_keys = Tuple(unique(_sanitizeNamedTupleKey.(group_values)))
group_nts = map(group_keys) do group_key
row_inds = rows_by_group[group_key]
name_keys = _uniqueNameKeysWithinGroup(group_key, row_inds, name_values, disambiguator_values)
NamedTuple{name_keys}(Tuple(_concreteNamedTuple(drop_namedtuple_fields(parameter_table[row_ind], exclude_fields)) for row_ind in row_inds))
end
return NamedTuple{group_keys}(group_nts)
endconvertRunFlagsToTypes
Sindbad.Setup.convertRunFlagsToTypes Function
convertRunFlagsToTypes(info)Converts model run-related flags from the experiment configuration into types for dispatch.
Arguments:
info: A NamedTuple containing the experiment configuration, including model run flags.
Returns:
- A NamedTuple
new_runwhere each flag is converted into a corresponding type instance.
Notes:
Flags are processed recursively:
If a flag is a
NamedTuple, its subfields are converted into types.If a flag is a scalar, it is directly converted into a type using
getTypeInstanceForFlags.
filter_nan_pixelsis true when the experiment does not set it.The resulting
new_runNamedTuple is used for type-based dispatch in SINDBAD's model execution.
Examples
julia> using Sindbad
julia> # Convert run flags to types
julia> # new_run = convertRunFlagsToTypes(info)Code
function convertRunFlagsToTypes(info)
new_run = (;)
# a pixel with invalid forcing cannot be simulated, so it is skipped unless the
# experiment turns filtering off
dr = merge((; filter_nan_pixels=true), deepcopy(info.settings.experiment.flags))
for pr in propertynames(dr)
prf = getfield(dr, pr)
prtoset = nothing
if isa(prf, NamedTuple)
st = (;)
for prs in propertynames(prf)
prsf = getfield(prf, prs)
st = set_namedtuple_field(st, (prs, getTypeInstanceForFlags(prs, prsf)))
end
prtoset = st
else
prtoset = getTypeInstanceForFlags(pr, prf)
end
new_run = set_namedtuple_field(new_run, (pr, prtoset))
end
return new_run
endcreateArrayofType
Sindbad.Setup.createArrayofType Function
createArrayofType(input_values, pool_array, num_type, indx, ismain, array_type::ModelArrayType)Creates an array or view of the specified type array_type based on the input values and configuration.
Arguments:
input_values: The input data to be converted or used for creating the array.pool_array: A preallocated array from which a view may be created.num_type: The numerical type to which the input values should be converted (e.g.,Float64,Int).indx: A tuple of indices used to create a view from thepool_array.ismain: A boolean flag indicating whether the main array should be created (true) or a view should be created (false).array_type: A type dispatch that determines the array type to be created:ModelArrayView: Creates a view of thepool_arraybased on the indicesindx.ModelArrayArray: Creates a new array by convertinginput_valuesto the specifiednum_type.ModelArrayStaticArray: Creates a static array (SVector) from theinput_values.
Returns:
- An array or view of the specified type, created based on the input configuration.
Notes:
When
ismainistrue, the function convertsinput_valuesto the specifiednum_type.When
ismainisfalse, the function creates a view of thepool_arrayusing the indicesindx.For
ModelArrayStaticArray, the function ensures that the resulting static array (SVector) has the correct type and length.
Examples
julia> using Sindbad
julia> # Create a view from a preallocated array
julia> # pool_array = rand(10, 10)
julia> # indx = (1:5,)
julia> # view_array = createArrayofType(nothing, pool_array, Float64, indx, false, ModelArrayView())
julia> # Create a new array with a specific numerical type
julia> # new_array = createArrayofType([1.0, 2.0, 3.0], nothing, Float64, nothing, true, ModelArrayArray())
julia> # Create a static array (SVector)
julia> # static_array = createArrayofType([1.0, 2.0, 3.0], nothing, Float64, nothing, true, ModelArrayStaticArray())Code
function createArrayofType end
function createArrayofType(input_values, pool_array, num_type, indx, ismain, ::ModelArrayView)
if ismain
num_type.(input_values)
else
@view pool_array[[indx...]]
end
end
function createArrayofType(input_values, pool_array, num_type, indx, ismain, ::ModelArrayView)
if ismain
num_type.(input_values)
else
@view pool_array[[indx...]]
end
end
function createArrayofType(input_values, pool_array, num_type, indx, ismain, ::ModelArrayArray)
return num_type.(input_values)
end
function createArrayofType(input_values, pool_array, num_type, indx, ismain, ::ModelArrayStaticArray)
input_typed = typeof(num_type(1.0)) === eltype(input_values) ? input_values : num_type.(input_values)
return SVector{length(input_values)}(input_typed)
# return SVector{length(input_values)}(num_type(ix) for ix ∈ input_values)
endcreateInitLand
Sindbad.Setup.createInitLand Function
createInitLand(pool_info, tem)Initializes the land state by creating a NamedTuple with pools, states, and selected models.
Arguments:
pool_info: Information about the pools to initialize.tem: A helper NamedTuple with necessary objects for pools and numbers.
Returns:
- A NamedTuple containing initialized pools, states, fluxes, diagnostics, properties, models, and constants.
Examples
julia> using Sindbad
julia> # Initialize land state for an experiment
julia> # init_land = createInitLand(pool_info, tem)Code
function createInitLand(pool_info, tem)
print_info(createInitLand, @__FILE__, @__LINE__, "creating Initial Land...")
init_pools = createInitPools(pool_info, tem.helpers)
initial_states = createInitStates(pool_info, tem.helpers)
out = (; fluxes=(;), pools=(; init_pools..., initial_states...), states=(;), diagnostics=(;), properties=(;), models=(;), constants=(;))
sortedModels = sort([_sm for _sm ∈ tem.models.selected_models.model])
for model ∈ sortedModels
out = set_namedtuple_field(out, (model, (;)))
end
return out
endcreateInitPools
Sindbad.Setup.createInitPools Function
createInitPools(info_pools::NamedTuple, tem_helpers::NamedTuple)Creates a NamedTuple with initial pool variables as subfields, used in land.pools.
Arguments:
info_pools: A NamedTuple containing pool information from the experiment configuration.tem_helpers: A NamedTuple containing helper information for numerical operations.
Returns:
- A NamedTuple with initialized pool variables.
Code
function createInitPools(info_pools::NamedTuple, tem_helpers::NamedTuple)
init_pools = (;)
for element ∈ propertynames(info_pools)
props = getfield(info_pools, element)
model_array_type = getfield(Types, to_uppercase_first(string(getfield(props, :arraytype)), "ModelArray"))()
var_to_create = getfield(props, :create)
initial_values = getfield(props, :initial_values)
for tocr ∈ var_to_create
input_values = deepcopy(getfield(initial_values, tocr))
init_pools = set_namedtuple_field(init_pools, (tocr, createArrayofType(input_values, Nothing[], tem_helpers.numbers.num_type, nothing, true, model_array_type)))
end
to_combine = getfield(getfield(info_pools, element), :combine)
if to_combine.docombine
combined_pool_name = to_combine.pool
zix_pool = getfield(props, :zix)
components = keys(zix_pool)
pool_array = getfield(init_pools, combined_pool_name)
for component ∈ components
if component != combined_pool_name
indx = getfield(zix_pool, component)
input_values = deepcopy(getfield(initial_values, component))
compdat = createArrayofType(input_values, pool_array, tem_helpers.numbers.num_type, indx, false, model_array_type)
init_pools = set_namedtuple_field(init_pools, (component, compdat))
end
end
end
end
return init_pools
endcreateInitStates
Sindbad.Setup.createInitStates Function
createInitStates(info_pools::NamedTuple, tem_helpers::NamedTuple)Creates a NamedTuple with initial state variables as subfields, used in land.states.
Arguments:
info_pools: A NamedTuple containing pool information from the experiment configuration.tem_helpers: A NamedTuple containing helper information for numerical operations.
Returns:
- A NamedTuple with initialized state variables.
Notes:
Extended from `createInitPools``
State variables are derived from the
state_variablesfield inmodel_structure.json.
Code
function createInitStates(info_pools::NamedTuple, tem_helpers::NamedTuple)
initial_states = (;)
for element ∈ propertynames(info_pools)
props = getfield(info_pools, element)
var_to_create = getfield(props, :create)
additional_state_vars = (;)
if hasproperty(props, :state_variables)
additional_state_vars = getfield(props, :state_variables)
end
initial_values = getfield(props, :initial_values)
model_array_type = getfield(Types, to_uppercase_first(string(getfield(props, :arraytype)), "ModelArray"))()
for tocr ∈ var_to_create
for avk ∈ keys(additional_state_vars)
avv = getproperty(additional_state_vars, avk)
Δtocr = Symbol(string(avk) * string(tocr))
vals = one.(getfield(initial_values, tocr)) * tem_helpers.numbers.num_type(avv)
newvals = createArrayofType(vals, Nothing[], tem_helpers.numbers.num_type, nothing, true, model_array_type)
initial_states = set_namedtuple_field(initial_states, (Δtocr, newvals))
end
end
to_combine = getfield(getfield(info_pools, element), :combine)
if to_combine.docombine
combined_pool_name = Symbol(to_combine.pool)
for avk ∈ keys(additional_state_vars)
avv = getproperty(additional_state_vars, avk)
Δ_combined_pool_name = Symbol(string(avk) * string(combined_pool_name))
zix_pool = getfield(props, :zix)
components = keys(zix_pool)
Δ_pool_array = getfield(initial_states, Δ_combined_pool_name)
for component ∈ components
if component != combined_pool_name
Δ_component = Symbol(string(avk) * string(component))
indx = getfield(zix_pool, component)
Δ_compdat = createArrayofType((one.(getfield(initial_values, component))) .* tem_helpers.numbers.num_type(avv), Δ_pool_array, tem_helpers.numbers.num_type, indx, false, model_array_type)
initial_states = set_namedtuple_field(initial_states, (Δ_component, Δ_compdat))
end
end
end
end
end
return initial_states
endcreateNestedDict
Sindbad.Setup.createNestedDict Function
createNestedDict(dict::AbstractDict)Creates a nested dictionary from a flat dictionary where keys are strings separated by dots (.).
Arguments:
dict::AbstractDict: A flat dictionary with keys as dot-separated strings.
Returns:
- A nested dictionary where each dot-separated key is converted into nested dictionaries.
Examples
julia> using Sindbad
julia> dict = Dict("a.b.c" => 2, "a.b.d" => 3)
Dict{String, Int64} with 2 entries:
"a.b.c" => 2
"a.b.d" => 3
julia> nested_dict = createNestedDict(dict)
Dict{Any, Any} with 1 entry:
"a" => OrderedDict{Any, Any}("b"=>OrderedDict{Any, Any}("c"=>2, "d"=>3))Code
function createNestedDict(dict::AbstractDict)
nested_dict = Dict()
for kii ∈ keys(dict)
key_list = split(kii, ".")
key_dict = Dict()
for key_index ∈ reverse(eachindex(key_list))
subkey = key_list[key_index]
if subkey == first(key_list)
subkey_name = subkey
else
subkey_name = subkey * string(key_index)
end
if subkey == last(key_list)
key_dict[subkey_name] = toOrderedDict(dict[kii])
else
if !haskey(key_dict, subkey_name)
# an OrderedDict, like the parsed json, so that a level that is new in
# the configuration is converted to a NamedTuple with the others
key_dict[subkey_name] = DataStructures.OrderedDict()
key_dict[subkey_name][key_list[key_index+1]] = key_dict[key_list[key_index+1]*string(key_index+1)]
else
tmp = Dict()
tmp[subkey_name] = key_dict[key_list[key_index+1]*string(key_index + 1)]
end
delete!(key_dict, key_list[key_index+1] * string(key_index + 1))
delete!(nested_dict, key_list[key_index+1] * string(key_index + 1))
end
nested_dict = deepMerge(nested_dict, key_dict)
end
end
return nested_dict
enddeepMerge
Sindbad.Setup.deepMerge Function
deepMerge(d::AbstractDict...) = merge(deepMerge, d...)
deepMerge(d...) = d[end]Recursively merges multiple dictionaries, giving priority to the last dictionary.
Arguments:
d::AbstractDict...: One or more dictionaries to merge.
Returns:
- A single dictionary with merged fields, where the last dictionary's values take precedence.
Examples
julia> using Sindbad
julia> dict1 = Dict("a" => Dict("b" => 1), "c" => 2)
Dict{String, Any} with 2 entries:
"a" => Dict("b" => 1)
"c" => 2
julia> dict2 = Dict("a" => Dict("d" => 3), "e" => 4)
Dict{String, Any} with 2 entries:
"a" => Dict("d" => 3)
"e" => 4
julia> merged = deepMerge(dict1, dict2)
Dict{String, Any} with 3 entries:
"a" => Dict{String, Any}("b" => 1, "d" => 3)
"c" => 2
"e" => 4Code
function deepMerge end
deepMerge(d::AbstractDict...) = merge(deepMerge, d...)
deepMerge(d...) = d[end]
"""
getRootDirs(local_root, sindbad_experiment)
Determines the root directories for the SINDBAD framework and the experiment.
# Arguments:
- `local_root`: The local root directory of the SINDBAD project.
- `sindbad_experiment`: The path to the experiment configuration file.
# Returns:
- A NamedTuple containing the root directories for the experiment, SINDBAD, and settings.
"""
function getRootDirs(local_root, sindbad_experiment)
# sindbad_root = join(split(local_root, path_separator)[1:(end-2)] |> collect, path_separator)
exp_base_path = dirname(sindbad_experiment)
root_dir = (; experiment=normalize_path_separator(local_root), settings=normalize_path_separator(exp_base_path))
# root_dir = (; experiment=local_root, sindbad=sindbad_root, settings=exp_base_path)
return root_dir
endfilterParameterTable
Sindbad.Setup.filterParameterTable Function
filterParameterTable(parameter_table::Table; prop_name::Symbol=:model, prop_values::Tuple{Symbol}=(:all,))Filters a parameter table based on a specified property and values.
Arguments
parameter_table::Table: The parameter table to filterprop_name::Symbol: The property to filter by (default: :model)prop_values::Tuple{Symbol}: The values to filter by (default: :all)
Returns
A filtered parameter table.
Examples
julia> using Sindbad
julia> # Filter parameter table by model
julia> # filtered_table = filterParameterTable(parameter_table; prop_name=:model, prop_values=(:gpp,))
julia> # Return all parameters
julia> # all_params = filterParameterTable(parameter_table; prop_values=:all)Code
function filterParameterTable(parameter_table::Table; prop_name::Symbol=:model, prop_values::Union{Vector{Symbol},Symbol}=:all)
if prop_values == :all
return parameter_table
elseif isa(prop_values, Symbol)
return filter(row -> getproperty(row, prop_name) == prop_values, parameter_table)
else
return filter(row -> getproperty(row, prop_name) in prop_values, parameter_table)
end
endgenerateSindbadApproach
Sindbad.Setup.generateSindbadApproach Function
generateSindbadApproach(model_name::Symbol, model_purpose::String, appr_name::Symbol, appr_purpose::String, n_parameters::Int; methods=(:define, :precompute, :compute, :update), force_over_write=:none)Generate a SINDBAD model and/or approach with code templates.
Due to risk of overwriting code, the function only succeeds if y|Y||Yes|Ya, etc., are given in the confirmation prompt. This function only works if the call is copy-pasted into the REPL and not evaluated from a file/line. See the example below for the syntax.
Description
The generateSindbadApproach function creates a SINDBAD model and/or approach by generating code templates for their structure, parameters, methods, and documentation. It ensures consistency with the SINDBAD framework and adheres to naming conventions. If the model or approach already exists, it avoids overwriting existing files unless explicitly permitted. The generated code includes placeholders for methods (define, precompute, compute, update) and automatically generates docstrings for the model and approach.
Note that the newly created approaches are tracked by changes in tmp_precompile_placeholder.jl in the Sindbad root. The new models/approaches are automatically included ONLY when REPL is restarted.
Arguments
model_name: The name of the SINDBAD model to which the approach belongs.model_purpose: A string describing the purpose of the model.appr_name: The name of the approach to be generated.appr_purpose: A string describing the purpose of the approach.n_parameters: The number of parameters required by the approach.methods: A tuple of method names to include in the approach (default:(:define, :precompute, :compute, :update)).force_over_write: A symbol indicating whether to overwrite existing files or types. Options are::none(default): Do not overwrite existing files or types.:model: Overwrite the model file and type.:approach: Overwrite the approach file and type.:both: Overwrite both model and approach files and types.
Returns
nothing: The function generates the required files and writes them to the appropriate directory.
Behavior
If the model does not exist, it generates a new model file with the specified
model_nameandmodel_purpose.If the approach does not exist, it generates a new approach file with the specified
appr_name,appr_purpose, andn_parameters.Ensures that the approach name follows the SINDBAD naming convention (
<model_name>_<approach_name>).Prompts the user for confirmation before generating files to avoid accidental overwrites.
Includes placeholders for methods (
define,precompute,compute,update) and generates a consistent docstring for the approach.
Example
# Generate a new SINDBAD approach for an existing model
generateSindbadApproach(:ambientCO2, "Represents ambient CO2 concentration", :constant, "Sets ambient CO2 as a constant", 1)
# Generate a new SINDBAD model and approach
generateSindbadApproach(:newModel, "Represents a new SINDBAD model", :newApproach, "Implements a new approach for the model", 2)
# Generate a SINDBAD model and approach with force_over_write
generateSindbadApproach(:newModel, "Represents a new SINDBAD model", :newApproach, "Implements a new approach for the model", 2; force_over_write=:both) # overwrite both model and approach
generateSindbadApproach(:newModel, "Represents a new SINDBAD model", :newApproach, "Implements a new approach for the model", 2; force_over_write=:approach) # overwrite just approach approachNotes
The function ensures that the generated code adheres to SINDBAD conventions and includes all necessary metadata and documentation.
If the model or approach already exists, the function does not overwrite the files unless explicitly confirmed by the user.
The function provides warnings and prompts to ensure safe file generation and minimize the risk of accidental overwrites.
Code
function generateSindbadApproach(model_name::Symbol, model_purpose::String, appr_name::Symbol, appr_purpose::String, n_parameters::Int; methods=(:define, :precompute, :compute, :update), force_over_write=:none)
was_model_created = false
was_approach_created = false
over_write_model = false
over_write_appr = false
if force_over_write == :none
@info "Overwriting of type and file is off. Only new objects will be created"
elseif force_over_write == :model
over_write_model = true
@warn "Overwriting of type and file for Model is permitted. Continue with care."
elseif force_over_write == :approach
over_write_appr = true
@warn "Overwriting of type and file for Approach is permitted. Continue with care."
elseif force_over_write == :both
@warn "Overwriting of both type and files for Model and Approach are permitted. Continue with extreme care."
over_write_model = true
over_write_appr = true
else
error("force_over_write can only be one of (:none, :both, :model, :approach)")
end
if !startswith(string(appr_name), string(model_name)*"_")
@warn "the name $(appr_name) does not start with $(model_name), which is against the SINDBAD model component convention. Using $(model_name)_$(appr_name) as the name of the approach."
appr_name = Symbol(string(model_name) *"_"* string(appr_name))
end
model_type_exists = model_name in nameof.(subtypes(LandEcosystem))
model_path = joinpath(split(pathof(SindbadTEM),"/SindbadTEM.jl")[1], "Processes", "$(model_name)", "$(model_name).jl")
model_path_exists = isfile(model_path)
appr_path = joinpath(split(pathof(SindbadTEM),"/SindbadTEM.jl")[1], "Processes", "$(model_name)", "$(appr_name).jl")
appr_path_exists = isfile(appr_path)
model_path_exists = over_write_model ? false : model_path_exists
model_exists = false
if model_type_exists && model_path_exists
@info "both model_path and model_type exist. No need to create the model."
model_exists = true
elseif model_type_exists && !model_path_exists
@warn "model_type exists but (model_path does not exist || force_over_write is enabled with :$(force_over_write)). If force_over_write is not enabled, fix the inconsistency by moving the definition of th type to the file itself."
elseif !model_type_exists && model_path_exists
@warn "model_path exists but model_type does not exist. Fix this inconsistency by defining the type in the file."
model_exists = true
else
@info "both model_type and (model_path do not exist || force_over_write is enabled with :$(force_over_write)). Model will be created."
end
if model_exists
@info "Not generating model "
else
@info "Generating a new model: $(model_name) at:\n$(appr_path)"
confirm_ = Base.prompt("Continue: y | n")
if startswith(confirm_, "y")
@info "Generating model code:"
m_string=generateModelCode(model_name, model_purpose)
mkpath(dirname(model_path))
@info "Writing model code:"
open(model_path, "w") do model_file
write(model_file, m_string)
end
@info "success: $(model_path)"
was_model_created = true
end
end
appr_exists = false
appr_type_exists = false
if hasproperty(SindbadTEM.Processes, model_name)
model_type = getproperty(SindbadTEM.Processes, model_name)
appr_types = nameof.(subtypes(model_type))
appr_type_exists = appr_name in appr_types
end
appr_path_exists = over_write_appr ? false : appr_path_exists
if appr_type_exists && appr_path_exists
@info "both appr_path and appr_type exist. No need to create the approach."
appr_exists = true
elseif appr_type_exists && !appr_path_exists
@warn "appr_type exists but (appr_path does not exist || force_over_write is enabled with :$(force_over_write))). If force_over_write is not enabled, fix this inconsistency by defining the type in the file itself."
elseif !appr_type_exists && appr_path_exists
@warn "appr_path exists but appr_type does not exist. Fix this inconsistency by defining the type in the file."
else
@info "both appr_type and (appr_path do not exist || force_over_write is enabled with :$(force_over_write)). Approach will be created."
end
if appr_exists
@info "Not generating approach."
else
appr_path = joinpath(split(pathof(SindbadTEM),"/SindbadTEM.jl")[1], "Processes", "$(model_name)", "$(appr_name).jl")
@info "Generating a new approach: $(appr_name) for existing model: $(model_name) at:\n$(appr_path)"
confirm_ = Base.prompt("Continue: y | n")
if startswith(confirm_, "y")
@info "Generating code:"
appr_string = generateApproachCode(model_name, appr_name, appr_purpose, n_parameters; methods=methods)
@info "Writing code:"
open(appr_path, "w") do appr_file
write(appr_file, appr_string)
end
@info "success: $(appr_path)"
was_approach_created = true
else
@info "Not generating approach file due to user input."
end
end
## append the tmp_precompile_placeholder file so that Sindbad is forced to precompile in the next run_helpers
if was_model_created || was_approach_created
# Specify the file path
file_path = joinpath(split(pathof(SindbadTEM),"/SindbadTEM.jl")[1], "tmp_precompile_placeholder.jl")
# The line you want to add
date = strip(read(`date +%d.%m.%Y`, String));
new_lines = []
if was_model_created
new_line = "# - $(date): created a model $model_path.\n"
push!(new_lines, new_line)
end
if was_approach_created
new_line = "# - $(date): created an approach $appr_path.\n"
push!(new_lines, new_line)
end
# Open the file in append mode
open(file_path, "a") do file
foreach(new_lines) do new_line
write(file, new_line)
end
end
end
return nothing
endgetAbsDataPath
Sindbad.Setup.getAbsDataPath Function
getAbsDataPath(info, data_path)Converts a relative data path to an absolute path based on the experiment directory.
Arguments:
info: The SINDBAD experiment information object.data_path: The relative or absolute data path.
Returns:
An absolute data path.
sourceCode
function getAbsDataPath(info, data_path)
if startswith(data_path, "http") || startswith(data_path, "s3://")
return data_path
end
if !isabspath(data_path)
d_data_path = getSindbadDataDepot(local_data_depot=data_path)
if data_path == d_data_path
data_path = joinpath(info.experiment.dirs.experiment, data_path)
else
data_path = joinpath(d_data_path, data_path)
end
end
return data_path
endgetConfiguration
Sindbad.Setup.getConfiguration Function
getConfiguration(sindbad_experiment::String; replace_info=Dict())Loads the experiment configuration from a JSON or JLD2 file.
Arguments:
sindbad_experiment::String: Path to the experiment configuration file.replace_info::Dict: A dictionary of fields to replace in the configuration.
Returns:
- A NamedTuple containing the experiment configuration.
Notes:
Supports both JSON and JLD2 formats.
If
replace_infois provided, the specified fields are replaced in the configuration.
Examples
julia> using Sindbad
julia> # Load configuration from JSON file
julia> # config = getConfiguration("experiment_config.json")
julia> # Load with configuration overrides
julia> # config = getConfiguration("experiment_config.json"; replace_info=Dict("output" => Dict("save_all" => true)))Code
function getConfiguration(sindbad_experiment::String; replace_info=Dict())
local_root = dirname(Base.active_project())
if !isabspath(sindbad_experiment)
sindbad_experiment = joinpath(local_root, sindbad_experiment)
end
roots = getRootDirs(local_root, sindbad_experiment)
roots = (; roots..., sindbad_experiment)
info = nothing
if endswith(sindbad_experiment, ".json")
info_exp = getExperimentConfiguration(sindbad_experiment; replace_info=replace_info)
info = readConfiguration(info_exp, roots.settings)
elseif endswith(sindbad_experiment, ".jld2")
#todo running from the jld2 file here still does not work because the loaded info is a named tuple and replacing the fields will not work due to issues with merge and creating a dictionary from nested namedtuple
# info = Dict(pairs(load(sindbad_experiment)["info"]))
info = load(sindbad_experiment)["info"]
else
error(
"sindbad can only be run with either a json or a jld2 data file. Provide a correct experiment file"
)
end
if !isempty(replace_info)
non_exp_dict = filter(x -> !startswith(first(x), "experiment"), replace_info)
if !isempty(non_exp_dict)
info = replaceInfoFields(info, non_exp_dict)
end
end
if !haskey(info["experiment"]["basics"]["config_files"], "optimization") && (info["experiment"]["flags"]["run_optimization"] || info["experiment"]["flags"]["calc_cost"])
@warn "The config files in experiment_json and changes in replace_info do not include optimization_json. But, the settings for flags to run_optimization [set as $(info["experiment"]["flags"]["run_optimization"])] and/or calc_cost [set as $(info["experiment"]["flags"]["calc_cost"])] are set to true. These flags will be set to false from now on. The experiment will not run as intended further downstream. Cannot run optimization or calculate cost without the optimization settings. "
info["experiment"]["flags"]["run_optimization"] = false
info["experiment"]["flags"]["calc_cost"] = false
end
new_info = DataStructures.OrderedDict()
new_info["settings"] = DataStructures.OrderedDict()
for (k,v) in info
new_info["settings"][k] = v
end
# @show keys(info)
if !endswith(sindbad_experiment, ".jld2")
infoTuple = dict_to_namedtuple(new_info)
end
infoTuple = (; infoTuple..., temp=(; experiment=(; dirs=roots)))
print_info_separator()
return infoTuple
# return info
endgetConstraintNames
Sindbad.Setup.getConstraintNames Function
getConstraintNames(optim::NamedTuple)Extracts observation and model variable names for optimization constraints.
Arguments:
optim: A NamedTuple containing optimization settings and observation constraints.
Returns:
- A tuple containing:
obs_vars: A list of observation variables used to calculate cost.optim_vars: A lookup mapping observation variables to model variables.store_vars: A lookup of model variables for which time series will be stored.model_vars: A list of model variable names.
Code
function getConstraintNames(optim::NamedTuple)
obs_vars = Symbol.(optim.observational_constraints)
model_vars = String[]
optim_vars = (;)
for v ∈ obs_vars
vinfo = getproperty(optim.observations.variables, v)
push!(model_vars, vinfo.model_full_var)
vf, vvar = Symbol.(split(vinfo.model_full_var, "."))
optim_vars = set_namedtuple_field(optim_vars, (v, tuple(vf, vvar)))
end
store_vars = getVariableGroups(model_vars)
return obs_vars, optim_vars, store_vars, model_vars
endgetCostOptions
Sindbad.Setup.getCostOptions Function
getCostOptions(optim_info::NamedTuple, vars_info, tem_variables, number_helpers, dates_helpers)Sets up cost optimization options based on the provided parameters.
Arguments:
optim_info: A NamedTuple containing optimization parameters and settings.vars_info: Information about variables used in optimization.tem_variables: Template variables for optimization setup.number_helpers: Helper functions or values for numerical operations.dates_helpers: Helper functions or values for date-related operations.
Returns:
- A NamedTuple containing cost optimization configuration options.
Notes:
- Configures temporal and spatial aggregation, cost metrics, and other optimization-related settings.
Examples
julia> using Sindbad
julia> # Setup cost options for optimization
julia> # cost_options = getCostOptions(optim_info, vars_info, tem_variables, number_helpers, dates_helpers)Code
function getCostOptions(optim_info::NamedTuple, vars_info, tem_variables, number_helpers, dates_helpers)
varlist = Symbol.(optim_info.observational_constraints)
time_aggrs = []
aggr_funcs = []
all_costs = map(varlist) do v
merge_namedtuple_prefer_nonempty(optim_info.observations.default_cost, getfield(getfield(optim_info.observations.variables, v), :cost_options))
end
all_options = []
push!(all_options, varlist)
prop_names = keys(all_costs[1])
props_to_keep = (:cost_metric, :spatial_weight, :cost_weight, :temporal_data_aggr, :aggr_obs, :aggr_order, :min_data_points, :spatial_data_aggr, :spatial_cost_aggr, :aggr_func,)
for (pn, prop) ∈ enumerate(props_to_keep)
prop_array = []
for vn ∈ eachindex(varlist)
sel_opt=all_costs[vn]
sel_value = sel_opt[prop]
if (sel_value isa Number) && !(sel_value isa Bool)
sel_value = number_helpers.num_type(sel_value)
elseif sel_value isa Bool
sel_value=getTypeInstanceForFlags(prop, sel_value, "Do")
elseif sel_value isa String && (prop == :cost_metric)
sel_value = getTypeInstanceForCostMetric(ErrorMetrics, sel_value)
elseif sel_value isa String && (prop ∈ (:aggr_order, :spatial_data_aggr))
sel_value = getTypeInstanceForCostMetric(Types, sel_value)
end
push!(prop_array, sel_value)
if prop == :temporal_data_aggr
push!(time_aggrs, sel_value)
end
if prop == :aggr_func
push!(aggr_funcs, sel_value)
end
end
if prop in props_to_keep
push!(all_options, prop_array)
end
end
mod_vars = vars_info.model
mod_field = [Symbol(split(_a, ".")[1]) for _a in mod_vars]
mod_subfield = [Symbol(split(_a, ".")[2]) for _a in mod_vars]
mod_ind = collect(1:length(varlist))
obs_sn = [i for i in mod_ind]
obs_ind = [i + 3 * (i - 1) for i in mod_ind]
mod_ind = [findfirst(s -> first(s) === mf && last(s) === msf, tem_variables) for (mf, msf) in zip(mod_field, mod_subfield)]
# map(cost_option_table) do cost_option
# # @show cost_option
# new_mod_ind = findfirst(s -> first(s) === cost_option.mod_field && last(s) === cost_option.mod_subfield, tem_variables)
# cost_option = Accessors.@set cost_option.mod_ind = new_mod_ind
# end
time_resamplers = map(enumerate(time_aggrs)) do (i, _aggr)
getCostTimeResampler(_aggr, dates_helpers, getAggrFunc(aggr_funcs[i]))
end
push!(all_options, obs_ind)
push!(all_options, obs_sn)
push!(all_options, mod_ind)
push!(all_options, mod_field)
push!(all_options, mod_subfield)
push!(all_options, time_resamplers)
all_props = [:variable, props_to_keep..., :obs_ind, :obs_sn, :mod_ind, :mod_field, :mod_subfield, :temporal_aggr]
return (; Pair.(all_props, all_options)...)
endgetDepthDimensionSizeName
Sindbad.Setup.getDepthDimensionSizeName Function
getDepthDimensionSizeName(vname::Symbol, info::NamedTuple)Retrieves the name and size of the depth dimension for a given variable.
Arguments:
vname: The variable name.info: A SINDBAD NamedTuple containing all information needed for setup and execution of an experiment.
Returns:
- A tuple containing the size and name of the depth dimension.
Notes:
- Validates the depth dimension against the
depth_dimensionsfield in the experiment configuration.
Code
function getDepthDimensionSizeName(v_full_pair, info::NamedTuple)
v_full_str = getVariableString(v_full_pair)
v_full_sym = Symbol(v_full_str)
v_name = split(v_full_str, '.')[end]
dim_size = 1
dim_name = v_name * "_idx"
tmp_vars = nothing
tmp_vars = info.settings.experiment.model_output.variables
if v_full_sym in keys(tmp_vars)
v_dim = tmp_vars[v_full_sym]
dim_size = 1
dim_name = v_name * "_idx"
if !isnothing(v_dim) && isa(v_dim, String)
dim_name = v_dim
end
if isnothing(v_dim)
dim_size = dim_size
elseif isa(v_dim, Int64)
dim_size = v_dim
elseif isa(v_dim, String)
if Symbol(v_dim) in keys(info.settings.experiment.model_output.depth_dimensions)
dim_size_K = getfield(info.settings.experiment.model_output.depth_dimensions, Symbol(v_dim))
if isa(dim_size_K, Int64)
dim_size = dim_size_K
elseif isa(dim_size_K, String)
dim_size = getPoolSize(info.temp.helpers.pools, Symbol(dim_size_K))
end
else
error(
"The output depth dimension for $(v_name) is specified as $(v_dim) but this key does not exist in depth_dimensions. Either add it to depth_dimensions or add a numeric value."
)
end
else
error(
"The depth dimension for $(v_name) is specified as $(typeof(v_dim)). Only null, integers, or string keys to depth_dimensions are accepted."
)
end
end
return dim_size, dim_name
endgetDepthInfoAndVariables
Sindbad.Setup.getDepthInfoAndVariables Function
getDepthInfoAndVariables(info, output_vars)Generates depth information and variable pairs for the output variables.
Arguments:
info: A SINDBAD NamedTuple containing experiment configuration.output_vars: A list of output variables.
Returns:
- A NamedTuple containing depth information and variable pairs.
Code
function getDepthInfoAndVariables(info, output_vars)
out_vars_pairs = Tuple(getVariablePair.(output_vars))
depth_info = map(out_vars_pairs) do vname_full
getDepthDimensionSizeName(vname_full, info)
end
output_info=(; info.output..., depth_info, variables=out_vars_pairs)
return output_info
endgetExperimentConfiguration
Sindbad.Setup.getExperimentConfiguration Function
getExperimentConfiguration(experiment_json::String; replace_info=Dict())Loads the basic configuration from an experiment JSON file.
Arguments:
experiment_json::String: Path to the experiment JSON file.replace_info::Dict: A dictionary of fields to replace in the configuration.
Returns:
- A dictionary containing the experiment configuration.
Code
function getExperimentConfiguration(experiment_json::String; replace_info=Dict())
parseFile = parsefile(experiment_json; dicttype=DataStructures.OrderedDict)
info = DataStructures.OrderedDict()
info["experiment"] = DataStructures.OrderedDict()
for (k, v) ∈ parseFile
info["experiment"][k] = v
end
if !isempty(replace_info)
exp_dict = filter(x -> startswith(first(x), "experiment"), replace_info)
if !isempty(exp_dict)
info = replaceInfoFields(info, exp_dict)
end
end
return info
endgetExperimentInfo
Sindbad.Setup.getExperimentInfo Function
getExperimentInfo(sindbad_experiment::String; replace_info=Dict())Loads and sets up the experiment configuration, saving the information and enabling debugging options if specified.
Arguments:
sindbad_experiment::String: Path to the experiment configuration file.replace_info::Dict: (Optional) A dictionary of fields to replace in the configuration.
Returns:
- A NamedTuple
infocontaining the fully loaded and configured experiment information.
Notes:
- The function performs the following steps:
Loads the experiment configuration using
getConfiguration.Sets up the experiment
infousingsetupInfo.Saves the experiment
infoifsave_infois enabled.Sets up a debug error catcher if
catch_model_errorsis enabled.Plots the IO structure of the selected model structure via
plotIOModelStructure.If a
cCycleBaseapproach is selected, plots its carbon flow matrix viaplotCarbonFlows.
Examples
julia> using Sindbad
julia> # Load experiment configuration
julia> # info = getExperimentInfo("experiment_config.json")
julia> # Load with configuration overrides
julia> # info = getExperimentInfo("experiment_config.json"; replace_info=Dict("output" => Dict("save_all" => true)))Code
function getExperimentInfo(sindbad_experiment::String; replace_info=Dict())
replace_info_text = isempty(replace_info) ? "none" : " $(Tuple(keys(replace_info)))"
print_info_separator()
print_info(getExperimentInfo, @__FILE__, @__LINE__, "loading experiment configurations", n_m=1)
print_info(nothing, @__FILE__, @__LINE__, "→→→ experiment_path: `$(sindbad_experiment)`", n_m=1)
print_info(nothing, @__FILE__, @__LINE__, "→→→ replace_info_fields: `$(replace_info_text)`", n_m=1)
info = getConfiguration(sindbad_experiment; replace_info=deepcopy(replace_info))
info = setupInfo(info)
print_info(getExperimentInfo, @__FILE__, @__LINE__, "checking IO structure of the selected model structure...", n_m=1)
checkIOModelStructure(info)
print_info(getExperimentInfo, @__FILE__, @__LINE__, "plotting IO signatures in the selected model structure...", n_m=1)
for model_func in (:define, :precompute, :compute,)
Base.moduleroot(@__MODULE__).Visualization.plotIOModelStructure(info, model_func)
end
carbon_cycle_index = findfirst(m -> nameof(supertype(typeof(m))) === :cCycleBase, info.models.forward)
if !isnothing(carbon_cycle_index)
print_info(getExperimentInfo, @__FILE__, @__LINE__, "plotting the carbon flow matrix of the selected carbon cycle model...", n_m=1)
Base.moduleroot(@__MODULE__).Visualization.plotCarbonFlows(info, info.models.forward[carbon_cycle_index])
end
saveInfo(info, info.helpers.run.save_info)
setDebugErrorCatcher(info.helpers.run.catch_model_errors)
return info
endgetGlobalAttributesForOutCubes
Sindbad.Setup.getGlobalAttributesForOutCubes Function
getGlobalAttributesForOutCubes(info)Generates global attributes for output cubes, including system and experiment metadata.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- A dictionary
global_attrcontaining global attributes such as:simulation_by: The user running the simulation.experiment: The name of the experiment.domain: The domain of the experiment.date: The current date.machine: The machine architecture.os: The operating system.host: The hostname of the machine.julia: The Julia version.
Notes:
- The function collects system information using Julia's
Sysmodule andversioninfo.
Code
function getGlobalAttributesForOutCubes(info)
os = Sys.iswindows() ? "Windows" : Sys.isapple() ? "macOS" : Sys.islinux() ? "Linux" : "unknown"
simulation_by = Sys.iswindows() ? ENV["USERNAME"] : ENV["USER"]
io = IOBuffer()
versioninfo(io)
str = String(take!(io))
julia_info = split(str, "\n")
# io = IOBuffer()
# Pkg.status("Sindbad", io=io)
# sindbad_version = String(take!(io))
global_attr = Dict(
"simulation_by" => simulation_by,
"experiment" => info.temp.experiment.basics.name,
"domain" => info.temp.experiment.basics.domain,
"date" => string(Date(now())),
# "SINDBAD" => sindbad_version,
"machine" => Sys.MACHINE,
"os" => os,
"host" => gethostname(),
"julia" => string(VERSION),
)
return global_attr
endgetLandVarDepthInfo
Sindbad.Setup.getLandVarDepthInfo Function
getLandVarDepthInfo(info, land, var_pairs)Gets the size and name of the depth dimension of each variable in var_pairs.
Arguments:
info: A NamedTuple containing experiment configuration and helper information.land: A core SINDBAD NamedTuple containing variables for a given time step.var_pairs: A tuple of field and subfield pairs of variables inland.
Returns:
- A tuple with one
(dim_size, dim_name)entry per variable.
Notes:
- A variable that is already an output variable keeps the depth information set in the output settings. Any other variable takes its size from its value in
land.
Code
function getLandVarDepthInfo(info, land, var_pairs)
depth_info = map(var_pairs) do v_full_pair
v_index = findfirst(x -> first(x) === first(v_full_pair) && last(x) === last(v_full_pair), info.output.variables)
dim_name = nothing
dim_size = nothing
if ~isnothing(v_index)
dim_name = last(info.output.depth_info[v_index])
dim_size = first(info.output.depth_info[v_index])
else
field_name = first(v_full_pair)
v_name = last(v_full_pair)
dim_name = string(v_name) * "_idx"
land_field = getproperty(land, field_name)
if hasproperty(land_field, v_name)
land_subfield = getproperty(land_field, v_name)
if isa(land_subfield, AbstractArray)
dim_size = length(land_subfield)
elseif isa(land_subfield, Number)
dim_size = 1
else
dim_size = 0
end
end
end
dim_size, dim_name
end
return depth_info
endgetNumberType
Sindbad.Setup.getNumberType Function
getNumberType(t)Retrieves the numerical type based on the input, which can be a string or a data type.
Arguments:
t: The input specifying the numerical type. Can be:A
Stringrepresenting the type (e.g.,"Float64","Int").A
DataTypedirectly specifying the type (e.g.,Float64,Int).
Returns:
- The corresponding numerical type as a
DataType.
Notes:
If the input is a string, it is parsed and evaluated to return the corresponding type.
If the input is already a
DataType, it is returned as-is.
Examples
julia> using Sindbad
julia> getNumberType("Float64")
Float64
julia> getNumberType(Float64)
Float64
julia> getNumberType("Int")
Int64Code
function getNumberType end
function getNumberType(t::String)
ttype = eval(Meta.parse(t))
return ttype
end
function getNumberType(t::String)
ttype = eval(Meta.parse(t))
return ttype
end
function getNumberType(t::DataType)
return t
endgetOptimizationParametersTable
Sindbad.Setup.getOptimizationParametersTable Function
getOptimizationParametersTable(parameter_table_all::Table, model_parameter_default, optimization_parameters)Creates a filtered and enhanced parameter table for optimization by combining input parameters with default model parameters with the table of all parameters in the selected model structure.
Arguments
parameter_table_all::Table: A table containing all model parametersmodel_parameter_default: Default parameter settings including distribution and a flag differentiating if the parameter is to be ML-parameter-learntoptimization_parameters: Parameters to be optimized, specified either as:::NamedTuple: Named tuple with parameter configurations::Vector: Vector of parameter names to use with default settings
Returns
A filtered Table containing only the optimization parameters, enhanced with:
is_ml: Boolean flag indicating if parameter uses machine learningdist: Distribution type for each parameterp_dist: Distribution parameters as an array of numeric values
Notes
Parameters can be specified using comma-separated strings for model.parameter pairs
For NamedTuple inputs, individual parameter configurations override model_parameter_default
The output table preserves the numeric type of the input parameters
Code
function getOptimizationParametersTable(parameter_table_all::Table, model_parameter_default, optimization_parameters)
parameter_list = []
parameter_keys = []
if isa(optimization_parameters, NamedTuple)
parameter_keys = keys(optimization_parameters)
else
parameter_keys = optimization_parameters
end
parameter_list = replaceCommaSeparatedParams(parameter_keys)
missing_parameters = filter(x -> !(x in parameter_table_all.name_full), parameter_list)
if !isempty(missing_parameters)
error("Model Inconsistency: $([missing_parameters...]) parameter(s) not found in the selected model structure. Check the model structure in model_structure.json to include the parameter(s) or change model_parameters_to_optimize in optimization.json to exclude the parameter(s).")
end
parameter_table_all_filtered = filter(row -> row.name_full in parameter_list, parameter_table_all)
num_type = typeof(parameter_table_all_filtered.default[1])
is_ml = parameter_table_all_filtered.is_ml
dist = parameter_table_all_filtered.dist
p_dist = parameter_table_all_filtered.p_dist
for (p_ind, p_key) ∈ enumerate(parameter_keys)
p_field = nothing
was_default_used = false
if isa(optimization_parameters, NamedTuple)
p_field = getproperty(optimization_parameters, p_key)
if isnothing(p_field)
p_field = model_parameter_default
was_default_used = true
end
else
was_default_used = true
p_field = model_parameter_default
end
if !isnothing(p_field)
if hasproperty(p_field, :is_ml)
ml_json = getproperty(p_field, :is_ml)
is_ml[p_ind] = ml_json
end
if hasproperty(p_field, :distribution)
nd_json = getproperty(p_field, :distribution)
dist[p_ind] = nd_json[1]
p_dist[p_ind] = [num_type.(nd_json[2])...]
end
end
end
return parameter_table_all_filtered
endgetParameterIndices
Sindbad.Setup.getParameterIndices Function
getParameterIndices(selected_models::LongTuple, parameter_table::Table)
getParameterIndices(selected_models::Tuple, parameter_table::Table)Retrieves indices for model parameters from a parameter table.
Arguments
selected_models::LongTuple: A long tuple of selected models::Tuple: A tuple of selected models
parameter_table::Table: Table containing parameter information
Returns
A Tuple of Pair of Name and Indices corresponding to the model parameters in the parameter table for selected models.
sourceCode
function getParameterIndices end
function getParameterIndices(selected_models::LongTuple, parameter_table::Table)
selected_models_tuple = to_tuple(selected_models)
return getParameterIndices(selected_models_tuple, parameter_table)
end
function getParameterIndices(selected_models::LongTuple, parameter_table::Table)
selected_models_tuple = to_tuple(selected_models)
return getParameterIndices(selected_models_tuple, parameter_table)
end
function getParameterIndices(selected_models::Tuple, parameter_table::Table)
r = (;)
tempvec = Pair{Symbol,Int}[]
for m in selected_models
r = (; r..., getModelParameterIndices(m, parameter_table, tempvec)...)
end
r
endgetParameterIndicesPerModel
Sindbad.Setup.getParameterIndicesPerModel Function
getParameterIndicesPerModel(parameter_to_index::NamedTuple, selected_models)Returns the indices of parameter_to_index with one element per model in the order of selected_models.
Arguments:
parameter_to_index: a NamedTuple of model approach names, each with a NamedTuple of parameter names and their index in the parameter vector, fromgetParameterIndicesselected_models: a tuple or long tuple of models
Returns:
- A tuple for a tuple of models, or a long tuple with the same chunks for a long tuple of models. Element k is
nothingwhen model k has no parameter inparameter_to_index, and otherwise the NamedTuple of its parameter names and their indices, e.g.,(εmax = 6,).
Notes:
parameter_to_indexis keyed by model name. Here the same indices are keyed by model position, sosetTypedParameterspairs model k with element k and looks up no model by name.
Code
function getParameterIndicesPerModel(parameter_to_index::NamedTuple, selected_models)
return map(selected_models) do model
model_indices = get(parameter_to_index, nameof(typeof(model)), (;))
isempty(model_indices) ? nothing : model_indices
end
endgetParameterInput
Sindbad.Setup.getParameterInput Function
getParameterInput(info)Returns the parameter input given as a json or a zarr/nc file in config_files.parameters, with every data_path made absolute.
Arguments:
info: the experiment info duringsetupInfo
Returns:
nothingwhen there is no parameters input, or when it is a csv file. A csv gives one global parameter vector and is applied bysetSpinupAndForwardModels.Otherwise a NamedTuple with the
default_parameter_mapand thevariablesof the input.
Notes:
The json is parsed by
readConfigurationlike every other config file. A zarr/nc file given directly is stored asdefault_parameter_map.data_pathwith novariables.No parameter file is opened here.
getParametersreads and checks the parameter variables when the forcing is loaded.
Code
function getParameterInput(info)
hasproperty(info.settings, :parameters) || return nothing
parameter_input = info.settings.parameters
parameter_input isa NamedTuple || return nothing
isempty(parameter_input) && return nothing
path_info = (; experiment=info.temp.experiment)
default_map = setAbsoluteDataPath(get(parameter_input, :default_parameter_map, (;)), path_info)
variables = map(v -> setAbsoluteDataPath(v, path_info), get(parameter_input, :variables, (;)))
return (; default_parameter_map=default_map, variables=variables)
endgetParameters
Sindbad.Setup.getParameters Function
getParameters(selected_models::Tuple, num_type, model_timestep; return_table=true)
getParameters(selected_models::LongTuple, num_type, model_timestep; return_table=true)Retrieves parameters for the specified models with given numerical type and timestep settings.
Arguments
selected_models: A collection of selected models::Tuple: as a tuple::LongTuple: as a long tuple
num_type: The numerical type to be used for parametersmodel_timestep: The timestep setting for the model simulationreturn_table::Bool=true: Whether to return results in table format
Returns
Parameters information for the selected models based on the specified settings.
Examples
julia> using Sindbad
julia> # Get parameters for selected models
julia> # param_table = getParameters(selected_models, Float64, model_timestep)Code
function getParameters end
function getParameters(selected_models::LongTuple, num_type, model_timestep; return_table=true, show_info=false)
selected_models = to_tuple(selected_models)
return getParameters(selected_models, num_type, model_timestep; return_table=return_table, show_info=show_info)
end
function getParameters(selected_models::LongTuple, num_type, model_timestep; return_table=true, show_info=false)
selected_models = to_tuple(selected_models)
return getParameters(selected_models, num_type, model_timestep; return_table=return_table, show_info=show_info)
end
function getParameters(selected_models::Tuple, num_type, model_timestep; return_table=true, show_info=false)
model_names_list = nameof.(typeof.(selected_models))
constrains = []
default = []
name = Symbol[]
model_approach = Symbol[]
timescale=String[]
for obj in selected_models
k_names = propertynames(obj)
push!(constrains, SindbadTEM.Processes.bounds(obj)...)
push!(default, [getproperty(obj, name) for name in k_names]...)
push!(name, k_names...)
push!(model_approach, repeat([nameof(typeof(obj))], length(k_names))...)
push!(timescale, SindbadTEM.Processes.timescale(obj)...)
end
# infer types by re-building
constrains = [c for c in constrains]
default = [d for d in default]
nbounds = length(constrains)
lower = [constrains[i][1] for i in 1:nbounds]
upper = [constrains[i][2] for i in 1:nbounds]
model = [Symbol(supertype(getproperty(SindbadTEM.Processes, m))) for m in model_approach]
model_str = string.(model)
name_full = [join((last(split(model_str[i], ".")), name[i]), ".") for i in 1:nbounds]
approach_func = [getfield(SindbadTEM.Processes, m) for m in model_approach]
model_prev = model_approach[1]
m_id = findall(x-> x==model_prev, model_names_list)[1]
model_id = map(model_approach) do m
if m !== model_prev
model_prev = m
m_id = findall(x-> x==model_prev, model_names_list)[1]
end
m_id
end
unts=[]
unts_ori=[]
for m in eachindex(name)
prm_name = Symbol(name[m])
appr = approach_func[m]()
p_timescale = SindbadTEM.Processes.timescale(appr, prm_name)
unit_factor = getUnitConversionForParameter(p_timescale, model_timestep)
lower[m] = lower[m] * unit_factor
upper[m] = upper[m] * unit_factor
if hasproperty(appr, prm_name)
p_unit = SindbadTEM.Processes.units(appr, prm_name)
push!(unts_ori, p_unit)
if ~isone(unit_factor)
p_unit = replace(p_unit, p_timescale => model_timestep)
end
push!(unts, p_unit)
else
error("$appr does not have a parameter $prmn")
end
end
# default = num_type.(default)
lower = map(lower) do low
if isa(low, Number)
low = num_type(low)
else
low = num_type.(low)
end
low
end
upper = map(upper) do upp
if isa(upp, Number)
upp = num_type(upp)
else
upp = num_type.(upp)
end
upp
end
# lower = num_type.(lower)
# upper = num_type.(upper)
timescale_run = map(timescale) do ts
isempty(ts) ? ts : model_timestep
end
checkParameterBounds(name, default, lower, upper, ScaleNone(), p_units=unts, show_info=show_info, model_names=model_approach)
is_ml = Array{Bool}(undef, length(default))
dist = Array{String}(undef, length(default))
p_dist = Array{Array{num_type,1}}(undef, length(default))
is_ml .= false
dist .= ""
p_dist .= [num_type[]]
output = (; model_id, name, initial=default, default,optimized=default, lower, upper, timescale_run=timescale_run, units=unts, timescale_ori=timescale, units_ori=unts_ori, model, model_approach, approach_func, name_full, is_ml, dist, p_dist)
output = return_table ? Table(output) : output
return output
endgetRestartVars
Sindbad.Setup.getRestartVars Function
getRestartVars(info, land)Collects the pools that are written to the restart file.
Arguments:
info: A SINDBAD NamedTuple with the pool helpers of the model structure.land: A core SINDBAD NamedTuple containing all variables for a given time step.
Returns:
- A tuple of
(:pools, name)pairs, one for each main pool, such ascEcoandTWS. For a main pool that is not inland.pools, its component pools that are inland.poolsare written instead.
Notes:
Only the main pools are saved when they exist, because each holds all of its component pools. The
SpinupInputspinup mode fills the component pools from the main pools.A warning is shown for a pool group without a main pool, because its pools are then missing from the restart file.
Code
function getRestartVars(info, land)
pool_helpers = info.helpers.pools
main_pools = keys(pool_helpers.vals.self)
for (group, group_info) ∈ pairs(info.pool_structure)
combine = hasproperty(group_info, :combine) ? group_info.combine : nothing
if isnothing(combine) || isempty(string(combine)) || Symbol(combine) ∉ main_pools
@warn "The pool group `$(group)` has no main pool, so its pools are not written to the restart file and cannot be set by `SpinupInput`."
end
end
restart_vars = []
for main ∈ main_pools
if hasproperty(land.pools, main)
push!(restart_vars, (:pools, main))
else
components = [c for c ∈ pool_helpers.all_components[main] if hasproperty(land.pools, c)]
print_info(getRestartVars, @__FILE__, @__LINE__, "the main pool `$(main)` is not in land.pools, so its components $(components) are written to the restart file instead")
append!(restart_vars, [(:pools, c) for c ∈ components])
end
end
return Tuple(restart_vars)
endgetSindbadDataDepot
Sindbad.Setup.getSindbadDataDepot Function
getSindbadDataDepot(; env_data_depot_var="SINDBAD_DATA_DEPOT", local_data_depot="../data")Retrieve the Sindbad data depot path.
Arguments
env_data_depot_var: Environment variable name for the data depot (default: "SINDBAD_DATA_DEPOT")local_data_depot: Local path to the data depot (default: "../data")
Returns
The path to the Sindbad data depot.
sourceCode
function getSindbadDataDepot(; env_data_depot_var="SINDBAD_DATA_DEPOT", local_data_depot="../data")
data_depot = isabspath(local_data_depot) ? local_data_depot : haskey(ENV, env_data_depot_var) ? ENV[env_data_depot_var] : local_data_depot
return data_depot
endgetSpinupSequenceWithTypes
Sindbad.Setup.getSpinupSequenceWithTypes Function
getSpinupSequenceWithTypes(seqq, helpers_dates)
getSpinupSequenceWithTypes(seqq, helpers_dates, aggregator_cache, spinup_models=nothing)Processes the spinup sequence and assigns types for temporal aggregators for spinup forcing.
Arguments:
seqq: The spinup sequence from the experiment configuration.helpers_dates: A NamedTuple containing date-related helpers.aggregator_cache: A Dict of already built aggregators, shared across the locations of a run.spinup_models: The model names and default spinup indices fromgetSpinupModelsInfo.
Returns:
- A processed spinup sequence with forcing types for temporal aggregators.
Code
function getSpinupSequenceWithTypes end
getSpinupSequenceWithTypes(seqq, helpers_dates) = getSpinupSequenceWithTypes(seqq, helpers_dates, Dict{Symbol,Any}())
function getSpinupSequenceWithTypes(seqq, helpers_dates, aggregator_cache, spinup_models=nothing)
seqq_typed = []
save_restart_flags = getSaveRestartFlags(seqq)
for (seq, save_restart) in zip(seqq, save_restart_flags)
mode_name, default_models = getSpinupModeName(seq["spinup_mode"])
spinup_mode = getTypeInstanceForNamedOptions(mode_name)
forcing_name = haskey(seq, "forcing") ? seq["forcing"] : nothing
if isnothing(forcing_name)
spinup_mode isa LoopTEM && error("The spinup step with `spinup_mode` $(seq["spinup_mode"]) has no forcing. A loop_TEM step runs the models over the forcing, so set its `forcing`, e.g., \"day_MSC\".")
forcing = :none
aggregator, aggregator_indices, n_timesteps, aggregator_type = nothing, Int[], 1, TimeIndexed()
else
forcing = Symbol(forcing_name)
aggregator, aggregator_indices, n_timesteps, aggregator_type = getSpinupAggregator(forcing_name, helpers_dates, aggregator_cache)
end
optns = haskey(seq, "options") ? seq["options"] : (;)
model_indices = getSpinupModelsIndex(seq, default_models, spinup_models)
sst = SpinupStepWithAggregator(forcing, seq["n_repeat"], n_timesteps, spinup_mode, optns, aggregator_indices, aggregator, aggregator_type, save_restart, model_indices);
push!(seqq_typed, sst)
end
return seqq_typed
end
function getSpinupSequenceWithTypes(seqq, helpers_dates, aggregator_cache, spinup_models=nothing)
seqq_typed = []
save_restart_flags = getSaveRestartFlags(seqq)
for (seq, save_restart) in zip(seqq, save_restart_flags)
mode_name, default_models = getSpinupModeName(seq["spinup_mode"])
spinup_mode = getTypeInstanceForNamedOptions(mode_name)
forcing_name = haskey(seq, "forcing") ? seq["forcing"] : nothing
if isnothing(forcing_name)
spinup_mode isa LoopTEM && error("The spinup step with `spinup_mode` $(seq["spinup_mode"]) has no forcing. A loop_TEM step runs the models over the forcing, so set its `forcing`, e.g., \"day_MSC\".")
forcing = :none
aggregator, aggregator_indices, n_timesteps, aggregator_type = nothing, Int[], 1, TimeIndexed()
else
forcing = Symbol(forcing_name)
aggregator, aggregator_indices, n_timesteps, aggregator_type = getSpinupAggregator(forcing_name, helpers_dates, aggregator_cache)
end
optns = haskey(seq, "options") ? seq["options"] : (;)
model_indices = getSpinupModelsIndex(seq, default_models, spinup_models)
sst = SpinupStepWithAggregator(forcing, seq["n_repeat"], n_timesteps, spinup_mode, optns, aggregator_indices, aggregator, aggregator_type, save_restart, model_indices);
push!(seqq_typed, sst)
end
return seqq_typed
endgetTypeInstanceForCostMetric
Sindbad.Setup.getTypeInstanceForCostMetric Function
getTypeInstanceForCostMetric(mode_name::String)Retrieves the type instance for a given cost metric based on its name.
Arguments:
mode_name::String: The name of the cost metric (e.g.,"RMSE","MAE").
Returns:
- An instance of the corresponding cost metric type.
Notes:
The function converts the cost metric name to a type by capitalizing the first letter of each word and removing underscores.
The type is retrieved from the
SindbadMetricsmodule and instantiated.Used for dispatching cost metric calculations in SINDBAD.
Examples
julia> using Sindbad, ErrorMetrics
julia> getTypeInstanceForCostMetric(ErrorMetrics, "MSE")
MSE()
julia> getTypeInstanceForCostMetric(ErrorMetrics, "NSE")
NSE()Code
function getTypeInstanceForCostMetric(source_module::Module, option_name::String)
opt_ss = to_uppercase_first(option_name)
struct_instance = getfield(source_module, opt_ss)()
return struct_instance
endgetTypeInstanceForFlags
Sindbad.Setup.getTypeInstanceForFlags Function
getTypeInstanceForFlags(option_name::Symbol, option_value, opt_pref="Do")Generates a type instance for boolean flags based on the flag name and value.
Arguments:
option_name::Symbol: The name of the flag (e.g.,:run_optimization,:save_info).option_value: A boolean value (trueorfalse) indicating the state of the flag.opt_pref::String: (Optional) A prefix for the type name. Defaults to"Do".
Returns:
- An instance of the corresponding type:
If
option_valueistrue, the type name is prefixed withopt_pref(e.g.,DoRunOptimization).If
option_valueisfalse, the type name is prefixed withopt_pref * "Not"(e.g.,DoNotRunOptimization).
Notes:
The function converts the flag name to a string, capitalizes the first letter of each word, and appends the appropriate prefix (
DoorDoNot).The resulting type is retrieved from the
Setupmodule and instantiated.This is used for type-based dispatch in SINDBAD's model execution.
Examples
julia> using Sindbad
julia> # Get type instance for a flag set to true
julia> # flag_type = getTypeInstanceForFlags(:run_optimization, true)
julia> # Returns: DoRunOptimization()
julia> # Get type instance for a flag set to false
julia> # flag_type = getTypeInstanceForFlags(:run_optimization, false)
julia> # Returns: DoNotRunOptimization()Code
function getTypeInstanceForFlags(option_name::Symbol, option_value, opt_pref="Do")
opt_s = string(option_name)
structname = to_uppercase_first(opt_s, opt_pref)
if !option_value
structname = to_uppercase_first(opt_s, opt_pref*"Not")
end
struct_instance = getfield(Setup, structname)()
return struct_instance
endgetTypeInstanceForNamedOptions
Sindbad.Setup.getTypeInstanceForNamedOptions Function
getTypeInstanceForNamedOptions(option_name)Retrieves a type instance for a named option based on its string or symbol representation. These options are mainly within the optimization and temporal aggregation.
Arguments:
option_name: The name of the option, provided as either aStringor aSymbol.
Returns:
- An instance of the corresponding type from the
Setupmodule.
Notes:
If the input is a
Symbol, it is converted to aStringbefore processing.The function capitalizes the first letter of each word in the option name and removes underscores to match the type naming convention.
This is used for type-based dispatch in SINDBAD's configuration and execution.
Temporal aggregation settings are resolved by
get_TimeSamplerfrom TimeSamplers.jl instead. It accepts the same snake case names, which it prefixes withTime, and also period specs such as"8-day_msc".
Examples
julia> using Sindbad
julia> # Get type instance for a named option
julia> # option_type = getTypeInstanceForNamedOptions("metric_sum")
julia> # Returns: MetricSum() type instance
julia> #
julia> # Examples of conversions:
julia> # - "cost_metric": "NSE_inv" → NSEInv type
julia> # - "temporal_data_aggr": "month_anomaly" → MonthAnomaly typeCode
function getTypeInstanceForNamedOptions end
function getTypeInstanceForNamedOptions(option_name::String)
opt_ss = to_uppercase_first(option_name)
struct_instance = getfield(Setup, opt_ss)()
return struct_instance
end
function getTypeInstanceForNamedOptions(option_name::String)
opt_ss = to_uppercase_first(option_name)
struct_instance = getfield(Setup, opt_ss)()
return struct_instance
end
function getTypeInstanceForNamedOptions(option_name::Symbol)
return getTypeInstanceForNamedOptions(string(option_name))
endparameterVariableAttributes
Sindbad.Setup.parameterVariableAttributes Function
parameterVariableAttributes(parameter_table, p_index)Returns the metadata of one parameter from a parameter table, used as the attributes of its variable in a parameter file and as its entry in the params.json.
Arguments:
parameter_table: a table of SINDBAD model parameters, e.g., the optimized parametersp_index: the row of the parameter in the table
Returns:
- An ordered dictionary with one entry per column of the table, except the value columns.
Notes:
Each key is the name of the table column it comes from.
Empty strings are left out, because some file formats cannot store them. A missing
timescale_runis read back as no timescale, which needs no unit conversion.is_mlis stored as an integer, because netCDF has no boolean attributes.
Code
function parameterVariableAttributes(parameter_table, p_index)
p_row = parameter_table[p_index]
attributes = DataStructures.OrderedDict{String,Any}()
attributes["model"] = string(p_row.model)
attributes["name"] = string(p_row.name)
attributes["name_full"] = string(p_row.name_full)
attributes["model_approach"] = string(p_row.model_approach)
for p_field ∈ (:units, :units_ori, :timescale_run, :timescale_ori, :dist)
hasproperty(p_row, p_field) || continue
p_value = getproperty(p_row, p_field)
if !ismissing(p_value) && !isempty(string(p_value))
attributes[string(p_field)] = string(p_value)
end
end
for p_field ∈ (:default, :initial, :lower, :upper)
attributes[string(p_field)] = Float64(getproperty(p_row, p_field))
end
if hasproperty(p_row, :is_ml)
attributes["is_ml"] = Int(p_row.is_ml)
end
if hasproperty(p_row, :p_dist) && p_row.p_dist isa AbstractVector && !isempty(p_row.p_dist)
attributes["p_dist"] = Float64.(collect(p_row.p_dist))
end
return attributes
endparameterVariableName
Sindbad.Setup.parameterVariableName Function
parameterVariableName(model, name)Returns the name of the variable of a parameter in a parameter file. It is the model and the parameter name joined by a double underscore, e.g., fAPAR__k_extinction.
Notes:
- A model structure has one approach per model, so the name is unique even when the same parameter name is used by several models.
perturbParameters
Sindbad.Setup.perturbParameters Function
perturbParameters(x::AbstractVector, lower::AbstractVector, upper::AbstractVector, percent_range::Tuple{Float64,Float64}=(0.0, 0.1))Modify each element of vector x by a random percentage within percent_range, while ensuring the result stays within the bounds defined by lower and upper vectors.
Arguments
x: Vector to modifylower: Vector of lower boundsupper: Vector of upper boundspercent_range: Tuple of (min_percent, max_percent) for random modification (default: (0.0, 0.1))
Returns
- Modified vector
x(modified in-place)
Example
x = [1.0, 2.0, 3.0]
lower = [0.5, 1.5, 2.5]
upper = [1.5, 2.5, 3.5]
modify_within_bounds!(x, lower, upper, (0.0, 0.1)) # Modify by 0-10%Code
function perturbParameters(x::AbstractVector, lower::AbstractVector, upper::AbstractVector; percent_range::Tuple{Float64,Float64}=(0.0, 0.1))
@assert length(x) == length(lower) == length(upper) "Vectors must have the same length"
@assert all(lower .<= upper) "Lower bounds must be less than or equal to upper bounds"
@assert all(lower .<= x .<= upper) "Initial values must be within bounds"
@assert percent_range[1] >= 0.0 "Minimum percent must be non-negative"
@assert percent_range[2] >= percent_range[1] "Maximum percent must be greater than or equal to minimum percent"
min_pct, max_pct = percent_range
for i in eachindex(x)
# Generate random percentage within range
pct = min_pct + rand() * (max_pct - min_pct)
# Calculate new value
new_val = x[i] * (1 + pct)
# Ensure value stays within bounds
x[i] = clamp(new_val, lower[i], upper[i])
end
return x
endpoolNames
Sindbad.Setup.poolNames Function
poolNames(configuration)Return the sub pool names of one carbon pool configuration, as a single tuple in the same order as helpers.pools.components.cEco. Named by type, symbol, string, or instance, like poolStructure/poolAliases/cFlowEdges.
Code
function poolNames(configuration)
sub_pool_names, _ = generatedPoolNames(poolStructure(configuration))
return Tuple(sub_pool_names)
endprepGradientParameterUpdater
Sindbad.Setup.prepGradientParameterUpdater Function
prepGradientParameterUpdater(parameter_table, selected_models)Returns a GradientParameterUpdater for selected_models and the parameters in parameter_table, to pass to updateModels as the parameter_updater.
Code
function prepGradientParameterUpdater(parameter_table, selected_models)
parameter_to_index = getParameterIndices(selected_models, parameter_table)
return GradientParameterUpdater(parameter_table, getParameterIndicesPerModel(parameter_to_index, selected_models))
endprepParameterUpdater
Sindbad.Setup.prepParameterUpdater Function
prepParameterUpdater(parameter_table, selected_models)Returns a ParameterUpdater for selected_models and the parameters in parameter_table, to pass to updateModels as the parameter_updater.
Arguments:
parameter_table: the table of the parameters to setselected_models: a tuple or long tuple of models, or a vector of them with one element per location. All elements of a vector have the same model structure, so the first one is used.
Code
function prepParameterUpdater(parameter_table, selected_models)
parameter_to_index = getParameterIndices(selected_models, parameter_table)
return ParameterUpdater(parameter_table, getParameterIndicesPerModel(parameter_to_index, selected_models))
endreadConfiguration
Sindbad.Setup.readConfiguration Function
readConfiguration(info_exp::AbstractDict, base_path::String)Reads the experiment configuration files (JSON or CSV) and returns a dictionary.
The parameters config file can also be a zarr or netCDF parameter file. It is then stored as a parameter input with only its data_path, see getParameterInput.
Arguments:
info_exp::AbstractDict: The experiment configuration dictionary.base_path::String: The base path for resolving relative file paths.
Returns:
- A dictionary containing the parsed configuration files.
Code
function readConfiguration(info_exp::AbstractDict, base_path::String)
info = DataStructures.OrderedDict()
print_info(readConfiguration, @__FILE__, @__LINE__, "reading configuration files")
for (k, v) ∈ info_exp["experiment"]["basics"]["config_files"]
config_path = joinpath(base_path, v)
print_info(nothing, @__FILE__, @__LINE__, "→→→ `$(k)` ::: `$(config_path)`")
info_exp["experiment"]["basics"]["config_files"][k] = config_path
if endswith(v, ".json")
tmp = parsefile(config_path; dicttype=DataStructures.OrderedDict)
info[k] = removeComments(tmp) # remove on first level
elseif endswith(v, ".csv")
prm = CSV.File(config_path)
tmp = Table(prm)
info[k] = tmp
elseif k == "parameters" && isParameterFile(v)
# a parameter file given directly uses all its variables, and their
# settings are read from the attributes in getParameters
default_map = DataStructures.OrderedDict{String,Any}("data_path" => config_path)
info[k] = DataStructures.OrderedDict{String,Any}("default_parameter_map" => default_map, "variables" => DataStructures.OrderedDict{String,Any}())
end
end
# rm second level
for (k, v) ∈ info
if typeof(v) <: Dict
ks = keys(info[k])
tmpDict = DataStructures.OrderedDict()
for ki ∈ ks
tmpDict[ki] = removeComments(info[k][ki])
end
info[k] = tmpDict
end
end
info["experiment"] = info_exp["experiment"]
return info
endresolvePoolStructure
Sindbad.Setup.resolvePoolStructure Function
resolvePoolStructure(info::NamedTuple)Normalize every element of model_structure.pools into one shape: its own block plus an aliases field, with anything else the block carried (state_variables included) passed through untouched.
An element's value may be:
absent: structure and aliases both come from the configuration the selected approach for that element's process declares (
poolConfigurationForApproach); an element with none this way is simply not in the result.a String: structure and aliases both come from the configuration it names.
a NamedTuple: the block supplies the structure verbatim. Aliases come from its own
aliaseskey, or otherwise from the process's approach configuration, in which case the block's pool names must match it.
Code
function resolvePoolStructure(info::NamedTuple)
pools = info.settings.model_structure.pools
resolved = (;)
elements = unique(vcat(collect(propertynames(pools)), collect(propertynames(POOL_ELEMENT_PROCESS))))
for element ∈ elements
if !hasproperty(pools, element)
configuration = poolConfigurationForApproach(info, element)
isnothing(configuration) && continue
structure = poolStructure(configuration)
aliases = poolAliases(configuration)
else
block = getproperty(pools, element)
if isa(block, AbstractString)
configuration = try
poolConfigurationFor(block)
catch e
error("pools.$(element) is set to `$(block)`: $(e.msg)")
end
structure = poolStructure(configuration)
if isnothing(structure)
error("pools.$(element) resolves to the configuration `$(nameof(configuration))`, " *
"which declares no poolStructure.")
end
aliases = poolAliases(configuration)
else
structure = block
aliases = hasproperty(block, :aliases) ? block.aliases : (;)
if !hasproperty(block, :aliases)
configuration = poolConfigurationForApproach(info, element)
if !isnothing(configuration)
aliases = poolAliases(configuration)
if !isempty(aliases)
block_sub_pool_names, _ = generatedPoolNames(block)
config_sub_pool_names, _ = generatedPoolNames(poolStructure(configuration))
if Set(block_sub_pool_names) != Set(config_sub_pool_names)
error("pools.$(element) is written out in full, and its aliases are " *
"taken from `$(nameof(configuration))`, but the two disagree on " *
"which pools exist. Only in the block: " *
"$(join(sort(String.(setdiff(block_sub_pool_names, config_sub_pool_names))), ", ")). " *
"Only in the configuration: " *
"$(join(sort(String.(setdiff(config_sub_pool_names, block_sub_pool_names))), ", ")). " *
"Declare an `aliases` key in the block, or match the configuration.")
end
end
end
end
end
end
validatePoolStructure(element, structure, aliases)
resolved = set_namedtuple_field(resolved, (Symbol(element), (; structure..., aliases=aliases)))
end
return resolved
endscaleParameters
Sindbad.Setup.scaleParameters Function
scaleParameters(parameter_table, <: ParameterScaling)Scale parameters from the input table using default scaling factors.
Arguments
parameter_table: Table containing parameters to be scaledParameterScaling: Type indicating the scaling strategy to be used::ScaleDefault: Type indicating scaling by default values::ScaleBounds: Type parameter indicating scaling by parameter bounds::ScaleNone: Type parameter indicating no scaling should be applied
Returns
Scaled parameters and their bounds according to default scaling factors
sourceCode
function scaleParameters end
function scaleParameters(parameter_table, _sc::ScaleNone)
init = copy(parameter_table.initial)
ub = copy(parameter_table.upper) # upper bounds
lb = copy(parameter_table.lower) # lower bounds
checkParameterBounds(parameter_table.name, init, lb, ub, _sc, p_units=parameter_table.units, show_info=true, model_names=parameter_table.model_approach)
return (init, lb, ub)
end
function scaleParameters(parameter_table, _sc::ScaleNone)
init = copy(parameter_table.initial)
ub = copy(parameter_table.upper) # upper bounds
lb = copy(parameter_table.lower) # lower bounds
checkParameterBounds(parameter_table.name, init, lb, ub, _sc, p_units=parameter_table.units, show_info=true, model_names=parameter_table.model_approach)
return (init, lb, ub)
end
function scaleParameters(parameter_table, _sc::ScaleDefault)
init = abs.(copy(parameter_table.initial))
ub = copy(parameter_table.upper ./ init) # upper bounds
lb = copy(parameter_table.lower ./ init) # lower bounds
init = parameter_table.initial ./ init
checkParameterBounds(parameter_table.name, init, lb, ub, _sc, p_units=parameter_table.units, show_info=true, model_names=parameter_table.model_approach)
return (init, lb, ub)
end
function scaleParameters(parameter_table, _sc::ScaleBounds)
init = copy(parameter_table.initial)
ub = copy(parameter_table.upper) # upper bounds
lb = copy(parameter_table.lower) # lower bounds
init = (init - lb) ./ (ub - lb)
lb = zero(lb)
ub = one.(ub)
checkParameterBounds(parameter_table.name, init, lb, ub, _sc, p_units=parameter_table.units, show_info=true, model_names=parameter_table.model_approach)
return (init, lb, ub)
endsetHybridInfo
Sindbad.Setup.setHybridInfo Function
setHybridInfo(info::NamedTuple)Processes and sets up the hybrid experiment information in the experiment configuration.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- The updated
infoNamedTuple with hybrid experiment information added.
Code
function setHybridInfo(info::NamedTuple)
print_info(setHybridInfo, @__FILE__, @__LINE__, "setting info for hybrid machine-learning + TEM experiment...", n_m=1)
# hybrid_options = info.settings.hybrid
# set
info = setProcessMLModels(info)
if hasproperty(info.settings.hybrid, :ml_model)
info = setHybridOptions(info, :ml_model)
info = setHybridOptions(info, :ml_training)
info = setHybridOptions(info, :ml_gradient)
info = setHybridOptions(info, :ml_optimizer)
checkpoint_path = ""
hybrid_root = joinpath(dirname(info.output.dirs.data),"hybrid")
mkpath(hybrid_root)
if info.settings.hybrid.save_checkpoint
checkpoint_path = joinpath(hybrid_root,"training_checkpoints")
mkpath(checkpoint_path)
end
output_dirs = info.temp.output.dirs
output_dirs = (; output_dirs..., hybrid=(; root=hybrid_root, checkpoint=checkpoint_path))
info = (; info..., temp = (info.temp..., output = (; info.temp.output..., dirs = output_dirs)))
fold_type = CalcFoldFromSplit()
fold_path = ""
which_fold = 1
ml_training = info.settings.hybrid.ml_training
if hasproperty(ml_training, :fold_path)
fold_path_file = ml_training.fold_path
fold_path = isnothing(fold_path_file) ? fold_path : fold_path_file
if !isempty(fold_path)
fold_type = LoadFoldFromFile()
if !isabspath(fold_path)
fold_path = joinpath(info.temp.experiment.dirs.settings, fold_path)
end
end
end
if hasproperty(ml_training, :which_fold)
which_fold = ml_training.which_fold
end
fold_s = (; fold_path, which_fold, fold_type)
info = set_namedtuple_subfield(info, :hybrid, (:fold, fold_s))
replace_value_for_gradient = hasproperty(info.settings.hybrid, :replace_value_for_gradient) ? info.settings.hybrid.replace_value_for_gradient : 0.0
info = set_namedtuple_subfield(info, :hybrid, (:replace_value_for_gradient, info.temp.helpers.numbers.num_type(replace_value_for_gradient)))
info = set_namedtuple_subfield(info, :hybrid, (:ml_experiment_type, getTypeInstanceForNamedOptions(info.settings.hybrid.ml_experiment_type)))
covariates_path = getAbsDataPath(info.temp, info.settings.hybrid.covariates.path)
covariates = (; path=covariates_path, options=info.settings.hybrid.covariates.options)
info = set_namedtuple_subfield(info, :hybrid, (:covariates, covariates))
info = set_namedtuple_subfield(info, :hybrid, (:random_seed, info.settings.hybrid.random_seed))
end
return info
endsetModelOutput
Sindbad.Setup.setModelOutput Function
setModelOutput(info::NamedTuple)Sets the output variables to be written and stored based on the experiment configuration.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- The updated
infoNamedTuple with output variables and depth information added.
Code
function setModelOutput(info::NamedTuple)
print_info(setModelOutput, @__FILE__, @__LINE__, "setting Model Output Info...")
output_vars = collect(propertynames(info.settings.experiment.model_output.variables))
info = (; info..., temp=(; info.temp..., output=getDepthInfoAndVariables(info, output_vars)))
return info
end
function setModelOutputLandAll(info, land)
output_vars = getAllLandVars(land)
depth_info = getLandVarDepthInfo(info, land, output_vars)
info = @set info.output.variables = output_vars
info = @set info.output.depth_info = depth_info
return info
endsetModelOutputLandAll
Sindbad.Setup.setModelOutputLandAll Function
setModelOutputLandAll(info, land)Retrieves all model variables from land and overwrites the output information in info.
Arguments:
info: A NamedTuple containing experiment configuration and helper information.land: A core SINDBAD NamedTuple containing variables for a given time step.
Returns:
- The updated
infoNamedTuple with output variables and depth information updated.
Code
function setModelOutputLandAll(info, land)
output_vars = getAllLandVars(land)
depth_info = getLandVarDepthInfo(info, land, output_vars)
info = @set info.output.variables = output_vars
info = @set info.output.depth_info = depth_info
return info
endsetOptimization
Sindbad.Setup.setOptimization Function
setOptimization(info::NamedTuple)Sets up optimization-related fields in the experiment configuration.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- The updated
infoNamedTuple with optimization-related fields added.
Notes:
Configures cost metrics, optimization parameters, algorithms, and variables to store during optimization.
Validates the parameters to be optimized against the model structure.
Code
function setOptimization(info::NamedTuple)
print_info(setOptimization, @__FILE__, @__LINE__, "setting ParameterOptimization and Observation Info...")
# The optimization info is built on its own and set in info once at the end, because
# every rebuild of info compiles anew for its large type.
n_threads_cost = 1
run_lazy =get(info.settings.experiment.flags, :run_lazy, false)
threaded_cost = info.settings.optimization.optimization_cost_threaded > 0 && info.settings.experiment.flags.run_optimization && !run_lazy
if threaded_cost
n_threads_cost = info.settings.optimization.optimization_cost_threaded > 1 ? info.settings.optimization.optimization_cost_threaded : Threads.nthreads()
end
# get types for optimization run options
multi_constraint_method = getTypeInstanceForNamedOptions(info.settings.optimization.multi_constraint_method)
cost_method = getTypeInstanceForNamedOptions(info.settings.optimization.optimization_cost_method)
scaling_method = isnothing(info.settings.optimization.optimization_parameter_scaling) ? "scale_none" : info.settings.optimization.optimization_parameter_scaling
parameter_scaling = getTypeInstanceForNamedOptions(scaling_method)
optimization_types = (; cost_method, parameter_scaling, multi_constraint_method, n_threads_cost)
optimization = (; run_options=optimization_types)
# set algorithm related options
optimization = setAlgorithmOptions(optimization, info, :algorithm_optimization)
optimization = setAlgorithmOptions(optimization, info, :algorithm_sensitivity_analysis)
parameter_table = getOptimizationParametersTable(info.temp.models.parameter_table, info.settings.optimization.model_parameter_default, info.settings.optimization.model_parameters_to_optimize)
checkOptimizedParametersInModels(info, parameter_table)
# use no scaling while checking bounds
checkParameterBounds(parameter_table.name, parameter_table.initial, parameter_table.lower, parameter_table.upper, ScaleNone(), p_units=parameter_table.units, show_info=true, model_names=parameter_table.model_approach)
# get the variables to be used during optimization
obs_vars, optim_vars, store_vars, model_vars = getConstraintNames(info.settings.optimization)
vars_info = (; obs=obs_vars, optimization=optim_vars, store=store_vars, model=model_vars)
# updateVariablesToStore reads the variables from info.optimization
temp = info.temp
if threaded_cost
# overwrite land array type when threaded optimization is set
temp = (; temp..., helpers=(; temp.helpers..., run=(; temp.helpers.run..., land_output_type=PreAllocArrayMT())))
end
info = (; info..., temp=temp, optimization=(; variables=vars_info))
info = updateVariablesToStore(info)
cost_options = getCostOptions(info.settings.optimization, vars_info, info.temp.output.variables, info.temp.helpers.numbers, info.temp.helpers.dates)
optimization = (; optimization..., cost_options=cost_options, parameter_table=parameter_table)
info = (; info..., optimization=optimization)
return info
endsetOrderedSelectedModels
Sindbad.Setup.setOrderedSelectedModels Function
setOrderedSelectedModels(info::NamedTuple)Retrieves and orders the list of selected models based on the configuration in model_structure.json.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- The updated
infoNamedTuple with the ordered list of selected models added toinfo.temp.models.
Notes:
Ensures consistency by validating the selected models using
checkSelectedModels.Orders the models as specified in
standard_sindbad_model.
Code
function setOrderedSelectedModels(info::NamedTuple)
print_info(setOrderedSelectedModels, @__FILE__, @__LINE__, "setting Ordered Selected Models...")
selected_models = collect(propertynames(info.settings.model_structure.models))
sindbad_models = getAllSindbadModels(info, selected_models=selected_models, selected_models_info=info.settings.model_structure.models)
checkSelectedModels(sindbad_models, selected_models)
t_repeat_models = getModelImplicitTRepeat(info, selected_models)
# checkSelectedModels(sindbad_models, selected_models)
order_selected_models = []
for msm ∈ sindbad_models
if msm in selected_models
push!(order_selected_models, msm)
end
end
@debug " setupInfo: creating initial out/land..."
info = (; info..., temp=(; info.temp..., models=(; sindbad_models=sindbad_models, selected_models=Table((; model=[order_selected_models...], t_repeat=t_repeat_models)))))
return info
endsetPoolsInfo
Sindbad.Setup.setPoolsInfo Function
setPoolsInfo(info::NamedTuple)Generates info.temp.helpers.pools and info.pools.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- The updated
infoNamedTuple with pool-related fields added.
Notes:
info.temp.helpers.poolsis used in the models.info.poolsis used for instantiating the pools for the initial output tuple.
Code
function setPoolsInfo(info::NamedTuple)
print_info(setPoolsInfo, @__FILE__, @__LINE__, "setting Pools Info...")
# after this, the source of the structure stops mattering: everything below reads
# the resolved blocks, never info.settings.model_structure.pools
resolved_pools = resolvePoolStructure(info)
elements = propertynames(resolved_pools)
tmp_states = (;)
hlp_states = (;)
model_array_type = getfield(Types, to_uppercase_first(info.settings.experiment.exe_rules.model_array_type, "ModelArray"))()
num_type = info.temp.helpers.numbers.num_type
for element ∈ elements
vals_tuple = (;)
vals_tuple = set_namedtuple_field(vals_tuple, (:zix, (;)))
vals_tuple = set_namedtuple_field(vals_tuple, (:self, (;)))
vals_tuple = set_namedtuple_field(vals_tuple, (:all_components, (;)))
elSymbol = Symbol(element)
tmp_elem = (;)
hlp_elem = (;)
tmp_states = set_namedtuple_field(tmp_states, (elSymbol, (;)))
hlp_states = set_namedtuple_field(hlp_states, (elSymbol, (;)))
resolved_elem = getproperty(resolved_pools, Symbol(element))
pool_info = getfield(resolved_elem, :components)
nlayers = Int64[]
layer_thicknesses = num_type[]
layer = Int64[]
inits = []
sub_pool_name = Symbol[]
main_pool_name = Symbol[]
main_pools = Symbol.(keys(pool_info))
layer_thicknesses, nlayers, layer, inits, sub_pool_name, main_pool_name = getPoolInformation(main_pools, pool_info, layer_thicknesses, nlayers, layer, inits, sub_pool_name, main_pool_name)
# set empty tuple fields
tpl_fields = (:components, :zix, :initial_values, :layer_thickness)
for _tpl ∈ tpl_fields
tmp_elem = set_namedtuple_field(tmp_elem, (_tpl, (;)))
end
hlp_elem = set_namedtuple_field(hlp_elem, (:layer_thickness, (;)))
# hlp_elem = set_namedtuple_field(hlp_elem, (:n_layers, (;)))
hlp_elem = set_namedtuple_field(hlp_elem, (:zix, (;)))
hlp_elem = set_namedtuple_field(hlp_elem, (:components, (;)))
hlp_elem = set_namedtuple_field(hlp_elem, (:all_components, (;)))
hlp_elem = set_namedtuple_field(hlp_elem, (:zeros, (;)))
hlp_elem = set_namedtuple_field(hlp_elem, (:ones, (;)))
# main pools
for main_pool ∈ main_pool_name
zix = Int[]
initial_values = []
# initial_values = num_type[]
components = Symbol[]
for (ind, par) ∈ enumerate(sub_pool_name)
if startswith(String(par), String(main_pool))
push!(zix, ind)
push!(components, sub_pool_name[ind])
push!(initial_values, inits[ind])
end
end
initial_values = createArrayofType(initial_values, Nothing[], num_type, nothing, true, model_array_type)
zix = Tuple(zix)
tmp_elem = set_namedtuple_subfield(tmp_elem, :components, (main_pool, Tuple(components)))
tmp_elem = set_namedtuple_subfield(tmp_elem, :zix, (main_pool, zix))
tmp_elem = set_namedtuple_subfield(tmp_elem, :initial_values, (main_pool, initial_values))
# hlp_elem = set_namedtuple_subfield(hlp_elem, :n_layers, (main_pool, length(zix)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :zix, (main_pool, zix))
hlp_elem = set_namedtuple_subfield(hlp_elem, :components, (main_pool, Tuple(components)))
onetyped = createArrayofType(ones(size(initial_values)), Nothing[], num_type, nothing, true, model_array_type)
hlp_elem = set_namedtuple_subfield(hlp_elem, :zeros, (main_pool, zero(onetyped)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :ones, (main_pool, onetyped))
end
# subpools
unique_sub_pools = Symbol[]
for _sp ∈ sub_pool_name
if _sp ∉ unique_sub_pools
push!(unique_sub_pools, _sp)
end
end
for sub_pool ∈ unique_sub_pools
zix = Int[]
initial_values = []
components = Symbol[]
ltck = num_type[]
# ltck = []
for (ind, par) ∈ enumerate(sub_pool_name)
if par == sub_pool
push!(zix, ind)
push!(initial_values, inits[ind])
push!(components, sub_pool_name[ind])
push!(ltck, layer_thicknesses[ind])
end
end
zix = Tuple(zix)
initial_values = createArrayofType(initial_values, Nothing[], num_type, nothing, true, model_array_type)
tmp_elem = set_namedtuple_subfield(tmp_elem, :components, (sub_pool, Tuple(components)))
tmp_elem = set_namedtuple_subfield(tmp_elem, :zix, (sub_pool, zix))
tmp_elem = set_namedtuple_subfield(tmp_elem, :initial_values, (sub_pool, initial_values))
tmp_elem = set_namedtuple_subfield(tmp_elem, :layer_thickness, (sub_pool, Tuple(ltck)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :layer_thickness, (sub_pool, Tuple(ltck)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :zix, (sub_pool, zix))
# hlp_elem = set_namedtuple_subfield(hlp_elem, :n_layers, (sub_pool, length(zix)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :components, (sub_pool, Tuple(components)))
onetyped = createArrayofType(ones(size(initial_values)), Nothing[], num_type, nothing, true, model_array_type)
# onetyped = ones(length(initial_values))
hlp_elem = set_namedtuple_subfield(hlp_elem, :zeros, (sub_pool, zero(onetyped)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :ones, (sub_pool, onetyped))
end
## combined pools
combine_pools = getfield(resolved_elem, :combine)
do_combine = true
tmp_elem = set_namedtuple_field(tmp_elem, (:combine, (; docombine=true, pool=Symbol(combine_pools))))
if do_combine
combined_pool_name = Symbol.(combine_pools)
create = Symbol[combined_pool_name]
components = Symbol[]
for _sp ∈ sub_pool_name
if _sp ∉ components
push!(components, _sp)
end
end
initial_values = inits
initial_values = createArrayofType(initial_values, Nothing[], num_type, nothing, true, model_array_type)
zix = collect(1:1:length(main_pool_name))
zix = Tuple(zix)
tmp_elem = set_namedtuple_subfield(tmp_elem, :components, (combined_pool_name, Tuple(components)))
tmp_elem = set_namedtuple_subfield(tmp_elem, :zix, (combined_pool_name, zix))
tmp_elem = set_namedtuple_subfield(tmp_elem, :initial_values, (combined_pool_name, initial_values))
# hlp_elem = set_namedtuple_subfield(hlp_elem, :n_layers, (combined_pool_name, length(zix)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :zix, (combined_pool_name, zix))
onetyped = createArrayofType(ones(size(initial_values)), Nothing[], num_type, nothing, true, model_array_type)
all_components = Tuple([_k for _k in keys(tmp_elem.zix) if _k !== combined_pool_name])
hlp_elem = set_namedtuple_subfield(hlp_elem, :all_components, (combined_pool_name, all_components))
vals_tuple = set_namedtuple_subfield(vals_tuple, :zix, (combined_pool_name, Val(hlp_elem.zix)))
vals_tuple = set_namedtuple_subfield(vals_tuple, :self, (combined_pool_name, Val(combined_pool_name)))
vals_tuple = set_namedtuple_subfield(vals_tuple, :all_components, (combined_pool_name, Val(all_components)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :components, (combined_pool_name, Tuple(components)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :zeros, (combined_pool_name, zero(onetyped)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :ones, (combined_pool_name, onetyped))
else
create = Symbol.(unique_sub_pools)
end
# check if additional variables exist
if hasproperty(resolved_elem, :state_variables)
state_variables = getfield(resolved_elem, :state_variables)
tmp_elem = set_namedtuple_field(tmp_elem, (:state_variables, state_variables))
end
arraytype = :view
if hasproperty(info.settings.experiment.exe_rules, :model_array_type)
arraytype = Symbol(info.settings.experiment.exe_rules.model_array_type)
end
tmp_elem = set_namedtuple_field(tmp_elem, (:arraytype, arraytype))
tmp_elem = set_namedtuple_field(tmp_elem, (:create, create))
# aliases: groupings that cut across the nesting, so they cannot be a nesting
# level. hlp_elem only -- an alias has no backing array, so keeping it out of
# tmp_elem keeps it out of `create`, `initial_values`, `all_components` and
# `n_layers`.
for alias ∈ propertynames(getfield(resolved_elem, :aliases))
targets = getproperty(getfield(resolved_elem, :aliases), alias)
alias_zix = Tuple(sort(vcat([collect(getproperty(hlp_elem.zix, t)) for t ∈ targets]...)))
hlp_elem = set_namedtuple_subfield(hlp_elem, :zix, (alias, alias_zix))
end
# the Vals built alongside the combined pool are what the generated
# setComponentFromMainPool dispatches on, so they have to travel with the
# element
hlp_elem = set_namedtuple_field(hlp_elem, (:vals, vals_tuple))
tmp_states = set_namedtuple_field(tmp_states, (elSymbol, tmp_elem))
hlp_states = set_namedtuple_field(hlp_states, (elSymbol, hlp_elem))
end
hlp_new = (;)
# tc_print(hlp_states)
eleprops = propertynames(hlp_states)
if :carbon in eleprops && :water in eleprops
for prop ∈ propertynames(hlp_states.carbon)
cfield = getproperty(hlp_states.carbon, prop)
wfield = getproperty(hlp_states.water, prop)
cwfield = (; cfield..., wfield...)
if prop == :vals
cwfield = (;)
for subprop in propertynames(cfield)
csub = getproperty(cfield, subprop)
wsub = getproperty(wfield, subprop)
cwfield = set_namedtuple_field(cwfield, (subprop, (; csub..., wsub...)))
end
end
hlp_new = set_namedtuple_field(hlp_new, (prop, cwfield))
end
elseif :carbon in eleprops && :water ∉ eleprops
hlp_new = hlp_states.carbon
elseif :carbon ∉ eleprops && :water in eleprops
hlp_new = hlp_states.water
else
hlp_new = hlp_states
end
# Every carbon pool name resolves, to real indices or to (), whatever the
# structure. An empty entry iterates zero times, statically, which is why models
# loop over helpers.pools.zix.X with no isempty branch and why a new pool needs no
# model edit -- the names come from the configurations themselves, see
# carbonPoolNames. Filled here rather than per element on
# purpose: the carbon+water merge above is `(; carbon..., water...)`, so water
# wins on any shared key, and a per-element skeleton would have water's empty
# entries overwrite carbon's real indices. Doing it after the merge also covers a
# model structure with no carbon element at all, where the merge takes
# hlp_states.water wholesale.
for pool_name ∈ carbonPoolNames()
if !hasproperty(hlp_new.zix, pool_name)
hlp_new = set_namedtuple_subfield(hlp_new, :zix, (pool_name, ()))
end
end
# get the number of layers per pool
n_layers = NamedTuple(map(propertynames(hlp_new.ones)) do one_pool
n_pool = num_type(length(getproperty(hlp_new.ones, one_pool)))
Pair(one_pool, n_pool)
end
)
hlp_new = (hlp_new..., n_layers=n_layers)
# provenance: info.settings.model_structure.pools stays exactly as the user wrote
# it, so the saved settings still read "carbon": "cCycleBase"; pool_structure
# records what that resolved to, so an output directory documents the structure
# actually run.
info = (; info..., pools=tmp_states, pool_structure=resolved_pools,
temp=(; info.temp..., helpers=(; info.temp.helpers..., pools=hlp_new)))
return info
endsetSpinupAndForwardModels
Sindbad.Setup.setSpinupAndForwardModels Function
setSpinupAndForwardModels(info::NamedTuple)Configures the spinup and forward models for the experiment.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- The updated
infoNamedTuple with the spinup and forward models added toinfo.temp.models.
Notes:
Allows for faster spinup by turning off certain models using the
use_in_spinupflag inmodel_structure.json.Ensures that spinup models are a subset of forward models.
Updates model parameters if additional parameter values are provided in the experiment configuration.
Code
function setSpinupAndForwardModels(info::NamedTuple)
print_info(setSpinupAndForwardModels, @__FILE__, @__LINE__, "setting Spinup and Forward Models...")
selected_approach_forward = ()
is_spinup = Int64[]
order_selected_models = info.temp.models.selected_models.model
default_model = getfield(info.settings.model_structure, :default_model)
for sm ∈ order_selected_models
model_info = getfield(info.settings.model_structure.models, sm)
selected_approach = model_info.approach
selected_approach = String(sm) * "_" * selected_approach
selected_approach_func = getTypedModel(Symbol(selected_approach), info.temp.helpers.dates.temporal_resolution, info.temp.helpers.numbers.num_type)
selected_approach_forward = (selected_approach_forward..., selected_approach_func)
if :use_in_spinup in propertynames(model_info)
use_in_spinup = model_info.use_in_spinup
else
use_in_spinup = default_model.use_in_spinup
end
if use_in_spinup == true
push!(is_spinup, 1)
else
push!(is_spinup, 0)
end
end
# change is_spinup to a vector of indices
is_spinup = findall(is_spinup .== 1)
info = (; info..., temp=(; info.temp..., models=(; info.temp.models..., forward=selected_approach_forward, is_spinup=is_spinup)))
# update the parameters of the approaches if a parameter value has been added from the experiment configuration
if length(selected_approach_forward) > 0
default_parameter_table = getParameters(selected_approach_forward, info.temp.helpers.numbers.num_type, info.temp.helpers.dates.temporal_resolution)
input_parameter_table = nothing
if hasproperty(info.settings.model_structure, :parameter_table) && !isempty(info.settings.model_structure.parameter_table)
print_info(setSpinupAndForwardModels, @__FILE__, @__LINE__, "---using input parameters from model_structure.parameter_table in replace_info", n_m=20)
input_parameter_table = info.settings.model_structure.parameter_table
elseif hasproperty(info[:settings], :parameters) && info.settings.parameters isa Table && !isempty(info.settings.parameters)
# only a csv input is a Table. A json or zarr/nc input sets the parameters
# per location, which are loaded in getForcing and applied in prepTEM.
print_info(setSpinupAndForwardModels, @__FILE__, @__LINE__, " ---using input parameters from settings.parameters passed from CSV input file", n_m=20)
input_parameter_table = info.settings.parameters
end
updated_parameter_table = copy(default_parameter_table)
if !isnothing(input_parameter_table)
if !isa(input_parameter_table, Table)
error("input_parameter_table must be a Table, but its type is $(typeof(input_parameter_table)). Fix the input error in the source (table or csv file) to continue...")
end
updated_parameter_table = setInputParameters(default_parameter_table, input_parameter_table, info.temp.helpers.dates.temporal_resolution)
selected_approach_forward = updateModelParameters(updated_parameter_table, selected_approach_forward, updated_parameter_table.optimized)
end
info = (; info..., temp=(; info.temp..., models=(; info.temp.models..., forward=selected_approach_forward, is_spinup=is_spinup, parameter_table=updated_parameter_table)))
end
return info
endsetTypedParameters
Sindbad.Setup.setTypedParameters Function
setTypedParameters(selected_models, parameter_to_index::NamedTuple, parameter_values)
setTypedParameters(selected_models, parameter_indices_per_model, parameter_values)Returns the models with their parameters set from parameter_values, keeping the type of each field.
Arguments:
selected_models: a tuple or long tuple of modelsparameter_to_index: a NamedTuple of model approach names, each with a NamedTuple of parameter names and their index inparameter_values, e.g., fromgetParameterIndices. The model of each entry is found by its name.parameter_indices_per_model: the same indices with one element per model in the order ofselected_models, fromgetParameterIndicesPerModel. A tuple for a tuple of models, and a long tuple with the same chunks for a long tuple of models. Model k is paired with element k, so no model is looked up by name.parameter_values: the values of the parameters
Notes:
Each value is converted to the type of the field it replaces, so the models keep their concrete types. This keeps a vector of models for all locations type stable.
The method with
parameter_to_indexsuits a call made once at setup. Use the one withparameter_indices_per_modelwhen the models are updated many times, as in the cost function of an optimization. It does not allocate for models without parameters to set and is inferred for a long tuple of models.updateModelParameterswith aparameter_to_indexdoes not convert, because the values can be dual numbers in gradient based runs. Use that one there.
Code
function setTypedParameters(selected_models, parameter_to_index::NamedTuple, parameter_values)
return map(selected_models) do model
model_index = parameter_to_index[nameof(typeof(model))]
isempty(model_index) && return model
p_keys = keys(model_index)
new_values = map(p_keys) do p_key
convertParameterValue(getproperty(model, p_key), parameter_values[getfield(model_index, p_key)])
end
ConstructionBase.setproperties(model, NamedTuple{p_keys}(new_values))
end
end
function setTypedParameters(selected_models::Tuple, parameter_indices_per_model::Tuple, parameter_values)
return map((model, model_indices) -> setTypedModelParameters(model, model_indices, parameter_values), selected_models, parameter_indices_per_model)
end
function setTypedParameters(selected_models::LongTuple{N}, parameter_indices_per_model::LongTuple{N}, parameter_values) where {N}
return LongTuple{N}(map((models, model_indices) -> setTypedParameters(models, model_indices, parameter_values), selected_models.data, parameter_indices_per_model.data))
endsetupInfo
Sindbad.Setup.setupInfo Function
setupInfo(info::NamedTuple)Processes the experiment configuration and sets up all necessary fields for model simulation.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- The updated
infoNamedTuple with all necessary fields for model simulation.
Code
function setupInfo(info::NamedTuple)
# Retain configuration for safe, pre-run rebuilding of site-specific pool geometry.
setup_input = deepcopy(info)
print_info(setupInfo, @__FILE__, @__LINE__, "Setting and consolidating Experiment Info...")
# @show info.settings.model_structure.parameter_table.optimized
info = setExperimentBasics(info)
# @info " setupInfo: setting Output Basics..."
info = setExperimentOutput(info)
# @info " setupInfo: setting Numeric Helpers..."
info = setNumericHelpers(info)
# @info " setupInfo: setting Pools Info..."
info = setPoolsInfo(info)
# @info " setupInfo: setting Dates Helpers..."
info = setDatesInfo(info)
# @info " setupInfo: setting Model Structure..."
info = setOrderedSelectedModels(info)
# @info " setupInfo: setting Spinup and Forward Models..."
info = setSpinupAndForwardModels(info)
# @info " setupInfo: ...saving Selected Models Code..."
_ = parseSaveCode(info)
# add information related to model run
# @info " setupInfo: setting Model Run Flags..."
info = setModelRunInfo(info)
# @info " setupInfo: setting Spinup Info..."
info = setSpinupInfo(info)
# @info " setupInfo: setting Model Output Info..."
info = setModelOutput(info)
# @info " setupInfo: creating Initial Land..."
land_init = createInitLand(info.pools, info.temp)
info = (; info..., temp=(; info.temp..., helpers=(; info.temp.helpers..., land_init=land_init)))
data_settings = (;)
if hasproperty(info.settings, :forcing)
data_settings = set_namedtuple_field(data_settings, (:forcing, setDefaultForcingBounds(info.settings.forcing, info.temp.helpers.numbers.num_type)))
end
data_settings = set_namedtuple_field(data_settings, (:parameters, getParameterInput(info)))
if (info.settings.experiment.flags.run_optimization || info.settings.experiment.flags.calc_cost) && hasproperty(info.settings.optimization, :algorithm_optimization)
# @info " setupInfo: setting ParameterOptimization and Observation info..."
info = setOptimization(info)
data_settings = set_namedtuple_field(data_settings, (:optimization, info.settings.optimization))
else
parameter_table = get(info.temp.models, :parameter_table, nothing)
if !isnothing(parameter_table)
checkParameterBounds(parameter_table.name, parameter_table.initial, parameter_table.lower, parameter_table.upper, ScaleNone(), p_units=parameter_table.units, show_info=true, model_names=parameter_table.model_approach)
end
end
if hasproperty(info.settings, :hybrid)
info = setHybridInfo(info)
end
print_info(setupInfo, @__FILE__, @__LINE__, "Cleaning Info Fields...")
exe_rules = info.settings.experiment.exe_rules
info = drop_namedtuple_fields(info, (:model_structure, :experiment, :output, :pools))
info = (; info..., info.temp...)
info = set_namedtuple_subfield(info, :experiment, (:data_settings, data_settings))
info = set_namedtuple_subfield(info, :experiment, (:exe_rules, exe_rules))
info = drop_namedtuple_fields(info, (:temp, :settings,))
# Converted last because every NamedTuple rebuild above compiles anew for the type of
# info, and a LongTuple of all models makes that type very large.
if !isnothing(exe_rules.longtuple_size)
longtuple_size = exe_rules.longtuple_size
# Julia does not unroll `map` over a tuple of 32 or more elements, so it is not
# inferred and allocates. Every chunk of a long tuple must be shorter than that.
if longtuple_size >= 32
@warn "experiment.exe_rules.longtuple_size is $(longtuple_size). With 32 or more models in a chunk, setting parameters on the models is not type stable and allocates on every call, which slows down optimizations. Use a value from 16 to 31."
end
forward = to_longtuple(info.models.forward, longtuple_size)
return (; info..., models=(; info.models..., forward=forward), setup_input)
end
return (; info..., setup_input)
endsindbadDefaultOptions
Sindbad.Setup.sindbadDefaultOptions Function
sindbadDefaultOptions(::MethodType)Retrieves the default configuration options for a given optimization or sensitivity analysis method in SINDBAD.
Arguments:
::MethodType: The method type for which the default options are requested. Supported types include:ParameterOptimizationMethod: General optimization methods.GSAMethod: General global sensitivity analysis methods.GSAMorris: Morris method for global sensitivity analysis.GSASobol: Sobol method for global sensitivity analysis.GSASobolDM: Sobol method with derivative-based measures.SpinupSequence: Methods that build the spinup sequence of a location.
Returns:
- A
NamedTuplecontaining the default options for the specified method.
Notes:
Each method type has its own set of default options, such as the number of trajectories, samples, or design matrix length.
For
GSASobolDM, the defaults are inherited fromGSASobol.Defaults can be given on an abstract type, so that every method below it shares them and only the methods that need more define their own.
SpinupSequenceis set up this way.
Code
function sindbadDefaultOptions end
# A basic empty options for all SindbadTypes
sindbadDefaultOptions(::SindbadTypes) = (;)
sindbadDefaultOptions(::CMAEvolutionStrategyCMAES) = (; maxfevals = 50)
sindbadDefaultOptions(::SpinupSequence) = (; n_repeat_base = 200, forcing_msc = "day_MSC")
# invoke reaches the SpinupSequence method above, so this stays the shared defaults plus the
# one option these methods add, rather than a second copy of them
sindbadDefaultOptions(m::Union{SequenceWithAge,SequenceWithSpinupInput}) =(; invoke(sindbadDefaultOptions, Tuple{SpinupSequence}, m)..., disturbance_variable = "f_dist_year")
# this method builds nothing, it replays the steps authored in the settings, so none of the
# shared options above apply to it
sindbadDefaultOptions(::SequenceList) = (; steps = [])
sindbadDefaultOptions(::GSAMorris) = (; total_num_trajectory = 200, num_trajectory = 15, len_design_mat=10)
sindbadDefaultOptions(::GSASobol) = (; samples = 5, method_options=(; order=[0, 1]), sampler="Sobol", sampler_options=(;))
sindbadDefaultOptions(::GSASobolDM) = sindbadDefaultOptions(GSASobol())updateModelParameters
Sindbad.Setup.updateModelParameters Function
updateModelParameters(parameter_table::Table, selected_models::Tuple, parameter_vector::AbstractArray)
updateModelParameters(parameter_table::Table, selected_models::LongTuple, parameter_vector::AbstractArray)
updateModelParameters(parameter_to_index::NamedTuple, selected_models::Tuple, parameter_vector::AbstractArray)Updates the parameters of SINDBAD models based on the provided parameter vector without mutating the original table of parameters.
Arguments:
parameter_table::Table: A table of SINDBAD model parameters selected for optimization. Contains parameter names, bounds, and scaling information.selected_models::Tuple: A tuple of all models selected in the given model structure.selected_models::LongTuple: A long tuple of models, which is converted into a standard tuple for processing.parameter_vector::AbstractArray: A vector of parameter values to update the models.parameter_to_index::NamedTuple: A mapping of parameter indices to model names, used for updating specific parameters in the models.
Returns:
- A tuple of updated models with their parameters modified according to the provided
parameter_vector.
Notes:
The function supports multiple input formats for
selected_models(e.g.,LongTuple,NamedTuple) and adapts accordingly.If
parameter_tableis provided, the function uses it to find and update the relevant parameters for each model.The
parameter_to_indexvariant allows for a more direct mapping of parameters to models, bypassing the need for a parameter table.
Examples:
- Using
parameter_tableandselected_models:
updated_models = updateModelParameters(parameter_table, selected_models, parameter_vector)- Using
parameter_to_indexfor direct mapping:
updated_models = updateModelParameters(parameter_to_index, selected_models, parameter_vector)Implementation Details:
The function iterates over the models in
selected_modelsand updates their parameters based on the providedparameter_vector.For each model, it checks if the parameter belongs to the model's approach (using
parameter_table.model_approach) and updates the corresponding value.The
parameter_to_indexvariant uses a mapping to directly replace parameter values in the models.
Code
function updateModelParameters end
function updateModelParameters(parameter_table::Table, selected_models::LongTuple, parameter_vector::AbstractArray)
selected_models = to_tuple(selected_models)
return updateModelParameters(parameter_table, selected_models, parameter_vector)
end
function updateModelParameters(parameter_table::Table, selected_models::LongTuple, parameter_vector::AbstractArray)
selected_models = to_tuple(selected_models)
return updateModelParameters(parameter_table, selected_models, parameter_vector)
end
function updateModelParameters(parameter_table::Table, selected_models::Tuple, parameter_vector::AbstractArray)
updatedModels = eltype(selected_models)[]
namesApproaches = nameof.(typeof.(selected_models)) # a better way to do this?
for (idx, modelName) ∈ enumerate(namesApproaches)
approachx = selected_models[idx]
model_obj = approachx
newapproachx = if modelName in parameter_table.model_approach
vars = propertynames(approachx)
newvals = Pair[]
for var ∈ vars
pindex = findall(row -> row.name == var && row.model_approach == modelName,
parameter_table)
pval = getproperty(approachx, var)
if !isempty(pindex)
pval = parameter_vector[pindex[1]]
end
push!(newvals, var => pval)
end
typeof(approachx)(; newvals...)
else
approachx
end
push!(updatedModels, newapproachx)
end
return (updatedModels...,)
end
function updateModelParameters(parameter_to_index::NamedTuple, selected_models, parameter_vector::AbstractArray)
map(selected_models) do model
modelmap = parameter_to_index[nameof(typeof(model))]
varsreplace = map(i->parameter_vector[i],modelmap)
ConstructionBase.setproperties(model,varsreplace)
end
endupdateModels
Sindbad.Setup.updateModels Function
updateModels(parameter_vector, parameter_updater, parameter_scaling_type, selected_models)Updates the parameters of selected models using the provided parameter vector.
Arguments
parameter_vector: Vector containing the new parameter valuesparameter_updater: the parameters to update and where they are in the models. One of:a
ParameterUpdaterfromprepParameterUpdater. Use this one when the models are updated many times, as in an optimization.the parameter table, from which the indices are found on every call
a
parameter_to_indexNamedTuple, which does not convert the value types, for gradient based runs
parameter_scaling_type: Specifies the type of scaling to be applied to parametersselected_models: Collection of models whose parameters need to be updated, or a vector of them with one element per location
Returns
Updated models with new parameter values. parameter_vector is not changed.
Code
function updateModels(parameter_vector, parameter_updater, parameter_scaling_type, selected_models)
parameter_vector = backScaleParameters(parameter_vector, parameter_updater, parameter_scaling_type)
updated_models = updateModelParameters(parameter_updater, selected_models, parameter_vector)
return updated_models
end
function updateModels(parameter_vector, parameter_updater, parameter_scaling_type, space_selected_models::AbstractVector)
parameter_vector = backScaleParameters(parameter_vector, parameter_updater, parameter_scaling_type)
parameter_to_index = parameter_updater isa Table ? getParameterIndices(first(space_selected_models), parameter_updater) : parameter_updater
return map(space_selected_models) do selected_models
setTypedParameters(selected_models, parameter_to_index, parameter_vector)
end
end
function updateModels(parameter_vector, parameter_updater::ParameterUpdater, parameter_scaling_type, selected_models)
parameter_vector = backScaleParameters(parameter_vector, parameter_updater.parameter_table, parameter_scaling_type)
return setTypedParameters(selected_models, parameter_updater.parameter_indices_per_model, parameter_vector)
end
function updateModels(parameter_vector, parameter_updater::GradientParameterUpdater, parameter_scaling_type, selected_models)
parameter_vector = backScaleParameters(parameter_vector, parameter_updater.parameter_table, parameter_scaling_type)
return setParameters(selected_models, parameter_updater.parameter_indices_per_model, parameter_vector)
end
function updateModels(parameter_vector, parameter_updater::ParameterUpdater, parameter_scaling_type, space_selected_models::AbstractVector)
parameter_vector = backScaleParameters(parameter_vector, parameter_updater.parameter_table, parameter_scaling_type)
return map(space_selected_models) do selected_models
setTypedParameters(selected_models, parameter_updater.parameter_indices_per_model, parameter_vector)
end
endupdateVariablesToStore
Sindbad.Setup.updateVariablesToStore Function
updateVariablesToStore(info::NamedTuple)Updates the output variables to store based on optimization or cost run settings.
Arguments:
info: A NamedTuple containing the experiment configuration.
Returns:
- The updated
infoNamedTuple with updated output variables.
Code
function updateVariablesToStore(info::NamedTuple)
output_vars = info.settings.experiment.model_output.variables
if info.settings.experiment.flags.calc_cost || !info.settings.optimization.subset_model_output
output_vars = union(String.(keys(output_vars)),
info.optimization.variables.model)
else
output_vars = map(info.optimization.variables.obs) do vo
vn = getfield(info.optimization.variables.optimization, vo)
Symbol(string(vn[1]) * "." * string(vn[2]))
end
end
info = (; info..., temp=(; info.temp..., output=getDepthInfoAndVariables(info, output_vars)))
return info
endwriteParameterInputJson
Sindbad.Setup.writeParameterInputJson Function
writeParameterInputJson(json_path, data_path, variable_attributes; space_dimensions=nothing)Writes the params.json of a parameter file, which can be given as the parameters config file of another experiment.
Arguments:
json_path: the path of the json file to writedata_path: the absolute path of the parameter filevariable_attributes: an ordered collection of variable name and attribute pairs, e.g., fromparameterVariableAttributesspace_dimensions: the names of the spatial dimensions of the parameter file, e.g., thedata_dimension.spaceof the experiment that wrote it, ornothing
Returns:
json_path
Notes:
default_parameter_mapgets thespaceofdimensionwhenspace_dimensionsis given, and always thespaceofinterpolation_methodasnearest, so the json shows how another experiment reads the file, seeputOnForcingGrid. The file isspatial, so no time settings are written.
Code
function writeParameterInputJson(json_path, data_path, variable_attributes; space_dimensions=nothing)
variables = DataStructures.OrderedDict{String,Any}()
for (variable_name, attributes) ∈ variable_attributes
entry = DataStructures.OrderedDict{String,Any}()
merge!(entry, attributes)
entry["source_variable"] = string(variable_name)
variables[string(variable_name)] = entry
end
default_map = DataStructures.OrderedDict{String,Any}("data_path" => data_path, "space_time_type" => "spatial")
if !isnothing(space_dimensions)
default_map["dimension"] = DataStructures.OrderedDict{String,Any}("space" => string.(collect(space_dimensions)))
end
default_map["interpolation_method"] = DataStructures.OrderedDict{String,Any}("space" => "nearest")
parameter_input = DataStructures.OrderedDict{String,Any}("default_parameter_map" => default_map, "variables" => variables)
open(json_path, "w") do io
write(io, json(parameter_input; pretty=true))
end
return json_path
endTypes
GradientParameterUpdater
Sindbad.Setup.GradientParameterUpdater Type
GradientParameterUpdater(parameter_table, parameter_indices_per_model)Holds what updateModels needs to set the parameters of a fixed model structure in a gradient, as in the loss of a hybrid run. Build it once with prepGradientParameterUpdater.
Fields:
parameter_table: the table of the parameters, used to back scale the parameter vectorparameter_indices_per_model: the index of each parameter in the parameter vector, with one element per model in model order, seegetParameterIndicesPerModel
Notes:
Like
ParameterUpdater, the models are paired with their indices by position, so no model is looked up by name and the update is inferred.Unlike
ParameterUpdater, the values are set as they are and not converted to the types of the model fields, so the duals of a gradient reach the models.
ParameterUpdater
Sindbad.Setup.ParameterUpdater Type
ParameterUpdater(parameter_table, parameter_indices_per_model)Holds what updateModels needs to set the parameters of a fixed model structure many times, as in the cost function of an optimization. Build it once with prepParameterUpdater.
Fields:
parameter_table: the table of the parameters, used to back scale the parameter vectorparameter_indices_per_model: the index of each parameter in the parameter vector, with one element per model in model order, seegetParameterIndicesPerModel