Ctrl+K

init: Initialisation schemes#

Parameter initialisation schemes.

The init submodule contains functionality for initialising (neural network) module parameters to reasonable defaults.

In the differentiable programming setting, initialisation provides a path for integrating domain expertise into neural network models. The initialisation schemes and routines provided here present a transparent framework for incorporating the discipline’s existing best practices as starting points for a differentiable workflow.

For example, the routines here enable the linear atlas layer to be initialised as most available volumetric and surface-based neuroimaging parcellations. Modifications can be applied easily to promote better learning signals. For the case of the atlas layer, for instance, hard parcels can be smoothed (in either Euclidean or spherical topology ) or constrained to the probability simplex (using a softmax domain transformation) to change the properties of the gradients they receive.

Also available are more general initialisation schemes for use cases where a clean slate is desired as a starting point. For example, a random Dirichlet initialisation , when combined with a probability simplex projection , lends columns in a parcellation matrix the intuitive interpretation of probability distributions over parcels.

Most initialisation scheme classes (eventually, all) can be combined with a parameter mapping. If a parameter mapping is used to initialise a compatible module’s parameters, the parameters are internally stored by the module as “original parameters” and then transformed through the mapping before they interact with inputs. Some schemes are paired with a mapping by default. For instance, Dirichlet initialisation is by default paired with a probability simplex projection domain to constrain Dirichlet-initialised weights to always be valid probability distributions.

Warning

Any and all APIs here are experimental and subject to change. Test coverage is inadequate and extreme discretion is warranted when using this functionality. Please report any bugs by opening an issue.

Contents:

Show Source