Parametric runs and optimization

IDA Parametric Runs is a versatile tool that can be used in any IDA application and is automatically installed with the Expert edition of IDA ICE. It allows

• running a series of simulations with systematic variation of selected parameters.
• either selecting or creating your own outcomes of one or more simulation types.
• selecting how the parameters are to be varied:
  • Brute force-based runs of all possible combinations
  • Single parameter sensitivity analysis
  • Monte-Carlo randomized simulations
  • Single- and multi-objective optimization
• different visualizations of the simulation results

The optimization algorithm available in Parametric runs is called AutoMOO. AutoMOO is primarily based on the Bayesian optimization functionality.

Access to Parametric Runs cases

To create a new or open an existing parametric run case, select "Parametric runs" from Tools menu.

To rename or delete or copy an existing a parametric runs case, select "Tools" > "Parametric runs" > "Organize". 

Main settings

The main settings can be found at the top of the "General" tab within the parametric runs form

Name

The default name is "ParmRun_1". It can be changed to any name to identify a parametric runs case, if there are several in the same model.

Description

Use this field for instructions, reminders, descriptions of your intention, etc.

Simulation type

Select the type of simulation to run with the parameter variation. The default type is "Energy (whole year)", but you can select more than one simulation type by selecting "Multiple simulations".

Keep generated models

Default “No” saves all added output results to Parametric runs and will not store any result from the simulation cases.  

Selecting "Yes" it will save all calculated ICE simulation cases and enables you if you want to look at detailed additional results beyond the output values that you specified. Note, however, that this setting can easily fill your hard drive.

Clear results from previous runs

Default “Yes” will remove all former simulation inputs and outputs and will start with a clear list or parametric runs.

Selecting "No" keeps the earlier delivered results in place.

Preparing input

In a new parametric runs case, the table of Input is empty. To add parameters, do one of the following:

• Select a parameter or a variable in any input table or form within your IDA model and choose "Add to ... <Parametric Runs case name>" from its context menu (accessible via right-click on the variable)
• Drag and drop a parameter or variable from the outline view of your IDA model to the "Input" table.
• Click "+" and select "browse" to choose a parameter or a variable
• Click "+" and select "custom" to add a custom Input (expert knowhow needed)

For variables, the start value can be treated as a parameter.

It is also possible to construct additional input parameters and let other parameters depend on them by Graphical Scripts.

To change the order of the inputs, the arrows ↑ and ↓ can be used. The “+” is an adder and “x” deletes selected inputs.

There are two ways to edit the input parameters: either by editing the entries in the table directly, or by selecting one input in the table and clicking the "edit" button to open the dialog shown below.

Name

You can rename the added parameters. These names will occur in your parametric runs result table. Only editable from the input table.

Value

Only used in Sensitivity analysis as the base value of the parameter and in Monte-Carlo runs with triangle distribution (see below) to define the peak. Note that it can differ from the value defined in the model itself, even if it is initially taken from there.

Unit

The unit is defined by the parameter and cannot be changed here.

OK range

The range is defined either as an interval of permissible values (called "Range" in the edit dialog) defined by a minimum and maximum value, or a list of permissible values (called as "Fixed set" in the Edit dialog). In the input table, these are represented in the following way:

• [min max]: Two numerical values in square brackets gives the minimum and maximum value that this parameter can be given within all parametric runs types. It is automatically generated by right-clicking “Add to..ParmRun” but can be typed in by the user as well.
• (v1 v2 … vn): Any number of numerical values in round brackets gives a list of allowed discrete values that can be used within all parametric runs types. This type of input must be typed in by the user and eliminates Precision and Resolution.
• <x item(s)>: Number of items in pointy brackets indicate that the parameter itself is discrete and items can be selected from a list of available resource objects that can be used within all parametric runs types.This type is added by “Add to ParmRun” and cannot be typed. Typical example: glazing type.

Range-Type

There are two options to further specify the range for real-valued input parameters which do not have a discrete range ("Range" rather than "Fixed set" in the edit dialog). Either the distribution and resolution is set, or a fixed precision is used. Note that integer-valued parameters do not have these options and that Monte-Carlo runs do not support fixed precision. These options are explained below.

Distribution

This is used for Monte-Carlo, brute force runs ("Run all") and in optimization. For Monte-Carlo runs, it defines the probability distribution used to select the values between the minimum and maximum; in run all, it defines how the number of points defined by the Resolution (see below) are distributed in the range; in optimization, it defines a rescaling of the input. Note that Log-Uniform only can be used for positive parameters and that triangle distribution is not available in optimization (uniform distribution is assumed instead).

Resolution or Number of points

This is only used for Brute force runs ("Run all"). It gives the number of values that are considered between the minimum and the maximum value, with the extreme ends counted in as well.

Precision

Defines the number of decimal places of the parameter in Brute force ("Run all") and optimization runs. Does not take effect with Run all Uniform distribution. Can be set as negative as well for optimization cases to round up the input.

Target

Gives the outline path of the input parameter, automatically generated and typically not needed to edit.

Description

This field can be used to explain input parameters that are not necessarily self-explanatory.

Preparing output

The results items are added in the same way as the input items: right-clicking “Add to ParmRun”. Additionally, outputs from HTML reports can be added, such as the Traded or Systems Energy report, Input Data Report, LCA calculation report etc.

It is also possible to construct additional post-processed results in Graphical Scripts and add those to the outputs.

To change the order of the outputs, the arrows ↑ and ↓ can be used. The “+” is an adder and “x” deletes selected outputs.

Name

You can rename the added parameters. These names will occur in your parametric runs result table.

Target

Gives the outline path of the output parameter, automatically generated and typically not needed to edit.

Restype

This column is only shown when multiple simulation types are selected in the main settings of the parametric runs case. The simulation from which the value is to be taken must then be determined for each parameter separately.

Function

Defines a lisp-function that is applied to the parameter. For outputs from reports, this function is filled in automatically when adding the output and typically not needed to edit.

Role

This function is only used for optimization runs and introduces additional options: scaling and constraint. The role specifies how the result is used:

• MIN | MAX: as an optimization target (Minimum or Maximum value to be found, ie. (minimizing or maximizing the output),
• CONSTRAIN: as a soft constraint (the case result outside of one of the constraint band will be still presented in case it is accepted in other output constraints)
• SHOW: not used for optimization but shown in the result diagram.

Min | Max

Used in optimization for two quite different purposes:

• As constraint: Set the Role to CONSTRAINT and define the feasible ranges at the min- and/or max values. Violating the default constrain will add penalty to each objective functions: the penalty is calculated based on the squared distance deviation.
  More information is available here.
• As scaling: Set the Role as Min or Max and define the min- and max values: this way they will represent the relative scale of the objective functions, however the user will not notice any difference in the plotting or in Result tab. Care to define all of the targets for scaling!
  Note: this is NOT to serve as a constraint function anymore! 

To combine constraints with scaling (formulating the same order of magnitude for competing objective functions), a user-generated Graphical Script can be useful.

Remember not to change the output name after adding from a Graphical Script!

An example of combined scaling and constrained minimizing problem, with scaled maximizing:

Description

This field can be used to explain output parameters that are not necessarily self-explanatory.

Runs running in parallel

Before launching parametric runs, you should be aware of the number of runs that will be executed. The simulations are performed by child processes, executed by the physical processors of the local or cloud computer, where you launch the runs from. It is recommended to adjust the number of processors from "Options" > "Preferences..." > "Simulation" > "No. of parallel processes to start". This setting can also be reached via the options button "Optimization" (see below).

As a rule of thumb, you need to calculate that each child is going to take around 2 processes and then you need to leave some (one or two) free for your computer’s additional functions.

Options

The batch size defines the size of the batch of simulations AutoMoo uses as it is iteratively running simulations and suggesting new sets of parameters to evaluate. By default, it is set equal to the number of available processes.

Manage Runs

In case of mistakes or problems detected during the run, it is possible to cancel the runs immediately. To do so, click "Child processes" and chose "Kill child processes". Additionally, after finished or killed parametric runs, the children can be closed with the same button.

Run simulations

Click "Monte-Carlo" to run the user-defined number of simulations with random choice of parameter values. The distribution of parameter values is given in the columns "Range" and "Distribution" (see "Preparing input" above). Precision setting does not affect Monte-Carlo.

Click "Optimization" to run the optimization runs. The total number of simulation runs is defined by the user in the numeric input field (100 by default).

Additional Runs

Click "Sensitivity analysis" to start runs with all parameters set on "Value" (see "Preparing input" above) as a base run and with additional runs differing only in one of the input parameters changed either to its minimum or its maximum value within the OK range. For a discrete parameter, one simulation run is performed for every value in the list. All other parameters keep the base value.

Click "All" to run all defined combinations of parameter values. These are brute force-based simulation runs and the number of simulations will be defined either by the "Resolution" number (101 by default) or the "Precision" depending on the "Range-Type" for each parameter. Note that the number of simulations grows exponentially with the number of parameters.

Results

Summary

This table shows the values of all input and output variables in all simulations that were successfully executed. Failed simulations will be marked as FAILED, an interrupted run will be marked as ABORTED.

Double-click on a row to open the case. If the simulated cases were saved (option "Keep generated models" = "Yes"), the saved case will be loaded and contain all simulation results. Otherwise, the case will be recreated without results.

Diagrams

To see the results graphically, open a diagram or click "+" to create a new one. This will open the "Design diagram" dialog, where you can define how to show input and output data.

Click the cell "Show-As" of one of the parameters to select its visualization mode. Do the following:

• Always select "Y-AXIS" for one parameter.
• Optionally select "X-AXIS" for another parameter. Default presentation: x-axis will be the numbers of the runs in the order when they were executed.
• Optionally select "Area" for a remaining parameter. Default presentation: all markers will have the same size.
• Optionally select "Color" for a remaining parameter. Default presentation: all markers will be red.
• Optionally select "Shape" for a remaining discrete parameter. Default presentation: all markers will be round.
• To exclude a selected parameter from being shown, select "Hide"
• You can delete diagrams from the created diagram list via right-click.

When a diagram is open, point with mouse the center of a spot representing one simulation run to see its details.

Explore...

This button opens a "parallel coordinates plot" graph where every executed run is visualized by one line connecting all input and output values. You can limit the results by a vertical slider on the axes to both filter and highlight the results.