Optimization Methods and Options for Tensor Toolbox
Within the optimization-based functions in Tensor Toolbox like cp_opt, we use a variety MATLAB optimization methods from different sources and with different interfaces. For some consisency, these methods are wrapped and given standardized parameters so that it is relatively easy to swap optimization methods. Here we describe the methods, the standardized parameters, installation instructions (if needed), and the mapping between the standardized parameters and the original ones.
- The optimization method is usually specified via the name-value pair with name 'method'.
- The standardized parameters listed below are additional name-value pairs that can be passed directly to the calling routine.
- The standardized parameters marked with asterisks are not intended to be modified by the user but instead set by the calling routine, such as cp_opt.
For more information on the details of these methods, see Developer Information for Optimization Methods in Tensor Toolbox.
Contents
lbfgsb: Limited-Memory Quasi-Newton with Bound Constraints
Source. Included with Tensor Toolbox, adopted without any modification from L-BFGS-B by Stephen Becker.
Standardized Parameters.
| Name | Description | Default |
| lower | Lower bounds, can be vector or scalar | -Inf |
| upper | Upper bounds, can be vector or scalar | +Inf |
| maxiters | Max outer iterations | 1000 |
| printitn | Printing frequency by iteration (0=no output) | 1 |
| m | Limited-memory parameter | 5 |
| subiters | Controls maximum calls to function-gradient evalations | 10 |
| ftol | Stopping condition based on relative function change | 1e-10 |
| gtol | Stopping condition based on gradient norm | 1e-5 |
| mdesc (*) | Method description printed out before run | 'L-BFGS-B Optimization' |
| xdesc (*) | Variable description | auto-generated |
Installation Instructions. Nothing should be required as this code is contained in libraries/lbfgs. However, if needed, please see the GitHub site for full details on references, installation, etc. Here we provide cursory instructions for installation:
- Download the zip file <https://github.com/stephenbeckr/L-BFGS-B-C/archive/master.zip>
- Unzip and goto the Matlab/ subdirectoy with MATLAB
- Type compile_mex
- Add this directory to your saved path!
Mapping of Standardized parameters. The wrapper for this method is tt_opt_lbfgsb in the Tensor Toolbox. Notes regarding mappings to the parameters of Becker's L-BFGS-B code:
- maxIts maps to maxiters and the default is increased from 100 to 1000
- printEvery maps to printitn
- maxTotalIts is set to maxiters*subiters and this effectively changes the default from 5000 to 10000
- factr is set to ftol / eps and this effectively changes the default from 1e7 to 4.5e5
- pgtol is set to gtol
lbfgs: Limited-Memory Quasi-Newton
Source. lbfgs method in Poblano Toolbox, v1.2.
Standardized Parameters.
| Name | Description | Default |
| maxiters | Max outer iterations | 1000 |
| printitn | Printing frequency by iteration (0=no output) | 1 |
| m | Limited-memory parameter | 5 |
| subiters | Controls maximum calls to function-gradient evalations | 10 |
| ftol | Stopping condition based on relative function change | 1e-10 |
| gtol | Stopping condition based on gradient norm | 1e-5 |
| mdesc (*) | Method description printed out before run | 'Poblano L-BFGS Optimization' |
| xdesc (*) | Variable description | auto-generated |
Installation Instructions. Download and install Poblano Toolbox, v1.2. Please see that web page for full details on references, installation, etc. Here we provide cursory instructions for installation:
- Download the zip file <https://github.com/sandialabs/poblano_toolbox/archive/v1.2.zip>
- Unzip and goto the poblano_toolbox-1.2/ subdirectoy within MATLAB
- Type install_poblano to save this directory to your path
Mapping of Standardized parameters. The wrapper for this method is tt_opt_lbfgs in the Tensor Toolbox. Notes regarding mappings to the parameters of Poblano's L-BFGS code:
- MaxIters maps to maxiters
- Display maps to printitn
- MaxFuncEvals is set to maxiters*subiters
- RelFuncTol is set to ftol
- StopTol is set to gtol
fminunc: Optimizaton Toolbox Unconstrained Optimization
Source. MATLAB Optimization Toolbox.
Standardized Parameters.
| Name | Description | Default |
| maxiters | Max outer iterations | 1000 |
| printitn | Display (0=no output) | 1 |
| subiters | Controls maximum calls to function-gradient evalations | 10 |
| gtol | Stopping condition based on gradient norm | 1e-5 |
| mdesc (*) | Method description printed out before run | 'Unconstrained Optimization (via Optimization Toolbox)' |
| xdesc (*) | Variable description | auto-generated |
Installation Instructions. Install MATLAB Optimization Toolbox.
Mapping of Standardized parameters.
- MaxFunctionEvaluations is set to maxiters*subiters
- MaxIterations is set to maxiters
- OptimalityTolerance is set to gtol
- Display is set of 'off' if printitn = 0, 'iter' if printitn = 1, and 'final' if printitn > 1.
fmincon: Optimizaton Toolbox Constrained Optimization
Source. MATLAB Optimization Toolbox.
Standardized Parameters. Same as for fminunc plus the following:
| Name | Description | Default |
| lower | Lower bounds, can be vector or scalar | -Inf |
| upper | Upper bounds, can be vector or scalar | +Inf |
| mdesc (*) | Method description printed out before run | 'Constrained Optimization (via Optimization Toolbox)' |
Installation Instructions. Install MATLAB Optimization Toolbox.
Mapping of Standardized parameters.
- The lower bound (7th input to fmincon) is set to [] if lower = Inf. If it's a scalar, then the lower bound is set to a vector of appropriate length with the scalar in every position. Otherwise, it's set to the provided vector.
- The upper bound (8th input) is analogous.
adam: Stochastic Gradient Descent with Momentum
Source. This is our own implementation of Adam. A failed epoch is one where the function value does not decrease. After a failed epoch, the method either reduces the learning rate (by decay) or exits (once the number of failed epochs exceeds maxfails).
Standardized Parameters.
| Name | Description | Default |
| lower | Lower bounds, can be vector or scalar | -Inf |
| subiters | Number of iterations per epoch | 100 |
| maxiters | Maximum number of epochs | 100 |
| rate | Initial learning rate | 1e-2 |
| maxfails | Maximum number of failed epochs | 1 |
| decay | Decay of learning rate after failed epoch | 0.1 |
| backup | Revert to end of previous epoch after failure | true |
| ftol | Quit if function value goes below this value | -Inf |
| beta1 | Adam parameter | 0.9 |
| beta2 | Adam parameter | 0.999 |
| epsilon | Adam parameter | 1e-8 |
| printitn | Printing frequency by epoch (0=no output) | 1 |
| state (*) | State of random number generator | current state |
| mdesc (*) | Method description printed out before run | 'Adam Stochastic Optimization' |
| xdesc (*) | Variable description | auto-generated |
| fdesc (*) | Description of (approximate) function computation | none |
| gdesc (*) | Description of stochastic gradient computation | none |
| fexact (*) | Boolean if function is computed exactly | true |