Applied Optimization with MATLAB Programming, 2nd Edition.pdf: A Comprehensive Textbook for Optimization Students and Practitioners
Applied Optimization with MATLAB Programming, 2nd Edition.pdf
Optimization is the process of finding the best solution to a problem among a set of possible alternatives. Optimization problems arise in many fields of science, engineering, business, and management. For example, you may want to minimize the cost of a product, maximize the efficiency of a system, or find the optimal design for a structure.
Applied Optimization with MATLAB Programming, 2nd Edition.pdf
MATLAB is a powerful software tool for numerical computation, data analysis, visualization, and programming. MATLAB can help you solve various types of optimization problems using built-in functions, toolboxes, and other features.
In this article, we will introduce you to the basics of optimization and MATLAB, and show you how to use MATLAB for solving different kinds of optimization problems. We will also cover some advanced topics and applications of optimization with MATLAB that can help you tackle more complex and realistic problems.
Introduction
What is optimization?
Optimization is the process of finding the best solution to a problem among a set of possible alternatives. The best solution is usually defined by some criteria or objective function that measures the quality or performance of a solution. For example, if you want to optimize the design of a car, you may use the fuel consumption or the speed as your objective function.
Optimization problems can be classified into different types depending on the characteristics of the objective function, the constraints, and the variables. Some common types are linear programming, nonlinear programming, integer programming, and multi-objective optimization. We will discuss these types in more detail later.
What is MATLAB?
MATLAB is a software tool for numerical computation, data analysis, visualization, and programming. MATLAB stands for MATrix LABoratory, because it allows you to manipulate matrices and perform various operations on them. MATLAB also supports other data types such as scalars, vectors, arrays, strings, cells, structures, and objects.
MATLAB has a user-friendly interface that lets you interact with the software using commands or scripts. You can also create graphical user interfaces (GUIs) or applications using MATLAB. MATLAB has many built-in functions that can help you perform various tasks such as solving equations, plotting graphs, performing statistical analysis, and more.
Why use MATLAB for optimization?
MATLAB can help you solve various types of optimization problems using built-in functions, toolboxes, and other features. Some of the advantages of using MATLAB for optimization are:
You can easily formulate and solve optimization problems using matrix notation and algebraic expressions.
You can access a wide range of solvers and algorithms for different types of optimization problems.
You can visualize and analyze your results using graphical tools and functions.
You can integrate your optimization code with other MATLAB functions or toolboxes for data processing, simulation, modeling, etc.
You can customize and extend your optimization code using MATLAB programming language or other languages such as C, C++, Java, etc.
Basic concepts and techniques of optimization
Types of optimization problems
Optimization problems can be classified into different types depending on the characteristics of the objective function, the constraints, and the variables. Some common types are:
Linear programming
Linear programming (LP) is a type of optimization problem where the objective function and the constraints are linear functions of the variables. For example, the following is a LP problem:
Minimize: $$f(x) = c^Tx$$
Subject to: $$Ax \leq b$$
Where: $$x = (x_1, x_2, ..., x_n)^T$$ is the vector of decision variables, $$c = (c_1, c_2, ..., c_n)^T$$ is the vector of coefficients of the objective function, $$A$$ is a matrix of coefficients of the constraints, and $$b = (b_1, b_2, ..., b_m)^T$$ is the vector of right-hand sides of the constraints.
LP problems can be solved using various methods such as simplex method, interior-point method, or dual simplex method. MATLAB has a built-in function called linprog that can solve LP problems using different solvers and options.
Nonlinear programming
Nonlinear programming (NLP) is a type of optimization problem where the objective function or the constraints are nonlinear functions of the variables. For example, the following is a NLP problem:
Minimize: $$f(x) = x_1^2 + x_2^2 + x_3^2$$
Subject to: $$x_1 + x_2 + x_3 = 1$$
Where: $$x = (x_1, x_2, x_3)^T$$ is the vector of decision variables.
NLP problems can be solved using various methods such as gradient-based methods, derivative-free methods, or metaheuristics. MATLAB has a built-in function called fmincon that can solve NLP problems using different solvers and options.
Integer programming
Integer programming (IP) is a type of optimization problem where some or all of the variables are required to be integers. For example, the following is a IP problem:
Maximize: $$f(x) = 5x_1 + 7x_2 + 9x_3$$
Subject to: $$x_1 + x_2 + x_3 \leq 10$$
Where: $$x = (x_1, x_2, x_3)^T$$ is the vector of decision variables and $$x_i \in \0, 1\$$ for $$i = 1, 2, 3$$.
IP problems can be solved using various methods such as branch-and-bound method, cutting-plane method, or branch-and-cut method. MATLAB has a built-in function called intlinprog that can solve IP problems using different solvers and options.
Multi-objective optimization
Multi-objective optimization (MOO) is a type of optimization problem where there are more than one objective function to be optimized simultaneously. For example, the following is a MOO problem:
Minimize: $$f(x) = (f_1(x), f_2(x), f_3(x))^T$$
Subject to: $$g(x) \leq 0$$
Where: $$x = (x_1, x_2)^T$$ is the vector of decision variables, $$f_i(x)$$ are the objective functions, and $$g(x)$$ are the constraints.
MOO problems can be solved using various methods such as weighted sum method, epsilon-constraint method, or evolutionary algorithms. MATLAB has a built-in function called fgoalattain that can solve MOO problems using goal attainment method.
Optimization algorithms
or not, and whether they are single-solution or population-based. Some common types are:
Gradient-based methods
Gradient-based methods are optimization algorithms that use the gradient or the derivative of the objective function to guide the search direction. Gradient-based methods can be divided into two categories: line search methods and trust region methods. Line search methods find a suitable step size along a given search direction, while trust region methods find a suitable search direction within a given region around the current point. Some examples of gradient-based methods are steepest descent method, Newton's method, conjugate gradient method, and quasi-Newton method.
Gradient-based methods are efficient and reliable for solving smooth and convex optimization problems, but they may have difficulties in dealing with noisy, discontinuous, or non-convex optimization problems. MATLAB has several built-in functions that can use gradient-based methods to solve optimization problems, such as fminunc, fmincon, fminsearch, and fminbnd.
Derivative-free methods
Derivative-free methods are optimization algorithms that do not use the gradient or the derivative of the objective function to guide the search direction. Derivative-free methods can be divided into two categories: direct search methods and model-based methods. Direct search methods compare the objective function values at different points to determine the search direction, while model-based methods construct an approximation of the objective function using interpolation or regression techniques. Some examples of derivative-free methods are Nelder-Mead simplex method, pattern search method, Powell's method, and trust region reflective method.
Derivative-free methods are useful for solving optimization problems where the objective function is noisy, discontinuous, or non-smooth, or where the derivatives are unavailable or unreliable. MATLAB has several built-in functions that can use derivative-free methods to solve optimization problems, such as fminsearch, fmincon, fminimax, and lsqnonlin.
Metaheuristics
Metaheuristics are optimization algorithms that use randomness or stochasticity to explore the search space and escape from local optima. Metaheuristics can be divided into two categories: single-solution metaheuristics and population-based metaheuristics. Single-solution metaheuristics start from a single point and modify it iteratively using random perturbations or local search techniques, while population-based metaheuristics maintain a set of points and update them using evolutionary or social mechanisms. Some examples of metaheuristics are simulated annealing, tabu search, genetic algorithm, particle swarm optimization, and ant colony optimization.
Metaheuristics are flexible and robust for solving complex and non-convex optimization problems, but they may require more computational resources and tuning efforts than other methods. MATLAB has several built-in functions that can use metaheuristics to solve optimization problems, such as ga, particleswarm, simulannealbnd, and patternsearch.
Optimization toolbox and other MATLAB features for optimization
Optimization toolbox overview
The Optimization Toolbox is a collection of functions that extend the capability of MATLAB for solving various types of optimization problems. The Optimization Toolbox provides:
Functions and options for defining and solving optimization problems.
Solvers and algorithms for different types of optimization problems.
Examples and applications of optimization in various domains.
Functions and options
The Optimization Toolbox provides several functions that can help you define and solve optimization problems. Some of these functions are:
optimproblem: Create an optimization problem object that contains the objective function, the constraints, the variables, and other properties.
optimvar: Create an optimization variable object that represents a decision variable in an optimization problem.
fobjective: Create an objective function object that represents a scalar-valued function to be minimized or maximized in an optimization problem.
fconstraint: Create a constraint object that represents a linear or nonlinear equality or inequality constraint in an optimization problem.
solve: Solve an optimization problem object using a specified solver and options.
optimoptions: Create an options object that contains the settings for a solver or an algorithm.
The Optimization Toolbox also provides several options that can help you control the behavior and performance of a solver or an algorithm. Some of these options are:
Display: Specify the level of output displayed by the solver.
MaxIterations: Specify the maximum number of iterations allowed by the solver.
MaxFunctionEvaluations: Specify the maximum number of function evaluations allowed by the solver.
OptimalityTolerance: Specify the tolerance for the optimality condition of the solution.
ConstraintTolerance: Specify the tolerance for the feasibility condition of the solution.
StepTolerance: Specify the tolerance for the step size of the solution.
Solvers and algorithms
The Optimization Toolbox provides several solvers and algorithms that can solve different types of optimization problems. Some of these solvers and algorithms are:
linprog: Solve linear programming problems using interior-point method or dual simplex method.
quadprog: Solve quadratic programming problems using interior-point method or active-set method.
fmincon: Solve nonlinear programming problems using interior-point method, trust region reflective method, active-set method, or sqp method.
fminunc: Solve unconstrained nonlinear optimization problems using quasi-Newton method or trust region method.
fminbnd: Solve bounded nonlinear optimization problems using golden section search and parabolic interpolation.
fminsearch: Solve unconstrained nonlinear optimization problems using Nelder-Mead simplex method.
fgoalattain: Solve multi-objective optimization problems using goal attainment method.
fminimax: Solve minimax optimization problems using sequential quadratic programming (sqp) method.
lsqnonlin: Solve nonlinear least squares problems using trust region reflective method or Levenberg-Marquardt method.
lsqlin: Solve linear least squares problems using normal equations or qr factorization.
intlinprog: Solve integer linear programming problems using branch-and-bound method.
bintprog: Solve binary integer programming problems using branch-and-bound method.
ga: Solve global optimization problems using genetic algorithm.
particleswarm: Solve global optimization problems using particle swarm optimization.
simulannealbnd: Solve global optimization problems using simulated annealing.
patternsearch: Solve global optimization problems using pattern search.
Examples and applications
The Optimization Toolbox provides several examples and applications of optimization in various domains. Some of these examples and applications are:
Data fitting: Fit a curve or a surface to a set of data points using nonlinear least squares or minimax optimization.
Portfolio optimization: Find the optimal allocation of assets in a portfolio to maximize return or minimize risk using quadratic programming or genetic algorithm.
Traveling salesman problem: Find the shortest tour that visits a set of cities exactly once using integer linear programming or genetic algorithm.
Mechanical design: Find the optimal shape or size of a mechanical component to minimize stress or maximize strength using nonlinear programming or genetic algorithm.
Chemical engineering: Find the optimal operating conditions or parameters of a chemical process to maximize yield or minimize cost using nonlinear programming or genetic algorithm.
Control systems: Find the optimal controller parameters or feedback gains to stabilize or optimize a dynamic system using linear quadratic regulator (lqr) or genetic algorithm.
Other MATLAB features for optimization
Besides the Optimization Toolbox, MATLAB also provides other features that can help you perform optimization tasks. Some of these features are:
Symbolic math toolbox
The Symbolic Math Toolbox is a collection of functions that extend the capability of MATLAB for symbolic computation, such as algebra, calculus, differential equations, etc. The Symbolic Math Toolbox can help you perform optimization tasks such as:
Analytically solve optimization problems using solve, solveInequality, fminimax, etc.
or gradients of objective functions or constraints using diff, gradient, jacobian, etc.
Analytically calculate integrals or areas of objective functions or constraints using int, area, integral, etc.
Analytically calculate limits or asymptotes of objective functions or constraints using limit, asymptote, singularity, etc.
Analytically simplify or manipulate expressions of objective functions or constraints using simplify, expand, factor, etc.
Numerically evaluate expressions of objective functions or constraints using vpa, subs, double, etc.
Plot graphs of objective functions or constraints using fplot, fimplicit, fcontour, etc.
Global optimization toolbox
The Global Optimization Toolbox is a collection of functions that extend the capability of MATLAB for solving global optimization problems. Global optimization problems are optimization problems that have multiple local optima and require finding the global optimum. The Global Optimization Toolbox provides:
Solvers and algorithms for global optimization problems.
Functions and options for defining and solving global optimization problems.
Examples and applications of global optimization in various domains.
Parallel computing toolbox
The Parallel Computing Toolbox is a collection of functions that extend the capability of MATLAB for performing parallel computation, such as using multiple processors, cores, or machines. The Parallel Computing Toolbox can help you perform optimization tasks such as:
Speed up the execution of optimization solvers or algorithms using parallel computing.
Distribute the workload of optimization solvers or algorithms across multiple workers using parallel computing.
Perform parameter tuning or sensitivity analysis of optimization solvers or algorithms using parallel computing.
Solve large-scale or distributed optimization problems using parallel computing.
Advanced topics and applications of optimization with MATLAB
Optimization under uncertainty
Optimization under uncertainty is a type of optimization problem where there is uncertainty in the objective function, the constraints, or the variables. Uncertainty can arise from various sources such as measurement errors, modeling errors, parameter variations, environmental changes, etc. Optimization under uncertainty can be classified into different types depending on how the uncertainty is modeled and handled. Some common types are:
Stochastic programming
Stochastic programming is a type of optimization under uncertainty where the uncertainty is modeled using probability distributions or scenarios. Stochastic programming aims to find a solution that optimizes the expected value or the worst-case value of the objective function under uncertainty. For example, the following is a stochastic programming problem:
Minimize: $$E[f(x,\xi)]$$
Subject to: $$P[g(x,\xi) \leq 0] \geq 1 - \alpha$$
Where: $$x = (x_1, x_2, ..., x_n)^T$$ is the vector of decision variables, $$\xi = (\xi_1, \xi_2, ..., \xi_m)^T$$ is the vector of random variables with known probability distributions, $$f(x,\xi)$$ is the objective function, $$g(x,\xi)$$ are the constraints, $$E[\cdot]$$ denotes the expectation operator, $$P[\cdot]$$ denotes the probability operator, and $$\alpha$$ is a given confidence level.</