Smooth maximum

In mathematics, a smooth maximum of an indexed family x₁, ..., x_n of numbers is a smooth approximation to the maximum function ${\displaystyle \max(x_{1},\ldots ,x_{n}),}$ meaning a parametric family of functions ${\displaystyle m_{\alpha }(x_{1},\ldots ,x_{n})}$ such that for every ⁠${\displaystyle \alpha }$⁠, the function ⁠${\displaystyle m_{\alpha }}$⁠ is smooth, and the family converges to the maximum function ⁠${\displaystyle m_{\alpha }\to \max }$⁠ as ⁠${\displaystyle \alpha \to \infty }$⁠. The concept of smooth minimum is similarly defined. In many cases, a single family approximates both: maximum as the parameter goes to positive infinity, minimum as the parameter goes to negative infinity; in symbols, ⁠${\displaystyle m_{\alpha }\to \max }$⁠ as ⁠${\displaystyle \alpha \to \infty }$⁠ and ⁠${\displaystyle m_{\alpha }\to \min }$⁠ as ⁠${\displaystyle \alpha \to -\infty }$⁠. The term can also be used loosely for a specific smooth function that behaves similarly to a maximum, without necessarily being part of a parametrized family.

For large positive values of the parameter ${\displaystyle \alpha >0}$, the following formulation is a smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

${\displaystyle {\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n})={\frac {\sum _{i=1}^{n}x_{i}e^{\alpha x_{i}}}{\sum _{i=1}^{n}e^{\alpha x_{i}}}}}$

${\displaystyle {\mathcal {S}}_{\alpha }}$ has the following properties:

1. ${\displaystyle {\mathcal {S}}_{\alpha }\to \max }$ as ${\displaystyle \alpha \to \infty }$
2. ${\displaystyle {\mathcal {S}}_{0}}$ is the arithmetic mean of its inputs
3. ${\displaystyle {\mathcal {S}}_{\alpha }\to \min }$ as ${\displaystyle \alpha \to -\infty }$

The gradient of ${\displaystyle {\mathcal {S}}_{\alpha }}$ is closely related to softmax and is given by

${\displaystyle \nabla _{x_{i}}{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n})={\frac {e^{\alpha x_{i}}}{\sum _{j=1}^{n}e^{\alpha x_{j}}}}[1+\alpha (x_{i}-{\mathcal {S}}_{\alpha }(x_{1},\ldots ,x_{n}))].}$

This makes the softmax function useful for optimization techniques that use gradient descent.

This operator is sometimes called the Boltzmann operator,¹ after the Boltzmann distribution.

Another smooth maximum is LogSumExp:

${\displaystyle \mathrm {LSE} _{\alpha }(x_{1},\ldots ,x_{n})={\frac {1}{\alpha }}\log \sum _{i=1}^{n}\exp \alpha x_{i}}$

This can also be normalized if the ${\displaystyle x_{i}}$ are all non-negative, yielding a function with domain ${\displaystyle [0,\infty )^{n}}$ and range ${\displaystyle [0,\infty )}$:

${\displaystyle g(x_{1},\ldots ,x_{n})=\log \left(\sum _{i=1}^{n}\exp x_{i}-(n-1)\right)}$

The ${\displaystyle (n-1)}$ term corrects for the fact that ${\displaystyle \exp(0)=1}$ by canceling out all but one zero exponential, and ${\displaystyle \log 1=0}$ if all ${\displaystyle x_{i}}$ are zero.

The mellowmax operator¹ is defined as follows:

${\displaystyle \mathrm {mm} _{\alpha }(x)={\frac {1}{\alpha }}\log {\frac {1}{n}}\sum _{i=1}^{n}\exp \alpha x_{i}}$

It is a non-expansive operator. As ${\displaystyle \alpha \to \infty }$, it acts like a maximum. As ${\displaystyle \alpha \to 0}$, it acts like an arithmetic mean. As ${\displaystyle \alpha \to -\infty }$, it acts like a minimum. This operator can be viewed as a particular instantiation of the quasi-arithmetic mean. It can also be derived from information theoretical principles as a way of regularizing policies with a cost function defined by KL divergence. The operator has previously been utilized in other areas, such as power engineering.²

LogSumExp and Mellowmax are the same function differing by a constant ${\displaystyle {\frac {\log {n}}{\alpha }}}$. LogSumExp is always larger than the true max, differing at most from the true max by ${\displaystyle {\frac {\log {n}}{\alpha }}}$ in the case where all n arguments are equal and being exactly equal to the true max when all but one argument is ${\displaystyle -\infty }$. Similarly, Mellowmax is always less than the true max, differing at most from the true max by ${\displaystyle {\frac {\log {n}}{\alpha }}}$ in the case where all but one argument is ${\displaystyle -\infty }$ and being exactly equal to the true max when all n arguments are equal.

Another smooth maximum is the p-norm:

${\displaystyle \|(x_{1},\ldots ,x_{n})\|_{p}=\left(\sum _{i=1}^{n}|x_{i}|^{p}\right)^{\frac {1}{p}}}$

which converges to ${\displaystyle \|(x_{1},\ldots ,x_{n})\|_{\infty }=\max _{1\leq i\leq n}|x_{i}|}$ as ${\displaystyle p\to \infty }$.

An advantage of the p-norm is that it is a norm. As such it is scale invariant (homogeneous): ${\displaystyle \|(\lambda x_{1},\ldots ,\lambda x_{n})\|_{p}=|\lambda |\cdot \|(x_{1},\ldots ,x_{n})\|_{p}}$, and it satisfies the triangle inequality.

The following binary operator is called the Smooth Maximum Unit (SMU):³

${\displaystyle {\begin{aligned}\textstyle \max _{\varepsilon }(a,b)&={\frac {a+b+|a-b|_{\varepsilon }}{2}}\\&={\frac {a+b+{\sqrt {(a-b)^{2}+\varepsilon }}}{2}}\end{aligned}}}$

where ${\displaystyle \varepsilon \geq 0}$ is a parameter. As ${\displaystyle \varepsilon \to 0}$, ${\displaystyle |\cdot |_{\varepsilon }\to |\cdot |}$ and thus ${\displaystyle \textstyle \max _{\varepsilon }\to \max }$.