Find all possible functions \(f\) such that \(f(x) = ax + b\) and \(f\left(f\left(f(x)\right)\right) = bx - a^2\) with \(a \neq 0\), \(b \neq 0\), \(a\), \(b \in \mathbb{R}\).
Solution
The solutions to the equation
\[a^2(a+1)\left(a^2 + 1\right) = 0\]
are \(a = 0\) or \(a = -1\). Note that \(a^2 + 1 \neq 0\) for \(a \in \mathbb{R}\).
However, \(a \neq 0\), so \(a = -1\).
Substituting \(a = -1\) into \((1)\), we get
Therefore, the only possible function \(f\), which satisfies the conditions, is \(f(x) = -x - 1\).