DOC: optimize.minimize: recipe for duplicating work when calling `fun`, `jac`, and `hess` at same `x`

Here is a simple optimization example (Poisson regression):

# loss function and gradient
def f(b):
    Xb = np.dot(X, b)
    exp_Xb = np.exp(Xb)
    loss = exp_Xb.sum() - np.dot(y, Xb)
    grad = np.dot(X.T, exp_Xb - y)
    return loss, grad

# hessian
def hess(b):
    return np.dot(X.T, np.exp(np.dot(X, b))[:, None]*X)

# optimize
result = minimize(f, np.zeros(p), jac=True, hess=hess, method='newton-cg')

This works, but since np.exp(np.dot(X, b)) is already computed inside f(b), it would be more efficient and more readable to be able to do:

# loss function and gradient
def f(b):
    Xb = np.dot(X, b)
    exp_Xb = np.exp(Xb)
    loss = exp_Xb.sum() - np.dot(y, Xb)
    grad = np.dot(X.T, exp_Xb - y)
    hess = np.dot(X.T, exp_Xb[:, None] * X) 
    return loss, grad, hess

# optimize
result = minimize(f, np.zeros(p), jac=True, hess=True, method='newton-cg')

But this doesn’t seem supported, right now I get:

TypeError: 'bool' object is not callable

[antonior92]

Do you think that there is only marginal performance improvements to be had along these lines?

To be completely sincere, I think there is not much to be gained in term of performance along these lines. And I think it give the user the false impresion that he is actually getting a better performace while doing this

The way of really getting the best performance is something like:

class ObjectiveWrapper:
    def __init__(self):
        self.common_computations= None
        self.x = None

    def core_computations(self, x, *args):
        if not np.array_equal(x, self.x):
            # ... compute common_computations
            self.common_computations = common_computations
         return self.common_computation

    def func(self, x, *args):
        core_computations = self.core_computations(x, *args)
        # ... compute func
        return func

    def grad(self, x, *args):
        core_computations = self.core_computations(x, *args)
        # ... compute grad
        return grad

    def hess(self, x, *args):
        core_computations = self.core_computations(x, *args)
        # ... compute hess
        return hess

But this would need to be done for your specific problem.

But in that case you’d also not need to recompute the gradient every time the objective is called.

Yes, to be completely since I don’t think the option jac=True is that great of a option neither

---

In summary, I think we would make the interface more complicated. It would give the users the impression that they would get some improvement in performance, while I don’t think it will make much of a difference.

Maybe a best option would be to include in the documentation a paragraph or two explaining how to do the above trick I just mentioned

[mdhaber]

@antonior92 quick question: core_computations in your example above should have self.x = x at the end, right?

I think this should go in the tutorial because it is relevant to many optimize functions.

@AtsushiSakai you’ve written some tutorials for optimize. Would you be willing to submit a PR for this? I think we can merge it quite quickly.