

30072020 MR don't add nesterovs momentum to TD batch, can be achieved by increased eta ,decreased mom:
https://stats.stackexchange.com/questions/179915/whats-the-difference-between-momentum-based-gradient-descent-and-nesterovs-acc
p_{new} = p + self.momentum^2 * m - (1 + self.momentum) * lr * g
where m is the previous weight update, and g is the current gradient with respect to the parameters p, lr is the learning rate, and self.momentum is a constant.
mom_old=0.75
mom_new= 0.75* 0.75 = 0.5625
eta_old=0.00008
eta_new=0.00008*(1+0.75)=0.00014

Add RMSprop
defaults eps=1e-7;eta=0.05;mu=0.995; mu1=1-mu
msg .. moving average of squared gradient

msg*=mu;
msg+=mu1*g*g;
p+=eta*g/(sqrt(msg)+eps);
g=0;


RMSprop with decay
g-=decay*p
msg*=mu;
msg+=mu1*g*g;
p+=eta*g/(sqrt(msg)+eps);
g=0;


RMSprop with momentum
https://github.com/coells/100days/blob/master/day%2069%20-%20rmsprop.ipynb
def rmsprop_momentum(F, dF, x, steps=100, lr=0.001, decay=.9, eps=1e-8, mu=.9):
    loss = []
    dx_mean_sqr = np.zeros(x.shape, dtype=float)
    momentum = np.zeros(x.shape, dtype=float)

    for _ in range(steps):
        dx = dF(x)
        dx_mean_sqr = decay * dx_mean_sqr + (1 - decay) * dx ** 2
        momentum = mu * momentum + lr * dx / (np.sqrt(dx_mean_sqr) + eps)
        x -= momentum
        loss.append(F(x))

    return x, loss
