Q18ML Model Development
An ML engineer has trained a neural network by using stochastic gradient descent (SGD). The neural network performs poorly on the test set. The values for training loss and validation loss remain high and show an oscillating pattern. The values decrease for a few epochs and then increase for a few epochs before repeating the same cycle. What should the ML engineer do to improve the training process?
← → navigate · a answer
Community votes
Discussion · 6
D 9
Selected Answer: D
The oscillating pattern in the training and validation loss suggests the learning rate is too high. A high learning rate makes the model overshoot the optimal point in the loss landscape, which leads to oscillation instead of convergence. Lowering the learning rate lets the model make smaller, more exact updates to the weights, which improves convergence.
A. Early stopping helps prevent overfitting by stopping training when validation performance no longer improves. However, it does not fix the underlying cause of the oscillating loss.
B. The size of the test set does not influence the training dynamics or loss patterns.
C. Raising the learning rate would make the oscillations worse and keep the model from converging.
D 4
Selected Answer: D
A. No, early stopping is used to prevent overfitting
B. No, increasing test will not help with oscillating loss
C. No, increasing learning rate will make things worse
D. Oscillating loss in training is a sign that the training is not converging, this can happen when learning rate is too high. Reducing learning rate will help here
D 2
Selected Answer: D
oscillating = lower the learning rate
D 2
Selected Answer: D
The oscillating loss during training is a clear sign that the learning rate is too high. Lowering the learning rate will steady the optimization process, letting the model converge smoothly.
D 1
Selected Answer: D
Oscillating patterns in train/validation loss show it's not converging to a minima. A low learning rate will make it converge.
D 1
Decrease the learning rate, as oscillation is an indication of using large learning rate which is avoiding the convergence to a global minima to reduce the loss.