python - Validation loss hitting minimum then increasing

Question

As can be seen, at around epoch 60, my validation loss starts to increase while my validation accuracy remains the same. It seems like it's beginning to overfit at around that time, but wouldn't the training loss continue to decrease to nearly zero if it's simply memorizing my training data? My model also seems very small for it to overfit (I'm trying to classify FFT data). Is there something I'm blatantly doing wrong?

Here is my model:

model = Sequential()
model.add(Conv1D(filters = 32, kernel_size = 3, activation = 'relu', input_shape = (size, 1)))
model.add(Dropout(dropout))
model.add(GlobalMaxPooling1D())
model.add(Dropout(dropout))
model.add(Dense(64, activation='relu'))
model.add(Dense(1, activation='sigmoid')) #Output layer

My training data shape:

x: (1038, 2206)

y: (1038, 1)

My parameters:

EPOCHS = 300

LR = 1e-3

DROPOUT = 0.5

BATCH_SIZE = 128

On a side note, my validation accuracy is around 98%, yet when I test my model on the same validation data, I get the incorrect output. I don't believe my validation data is incorrectly made because I made it in exactly the same way as my training data.

Answer 1

You plots of loss look to be classical over fitting which is strange given the simplicity of the model. One thing in the model that I would change is you have

model.add(Dropout(dropout))
model.add(GlobalMaxPooling1D())
model.add(Dropout(dropout))

Global max pooling has no activation function so I do not see the need to the second dropout layer. Actually with a dropout rate of .5 I am surprised your model trains as well as it does. You mention creating your validation set so I assume YOU selected the validation data. It is usually better to create the validation data via a random selection from the full data set using something like sklearn's train_test_split function. With the amount of dropout you have it makes sense your model might not reach 100% training accuracy. One thing I would try is to use an adjustable learning rate using the Keras callback ReduceLROnPlateau. Set it up to monitor the validation loss. If the loss fails to reduce for a 'patience' number of epochs the learning rate will be reduce by a 'factor' where factor is a value less than 1.0. The documentation is located here.Below is my recommendation for the code.

lr_adjust= tf.keras.callbacks.ReduceLROnPlateau(
    monitor="val_loss", factor=0.5, patience=2,verbose=1,mode="auto")

Now in model.fit add callbacks=[lr_adjust] You do not show your model.compile code but you might try using a different optimizer to see if it has an effect. I recommend the adam optimizer. What I suspect is happening is that the probability distribution of your test set is significantly different than that of your training and validation sets. Given you created the later synthetically while your test data is "real life data" this makes the case of dissimilar probability distributions likely.