Randomize training folds?

[TheresaSchmidt]

Current state: Each fold trains on 9 dishes and tests on 1 dish. The order of the dishes is alphabetical.
Suggestions: Would it make more sense to randomize the order of the dishes?
Code reference:

alignment-models/training_testing.py

Line 656 in a4ef3ce

dish_list.sort()

[TheresaSchmidt]

Going one step further by randomizing the order of recipe pairs in the training split would also be an option.

[irisferrazzo]

If we would consider the fact of not maintaining the one-dish-per-fold structure, another idea could be to put all dishes in one corpus, to section this corpus for train-dev-test, and to train the model on different batches, also by trying several batch sizes out. We could also randomize the sub-corpora before running the model (by maintaining the content of the sub-corpora constant).

I don't know whether the order of the dishes was intended to better grasp the per-dish-training-accuracy, though. I'm also not sure in terms of runtime how the training would be affected by this change.

[TheresaSchmidt]

I'm also not sure in terms of runtime how the training would be affected by this change.

I can't imagine the runtime would increase significantly. And as we said in last week's meeting: we don't need to worry about runtime too much.

[TheresaSchmidt]

Summary from meeting with Arne:

1. One dish per fold: better estimate of real accuracy on unknown data.
2. Randomization within training split could be interesting. Prediction: slightly better accuracy (not too much though because each piece of data is seen several times in the overall training process due to cross validation). Currently, we get point estimates for the accuracies of each fold. With randomization we would get accuracy distributions.
3. Sorting the dishes and recipes alphabetically was not motivated as a design feature; was just simple and easy and shouldn't really make a difference compared to random order.