Summary
Optional follow-up to #1077. The transfer-learning splitter is a standalone scikit-learn cross-validator and already works on its own (for train, test in CrossSubjectTransferSplitter(...).split(y, metadata)), so it needs no change here to be usable. This issue is only about the convenience of running it through CrossSubjectEvaluation, to reuse its caching, results DB, parallelism, and scoring. That path is currently blocked because the top-level splitter is hardcoded.
Not a prerequisite for the splitter PR; track separately.
Current behaviour
Every evaluation builds its top-level splitter in _create_splitter() and passes cv_class/cv_kwargs only to the inner cross-validator, not to the splitter itself:
WithinSessionEvaluation -> WithinSessionSplitter (evaluations.py:86)
CrossSessionEvaluation -> CrossSessionSplitter (evaluations.py:299)
CrossSubjectEvaluation -> CrossSubjectSplitter (evaluations.py:479)
WithinSubjectEvaluation -> WithinSubjectSplitter (evaluations.py:660)
def _create_splitter(self):
...
cv_class, cv_kwargs = self._resolve_cv(default_class, default_kwargs)
return CrossSubjectSplitter(
cv_class=cv_class, random_state=self.random_state, **cv_kwargs
)
And the runner does self.cv = self._create_splitter() (base.py:758-764), with no public hook to inject a splitter. So CrossSubjectEvaluation(cv_class=CrossSubjectTransferSplitter, ...) does not swap the splitter; cv_class is forwarded into CrossSubjectSplitter as its inner fold strategy, which is not what a transfer setup needs.
Proposal
Let CrossSubjectEvaluation accept and use a top-level splitter, keeping the scikit-learn contract (split(y, metadata) -> (train_idx, test_idx)) untouched so caching, scoring, and result handling stay as-is. Two options, smallest first:
-
Allow passing a splitter instance. If cv (or a splitter= argument) is a BaseCrossValidator that yields (train_idx, test_idx) over metadata, use it directly in _create_splitter() instead of hardcoding CrossSubjectSplitter. This makes CrossSubjectTransferSplitter usable today and is additive.
-
Let cv_class select the top-level splitter when it is a MOABB splitter (detected by its split(y, metadata) signature), falling back to current behaviour otherwise.
I lean towards option 1: it is the least surprising, does not change any default, and keeps the transfer splitter a plain cross-validator rather than a new evaluation engine. Background and the splitter sketch are in #1077.
Scope
Summary
Optional follow-up to #1077. The transfer-learning splitter is a standalone scikit-learn cross-validator and already works on its own (
for train, test in CrossSubjectTransferSplitter(...).split(y, metadata)), so it needs no change here to be usable. This issue is only about the convenience of running it throughCrossSubjectEvaluation, to reuse its caching, results DB, parallelism, and scoring. That path is currently blocked because the top-level splitter is hardcoded.Not a prerequisite for the splitter PR; track separately.
Current behaviour
Every evaluation builds its top-level splitter in
_create_splitter()and passescv_class/cv_kwargsonly to the inner cross-validator, not to the splitter itself:WithinSessionEvaluation->WithinSessionSplitter(evaluations.py:86)CrossSessionEvaluation->CrossSessionSplitter(evaluations.py:299)CrossSubjectEvaluation->CrossSubjectSplitter(evaluations.py:479)WithinSubjectEvaluation->WithinSubjectSplitter(evaluations.py:660)And the runner does
self.cv = self._create_splitter()(base.py:758-764), with no public hook to inject a splitter. SoCrossSubjectEvaluation(cv_class=CrossSubjectTransferSplitter, ...)does not swap the splitter;cv_classis forwarded intoCrossSubjectSplitteras its inner fold strategy, which is not what a transfer setup needs.Proposal
Let
CrossSubjectEvaluationaccept and use a top-level splitter, keeping the scikit-learn contract (split(y, metadata) -> (train_idx, test_idx)) untouched so caching, scoring, and result handling stay as-is. Two options, smallest first:Allow passing a splitter instance. If
cv(or asplitter=argument) is aBaseCrossValidatorthat yields(train_idx, test_idx)overmetadata, use it directly in_create_splitter()instead of hardcodingCrossSubjectSplitter. This makesCrossSubjectTransferSplitterusable today and is additive.Let
cv_classselect the top-level splitter when it is a MOABB splitter (detected by itssplit(y, metadata)signature), falling back to current behaviour otherwise.I lean towards option 1: it is the least surprising, does not change any default, and keeps the transfer splitter a plain cross-validator rather than a new evaluation engine. Background and the splitter sketch are in #1077.
Scope
CrossSubjectEvaluationfor the transfer-learning case to start.FakeDataset, mirroring howCrossSubjectSplitteris tested.