New CrossSubjecEvaluation that supports transfer learning methods

[toncho11]

I am proposing a new class called CrossSubjectTargetAwareEvaluation that extends cross-subject evaluation to pipelines that require subject metadata or optional unlabeled/labeled target-subject calibration data. The existing CrossSubjectEvaluation becomes one of 6 options that can be configured.

All this is motivated by the large number of papers on Transfer Learning. So we need:

• a way to provide not merged subjects, but all subjects separately, so that transfer learning methods can work. Let's call this "metadata-aware".
• allow a well defined controlled access to the held out subject (HOS), also called the test subject or heldout target

I myself am working on a transfer learning cross subject method and all this is motivated by real needs.

Currently in my code I have defined:

class CsMode(Enum):
    HOS_SOURCE_ONLY_TRIALWISE = auto()  # Source-only training; each held-out target trial is predicted independently.
    HOS_SOURCE_ONLY_BLOCKWISE = auto()  # Source-only training; held-out target trials are predicted as a block, matching standard MOABB behavior.
    HOS_UNLABELED_20P = auto()          # First 20% of held-out target trials are used unlabeled for adaptation; remaining 80% are evaluated.
    HOS_UNLABELED_50P = auto()          # First 50% of held-out target trials are used unlabeled for adaptation; remaining 50% are evaluated.
    HOS_UNLABELED_100P = auto()         # All held-out target trials are used unlabeled for transductive adaptation and also evaluated.
    HOS_LABELED_20P = auto()            # First 20% of held-out target trials are used with labels for supervised calibration; remaining 80% are evaluated. 

This will allow people to start comparing their methods clearly. Some will focus on HOS_UNLABELED_20P, others on HOS_SOURCE_ONLY_TRIALWISE, but the results will become comparable. I work on HOS_SOURCE_ONLY_TRIALWISE.

So the objective is:

• preserve MOABB cache/result/save mechanics
• support old pipelines
• support final metadata-aware classifiers (heldout subject aware classifiers)
• support intermediate metadata-aware transformers (heldout subject aware transformers)
• transform X_target_unlabeled through earlier steps

Notes:

• HOS_SOURCE_ONLY_BLOCKWISE predicts the target block after source-only training (the heldout subject is not provided during training), while HOS_UNLABELED_100P explicitly uses the entire unlabeled target block during fit/adaptation before predicting it.

I will provide code in a PR later. As you can imagine, it is a big feature :)

I think the best strategy is to have CrossSubjectTargetAwareEvaluation along the CrossSubjectEvaluation. And allow people to test it for a while.

[gcattan]

I agree, this topic of transfer learning become hot and we are missing "metadata awereness" in CrossSubject.

I did something similar in pyRiemann-qiskit to use RPA pipeline, but it only covers one of the use cases you mentioned.

I see potential in this proposal.

@bruAristimunha FYI

[toncho11]

@bruAristimunha I am preparing the PR. It will be a big one. The new Cross Subject will be fully adapted for the latest trends in Cross Subject Transfer Learning. It will allow people to fairly compare their results in Transfer Learning, which is the main purpose of MOABB - fair comparable results.

[bruAristimunha]

Hey @toncho11, short version: I'm in favour of adding transfer-learning splits!

My one constraint is that they go in as scikit-learn cross-validators, not as a new evaluation engine with modes. The reason is narrow and practical: the protocol/evaluation determines the ranking. If our split contract drifts from sklearn's, two results stop being comparable, and people stop trusting the leaderboard. So the design target is to add the capability without touching the contract.

Why I want to stay close to scikit-learn. Everything in moabb/evaluations/splitters.py is a BaseCrossValidator that does one thing: def split(self, y, metadata) -> Iterator[(train_idx, test_idx)]. And base.py consumes it in exactly one place: for cv_ind, (train_idx, test_idx) in enumerate(splitter.split(y, metadata)). That is the whole interface. A transfer split that yields (train_idx, test_idx) into metadata.index inherits the existing caching, scoring, and result handling for free. No special case, no parallel evaluation class to keep in sync.

The proposal. Transfer learning with calibration means: train on the source subjects, optionally add a labelled slice of the target subject, then test on the rest of the target. That is one parameter on top of CrossSubjectSplitter, not a new protocol. The calibration slice has to be drawn per session, otherwise a single session can land entirely in train and the test score reads as cross-subject when it is really within-session. So the parameter controls per-session calibration, and the per-session draw is the part that keeps the claim honest.

class CrossSubjectTransferSplitter(BaseCrossValidator):
    """Leave-one-subject-out, with optional labelled calibration from the target.

    For each held-out target subject:
      train = all source subjects  (+ ``calibration_size`` of each target session)
      test  = the remaining target trials

    calibration_size=0.0 reproduces CrossSubjectSplitter exactly (zero-calibration
    transfer). calibration_size=0.2 gives the "20% labelled target" setting.
    Calibration trials are drawn per session and stratified by class, so no session
    leaks wholesale into train and the test set stays balanced.
    """

    def __init__(self, calibration_size: float = 0.0, random_state=None):
        self.calibration_size = calibration_size
        self.random_state = random_state

    def split(self, y, metadata):
        rng = check_random_state(self.random_state)
        all_index = metadata.index.values
        subjects = metadata["subject"].unique()

        for target in subjects:
            target_mask = (metadata["subject"] == target).to_numpy()
            source_idx = all_index[~target_mask]

            if self.calibration_size <= 0:
                yield source_idx, all_index[target_mask]
                continue

            calib_parts, test_parts = [], []
            target_meta = metadata[target_mask]
            for session in target_meta["session"].unique():
                sess_mask = (target_meta["session"] == session).to_numpy()
                sess_idx = all_index[target_mask][sess_mask]
                calib_idx, test_idx = train_test_split(
                    sess_idx,
                    train_size=self.calibration_size,
                    stratify=y[target_mask][sess_mask],
                    random_state=rng,
                )
                calib_parts.append(calib_idx)
                test_parts.append(test_idx)

            train_idx = np.concatenate([source_idx, *calib_parts])
            yield train_idx, np.concatenate(test_parts)

Usage is the plain scikit-learn loop, so it works as a standalone splitter without touching any evaluation class:

from moabb.evaluations import CrossSubjectTransferSplitter

# calibration_size=0.0 is source-only (same as today's CrossSubjectSplitter);
# 0.2 adds a per-session labelled slice of the target subject to train.
splitter = CrossSubjectTransferSplitter(calibration_size=0.2, random_state=42)

for train_idx, test_idx in splitter.split(y, metadata):
    ...

Wiring it into CrossSubjectEvaluation (to reuse its caching/results/scoring) is a separate, optional follow-up, tracked in #1088. If the framing works for you, I'll open a PR with the splitter and tests against FakeDataset, mirroring how CrossSubjectSplitter is tested, and we iterate from there.

[bruAristimunha]

I updated the answer, guys @toncho11 and @gcattan, in case you are seeing this by e-mail.

[toncho11]

I understand the motivation for keeping the standard scikit-learn splitter contract, and I agree that this is a good fit for labelled calibration, where target calibration samples can simply be added to train_idx.

However, my current code also covers unlabeled target adaptation and trialwise source-only prediction. The unlabeled case cannot be expressed cleanly with only (train_idx, test_idx): unlabeled target samples are not labelled training data, but the estimator still needs access to them during fit/adaptation.

For trialwise prediction, a splitter could technically yield one target trial at a time as test_idx, but then the folds are no longer one fold per held-out subject. My implementation keeps the LOSO subject-level fold and only changes the prediction call so the estimator sees one test trial at a time.

In short:

• My code supports more transfer-learning cases, especially unlabeled target adaptation and trialwise prediction. This is important for methods such as Riemannian Procrustes Analysis, a common transfer-learning technique that uses only unlabeled target data.
• I tried to reuse the existing MOABB machinery as much as possible: data loading, splitting, scoring, result construction, model saving, fitted checks, and the general cross-subject structure are still based on MOABB’s existing evaluation code.
• My class inherits from CrossSubjectEvaluation and can be used independently. It does not replace the existing CrossSubjectEvaluation; it adds a separate target-aware evaluation for protocols that cannot be represented cleanly by a standard (train_idx, test_idx) splitter.

I have thought carefully about this design and tried to cover the important details. Modes are actually importatant because in an article you can put the string HOS_UNLABELED_50P and this way it is clear in which category you are competing.

I will put my current code in a PR so we can discuss the design concretely there.

[toncho11]

@bruAristimunha @gcattan Also it might be more productive to make a Teams meeting next week and discuss it.

[bruAristimunha]

We can change the train_idx, test_idx to something more generic, such train_idx, valid_idx, test_idx, and the train can be empty too.

Can you point for the code for me look?

[toncho11]

I have been working on it for the last 1 month. I just need a few more touches.

Also let's take Riemannian Procrustes Analysis (RPA) for example. RPA needs access to which data belongs to which subject. In my code I say these samples belong to this subject in the fit() method. Without this information there is no transfer learning.