Cross-subject transfer learning via CrossSubjectEvaluation (calibration_size)

[bruAristimunha]

Addresses #1077. The split-based alternative to the dedicated evaluation engine in #1091, following the direction discussed in #1077: target-aware transfer learning runs through the existing CrossSubjectEvaluation with a few lines of change — no new evaluation class, no separate transfer module.

Usage

CrossSubjectEvaluation(paradigm=..., datasets=..., calibration_size=0.2).process(pipelines)

What it adds

• TransferSplitter(base_splitter, calibration_size) (splitters.py) — a generic wrapper that carves the first calibration_size fraction off the held-out group of any leave-one-group-out splitter, yielding (train, calibration, test). Gives subject-, session-, and dataset-transfer from one mechanism.
• CrossSubjectEvaluation gains calibration_size (+ calibration_labeled); when > 0, _create_splitter wraps its CrossSubjectSplitter in TransferSplitter.
• base.py consumes the split with train, *cal, test (parallel + serial paths). The held-out calibration slice is routed raw to the pipeline steps that request it via scikit-learn metadata routing (set_fit_request): subjects, X_target_unlabeled / X_target_labeled. Plain pipelines request nothing → {} → the fit is unchanged.

Design notes

• The calibration slice is passed raw; base.py does not transform it through the pipeline (no transform-through-steps). The transfer estimator owns the target representation. This is what keeps the change minimal.
• Labeled vs unlabeled is just which kwarg carries the slice — a fit-time concern, not split geometry.
• Metadata routing (SLEP006) replaces signature inspection for passing subjects/target data to estimators.
• No trialwise predict wrapper (a no-op for inductive estimators); no new dependency.

Verification

Full evaluation + splitter test suites pass. Integration test (test_cross_subject_calibration_*): a target-aware step receives the routed subjects + non-empty X_target_unlabeled on every fold, and a plain pipeline runs untouched at calibration_size=0.5.

cc @toncho11 — concrete, minimal counter-proposal to #1091; an existing target-aware estimator (e.g. RPA) drops in by declaring set_fit_request(subjects=True, X_target_unlabeled=True, ...).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da2b30015c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject labeled 100% calibration to avoid leaking test labels

When calibration_size=1.0 is combined with labeled=True, TransferSplitter makes calib identical to test, so this block routes y[test] as y_target_labeled before the same trials are scored. Any target-aware estimator that requests y_target_labeled can train on the labels of every evaluated sample, making the reported score invalid; please disallow this combination or avoid passing labels when the calibration slice overlaps the test slice.

Useful? React with 👍 / 👎.

[toncho11]

I will need more time to analyze your code Monday. Here are my initial comments:

• Trialwise / one-shot cross-subject evaluation is the protocol I currently need for an important publication I am preparing. Infact, the additional modes were added to make the evaluation more general, but HOS_SOURCE_ONLY_TRIALWISE is the one I really really need. Everyone who works on trailwise/one shot will benefit from it. I would like to keep HOS_SOURCE_ONLY_TRIALWISE in the loop, because I would like to be able to say that my results were obtained with the latest MOABB version. Also I want to make sure that when people use HOS_SOURCE_ONLY_TRIALWISE - they can only acccess a single trial per evaluation, no doubt about that.

• I think explicit modes are useful for standardizing comparisons in the community. If the calibration/adaptation fraction is only a free float, different papers may use different values such as 0.2, 0.3, 0.5, or 0.7, which makes results harder to compare. Having predefined modes such as 20%, 50% would make the evaluated protocol clearer and easier to benchmark consistently.

• I have to study more your set_fit_request(subjects=True, X_target_unlabeled=True, ...), but I have a concern about the representation of X_target_unlabeled. If the calibration slice is passed raw and is not transformed through the previous pipeline steps, then an existing RPA step placed after Covariances() would not directly drop in. In that pipeline, RPA would receive source covariance matrices as X, but raw target epochs as X_target_unlabeled.

For example:

Covariances()
RPA()
TangentSpace()
Classifier()

would lead in the RPA transformer to:

X                  = source covariance matrices
X_target_unlabeled = raw target epochs

Not sure how this can be handled.

I also remind you that we can do a Teams meeting and discuss. It will be easier to discuss some details in person.

[gcattan]

Thanks guys for looking into this.

I see good comments on both side.

May be we can have a short call next week to synchronize, and decide a common implementation?

[gcattan]

@toncho11 does this lines answer your concern on RPA?

[toncho11]

No. Bacause the test does not check if X and X_target_unlabeled have the same representation. A real RPA step would need either transformed target covariance data, or it would need to compute target covariances internally from the raw calibration slice.

[toncho11]

I am preparing some improvements to this PR.

[toncho11]

@bruAristimunha I have created a new Pull Request to add to your current code. It is here: bruAristimunha#8
It adds 2 of the important features we discussed.

Next will be the adapatation of RPA. I am currently thinking on what is the best strategy.

[bruAristimunha]

hey @toncho11!

I merged, but as maintainer, you can direct commit here ;)

https://tighten.com/insights/adding-commits-to-a-pull-request/

[bruAristimunha]

I'm happy to discuss today if you have any time, guys!

[gcattan]

Nice! Sending invitation for 3pm CET time, to discuss last point on RPA.

[toncho11]

So the problem is not about just RPA algorithm, it is about all Transfer Learnig classfiers / tansformers:

If we use a pipeline like:

Covariances() -> RPA() -> TangentSpace() -> Classifier

then RPA.fit(X, ...) receives X after Covariances(), so source data is covariance matrices. But the routed metadata X_target_unlabeled / X_target_labeled usually comes directly from the evaluation, so it is still raw epochs. Therefore RPA may receive:

source X = covariance matrices
target X_target_unlabeled = raw epochs

That is inconsistent.

Solutions:

1. Make RPA compute covariances internally for both source and target data.

2. Add a target-aware pipeline / metadata-routing mechanism: when a step such as RPA requests X_target_unlabeled or X_target_labeled, the target data should first be transformed through all previous fitted pipeline steps before being passed to RPA.fit(). This ensures that source X and target metadata are in the same representation, for example both as covariance matrices after Covariances().

3. We could make the evaluation pipeline target-aware. Since X_target_unlabeled or X_target_labeled is already known at the beginning of each fold, it can be carried in parallel with the normal source X during pipeline fitting. Each fitted transformer would transform both the source data and the target calibration data. Then, when a target-aware step such as RPA requests target metadata, it would receive the target data already transformed through the previous fitted steps. For example, in Covariances() -> RPA(), both source X and X_target_unlabeled would reach RPA as covariance matrices. This avoids representation mismatch and provides a general solution for RPA and other target-aware estimators.

   • The main drawback is that this is no longer standard sklearn Pipeline behavior. Sklearn transforms only the main X; metadata such as X_target_unlabeled is routed to the estimator but is not automatically transformed through previous steps. Therefore, MOABB would need a custom target-aware pipeline or a custom evaluation fit loop. This adds complexity.

The problem is that the pipeline before RPA must be the fitted clone from the current CV fold, not the original pipeline object. Otherwise we risk using an unfitted transformer, the wrong fitted state, or leakage between folds.

[toncho11]

After our meeting: quick check on the solution proposed by @bruAristimunha

I think using the MOABB paradigm's process pipeline could solve the representation problem only for preprocessing steps that do not need to be fitted. If the transformation is fixed or stateless, for example a deterministic conversion such as covariance computation with Covariances(), then both source data and target calibration data can be converted before entering the evaluation pipeline, and RPA will receive them in the same representation.

But this does not work for preprocessing steps that need to be fitted on the training fold, such as CSP, PCA, scaling, feature selection, or other learned transformations. The MOABB paradigm's postprocess pipeline is fixed and is not trained with fit(), so it cannot learn fold-specific parameters.

[bruAristimunha]

Thanks @toncho11 — I went deep on all three points. Short version: each is addressable on the existing CrossSubjectEvaluation, no parallel engine to keep in sync.

1. Trialwise / one-shot

For any inductive estimator (every standard Riemannian/CSP source-only pipeline), predict is computed per row, so trialwise and blockwise give the same number. I checked Covariances + TangentSpace + LR (source-only) on BNCI2014_004: predictions identical, predict_proba differs by 2e-16, identical ROC-AUC over 720 target trials. So HOS_SOURCE_ONLY_TRIALWISE reproduces on the standard cross-subject score — you can cite it as the latest MOABB.

If you want the ironclad runtime guarantee that the estimator only ever sees one trial, that's a tiny reusable wrapper at the pipeline level — OneShot(pipeline) — which forces predict(X) == [predict(X[i:i+1]) for i]. It travels with your method, works in any evaluation, and is a no-op for inductive pipelines (free insurance). It doesn't need to be an evaluation mode.

2. Named modes for comparability

Agreed that a free float fragments benchmarks. We can ship named presets so papers cite a name, not a raw number:

from moabb.evaluations.protocols import CALIB_20, CALIB_50   # {"calibration_size": 0.2}, ...
CrossSubjectEvaluation(cv_kwargs=CALIB_20)

Sugar over cv_kwargs, fully standardized, no engine.

3. Representation (RPA after Covariances()) — solved natively

You're right that routing the slice raw means RPA would get raw target. scikit-learn ≥ 1.6 has Pipeline(transform_input=[...]) for exactly this — it transforms the routed input through the earlier steps before the consuming step:

make_pipeline(
    Covariances(), RPA().set_fit_request(subjects=True, X_target_unlabeled=True),
    TangentSpace(), clf,
    transform_input=["X_target_unlabeled"],
)

RPA then receives cov(target) — same representation as its X. Your unmodified RPA drops in: I ran it end-to-end through CrossSubjectEvaluation(cv_kwargs={"calibration_size": ...}) and it reproduces your CrossSubjectTargetAwareEvaluation.process() numbers exactly (max abs diff 0.0) on BNCI2014_004 (HOS_UNLABELED_20P + 50P).

One caveat: pyriemann's stateless Covariances doesn't mark itself fitted, which transform_input needs — a one-liner (__sklearn_is_fitted__) upstream in pyriemann, or a thin wrapper our side.

Runnable example on the PR: examples/how_to_benchmark/noplot_cross_subject_transfer.py (a target-aware Riemannian Alignment step receiving subjects + transformed target via routing + transform_input).

Net: split (calibration_size) + metadata routing + transform_input covers your cases — including unlabeled adaptation with a mid-pipeline RPA — on the standard machinery. Happy to do the Teams call to walk through it.

[toncho11]

Thank you Bruno! Here are my quick thoughts:

1) Trialwise / one-shot

It was never about different results. I never doubted that. It was about preventing the client (of the MOABB library) from accessing the whole block. It is about imposing a restriction over the user. If you do not have this rule, somebody might say “OK, I have access to the whole, block, so I am allowed to use it”. It is about preventing cheating and confusion. The user gets only what he is allowed to receive. So the responsibility should be on the evaluation/benchmark side, not the user. The objective is to prevent both “cheating” and unintended accidental leakage. Somebody gets better results, he publishes and later for everyone’s disappointment, he was using unfairly (for the category/mode he has selected) extra data, he was not supposed to.

2) Named modes for comparability

This is a bit related to 1). The idea is to impose clear rules and limitations for fair, proper benchmarking. The idea is that once you select a mode, you get only what you are allowed and not what you request. You might request labeled data, but if the official benchmark mode you have selected says “unlabeled_50” then you will get only unlabeled 50%. The idea in general is to select not only calibration size, but also labeled/unlabeled, trialwise or blockwise. Also, I cannot say right now if with this new way you suggest we can easily set all the modes/protocols that we need. In my PR we have almost maximum flexibility which modes to define - now and in the future. I predict there will be changes. Someone or me might ask for adding more modes/protocols.

3) Representation (RPA after Covariances())

Sounds great! I did not know this was possible with sklearn. Thank you! I need to study it further and test it.

I am also busy with my other projects. So, I might react with delays. And I agree that a meeting is a good idea :)
Thank you.

[toncho11]

On the blockwise vs trialwise:

If predict(X_test) receives all target trials together, the method can estimate subject-level statistics or a target distribution from the entire block, such as a mean covariance, normalization parameters, outlier structure, class-independent alignment, or other batch statistics. This can improve classification of the whole block compared with a strict trial-by-trial setting, where each prediction only has access to one target trial. Therefore, blockwise prediction and trialwise prediction are different access protocols.

[gcattan]

Ok, I think everyone agree on Point 2 and 3 (nice finding btw).

On the blockwise vs trialwise, I see it requires more discussion. @bruAristimunha @toncho11 same time next week?

[bruAristimunha]

Follow-up on point 1 (trialwise) — there's a pure scikit-learn recipe, no custom wrapper:

from sklearn.frozen import FrozenEstimator            # >=1.6, blocks re-fit
from sklearn.model_selection import cross_val_predict, LeaveOneOut

pred = cross_val_predict(FrozenEstimator(source_model), X_target, y_target,
                         cv=LeaveOneOut(), method="predict")

LeaveOneOut makes every test fold exactly one trial; FrozenEstimator stops any re-fit, so the source model predicts each held-out trial in isolation — your "single trial, no doubt" guarantee, by construction. It's a no-op for inductive pipelines and provably catches leakage for transductive ones. Added to examples/how_to_benchmark/noplot_cross_subject_transfer.py.

[toncho11]

Thank you Bruno. I have been working on improvements of my own this morning.

I agree that the FrozenEstimator + LeaveOneOut recipe is a good pure scikit-learn way to demonstrate one-trial-at-a-time prediction in an example. However, my concern is that this still leaves the trialwise guarantee at the user/example level. For benchmark reporting, especially for the paper I am working on, I think this restriction should be owned by MOABB directly through the evaluation protocol. The important point is not only that a user can manually reproduce trialwise prediction, but that the official MOABB evaluation mode enforces the allowed target access by construction. Otherwise, two users could report “trialwise” results while relying on slightly different user-side code. Encoding this as CsMode.HOS_SOURCE_ONLY_TRIALWISE makes the protocol explicit, reproducible, and harder to misuse.

I took into account some of your comments and I removed the broader public predict_mode API.

• removed the public PredictMode enum;
• removed the public/general prediction wrapper class;
• removed the general user-configurable predict_mode parameter;
• kept trialwise scoring only as a small internal helper used by the HOS_SOURCE_ONLY_TRIALWISE mode.

This keeps the API simpler while also still allowing MOABB to take responsibility for the trialwise benchmark protocol.

[toncho11]

Also:

MOABB still supports both:

1. named protocol presets through cs_mode;
2. manual calibration settings through cv_kwargs.

[toncho11]

Quck summary of my last changes:

• the API is smaller
• trialwise evaluation remains supported, but in a less intrusive way
• the default is clearer - HOS_SOURCE_ONLY
• the protocol descriptions are updated, the added explanation of blockwise vs trialwise prediction reduces ambiguity
• manual calibration remains possible

[gcattan]

I think I get the point. Even not on purpose, user can still do honest mistake.
The changes are small, and taken into account it doesn't bring too much complexity, looks like a good compromise.
@bruAristimunha wdyt?

[toncho11]

Hi @bruAristimunha

Here are my thoughts on two points:

1. I agree that the FrozenEstimator + LeaveOneOut recipe is a useful pure scikit-learn demonstration of one-trial-at-a-time prediction. However, I do not think this fully replaces the MOABB trialwise protocol for subject-aware methods that I have proposed. The recipe addresses the prediction side: it freezes a source-trained model and calls prediction with one target trial at a time. My concern is that it does not cover the fit-time metadata that some cross-subject algorithms need. In my case, my publication, the method needs both pieces: per-source-subject metadata during fitting, similar to RPA, and trialwise prediction at scoring time.
   This is why I think the trialwise protocol should remain in CrossSubjectEvaluation: MOABB can route the required fit-time metadata, such as source-subject information and, when the selected protocol allows it, labeled or unlabeled target calibration data. It can then enforce one-trial-at-a-time target access during scoring, as the current implementation does.

2. Adding CsMode to fit()
   The idea is that a classfication algorithm should be able to reject modes for which it was not designed, or adapt its behavior according to the selected mode. For example, I do not want a trialwise-only method to be compared unfairly with methods that receive 20% or 50% of the held-out subject for calibration, because that is a different protocol from the one it was designed for. Similarly, labeled and unlabeled calibration modes may require different algorithmic behavior. In that case, it is better if the selected protocol is provided explicitly to the estimator, rather than forcing the estimator to infer the mode only from which metadata happen to be present.

def fit(self, X, y, *, subjects=None, cs_mode=None, **fit_params):
    if CsMode(cs_mode) != CsMode.HOS_SOURCE_ONLY_TRIALWISE:
        raise ValueError(
            "This method is designed only for "
            "CsMode.HOS_SOURCE_ONLY_TRIALWISE."
        )

I can work/discuss on this only Wednesday. So we can make a meeting Wednesday or 16 July. I will be on vacation next week. Wednesday could be to just sync ideas and then we continue around 16 of July.

[bruAristimunha]

let's talk wed in the morning?

[toncho11]

OK, I have sent you a Teams invitation for 10h30 this morning on your gmail.
Let me know if it OK for you. Gregoire won't be able to participate.

[toncho11]

Or it might be better if you create a meeting and I join? If this works better.
I am available.

[bruAristimunha]

• limit to cross-subject and cross-dataset,
• renaming to a more explicit name;
• Create the 2 examples;

[bruAristimunha]

we keep the enum functionality

[toncho11]

My last contributions are:

1. cs modes kept, renamed to more explicit names
2. Add the possibility to request the CrossSubject mode. This allows an analgorithm to adapt to the selected Cross Subject benchmark mode and also to refuse running in certain benchmark modes for which it is was designed for
3. Created the 2 examples