Transformer-based NextItemRecommender models

README.md

@@ -156,6 +156,7 @@ The table below lists the recommendation models/algorithms featured in Cornac. E
 | 2023 | [Scalable Approximate NonSymmetric Autoencoder (SANSA)](cornac/models/sansa), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.sansa.recom_sansa), [paper](https://dl.acm.org/doi/10.1145/3604915.3608827) | Collaborative Filtering | [requirements](cornac/models/sansa/requirements.txt), CPU | [quick-start](examples/sansa_movielens.py), [150k-items](examples/sansa_tradesy.py)
 | 2022 | [Disentangled Multimodal Representation Learning for Recommendation (DMRL)](cornac/models/dmrl), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.dmrl.recom_dmrl), [paper](https://arxiv.org/pdf/2203.05406.pdf) | Content-Based / Text & Image | [requirements](cornac/models/dmrl/requirements.txt), CPU / GPU | [quick-start](examples/dmrl_example.py)
 | 2021 | [Bilateral Variational Autoencoder for Collaborative Filtering (BiVAECF)](cornac/models/bivaecf), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.bivaecf.recom_bivaecf), [paper](https://dl.acm.org/doi/pdf/10.1145/3437963.3441759) | Collaborative Filtering / Content-Based | [requirements](cornac/models/bivaecf/requirements.txt), CPU / GPU | [quick-start](https://github.com/PreferredAI/bi-vae), [deep-dive](https://github.com/recommenders-team/recommenders/blob/main/examples/02_model_collaborative_filtering/cornac_bivae_deep_dive.ipynb)
+|      | [GPT-2 for Sequential Recommendation (GPT2Rec)](cornac/models/gpt2rec), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.gpt2rec.recom_gpt2rec), [paper](https://dl.acm.org/doi/10.1145/3460231.3474255) | Next-Item | [requirements](cornac/models/gpt2rec/requirements.txt), CPU / GPU | [quick-start](examples/transformer_rec_diginetica.py)
 |      | [Causal Inference for Visual Debiasing in Visually-Aware Recommendation (CausalRec)](cornac/models/causalrec), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.causalrec.recom_causalrec), [paper](https://arxiv.org/abs/2107.02390) | Content-Based / Image | [requirements](cornac/models/causalrec/requirements.txt), CPU / GPU | [quick-start](examples/causalrec_clothing.py)
 |      | [Explainable Recommendation with Comparative Constraints on Product Aspects (ComparER)](cornac/models/comparer), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.comparer.recom_comparer_sub), [paper](https://dl.acm.org/doi/pdf/10.1145/3437963.3441754) | Explainable | CPU | [quick-start](https://github.com/PreferredAI/ComparER)
 | 2020 | [Adversarial Multimedia Recommendation (AMR)](cornac/models/amr), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.amr.recom_amr), [paper](https://ieeexplore.ieee.org/document/8618394) | Content-Based / Image | [requirements](cornac/models/amr/requirements.txt), CPU / GPU | [quick-start](examples/amr_clothing.py)
@@ -166,10 +167,12 @@ The table below lists the recommendation models/algorithms featured in Cornac. E
 |      | [Temporal-Item-Frequency-based User-KNN (TIFUKNN)](cornac/models/tifuknn), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.tifuknn.recom_tifuknn), [paper](https://arxiv.org/pdf/2006.00556.pdf) | Next-Basket | CPU | [quick-start](examples/tifuknn_tafeng.py)
 |      | [Variational Autoencoder for Top-N Recommendations (RecVAE)](cornac/models/recvae), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.recvae.recom_recvae), [paper](https://doi.org/10.1145/3336191.3371831) | Collaborative Filtering | [requirements](cornac/models/recvae/requirements.txt), CPU / GPU | [quick-start](examples/recvae_example.py)
 | 2019 | [Correlation-Sensitive Next-Basket Recommendation (Beacon)](cornac/models/beacon), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#correlation-sensitive-next-basket-recommendation-beacon), [paper](https://www.ijcai.org/proceedings/2019/0389.pdf) | Next-Basket | [requirements](cornac/models/beacon/requirements.txt), CPU / GPU | [quick-start](examples/beacon_tafeng.py)
+|      | [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer (BERT4Rec)](cornac/models/bert4rec), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.bert4rec.recom_bert4rec), [paper](https://arxiv.org/pdf/1904.06690.pdf) | Next-Item | [requirements](cornac/models/bert4rec/requirements.txt), CPU / GPU | [quick-start](examples/transformer_rec_diginetica.py)
 |      | [Embarrassingly Shallow Autoencoders for Sparse Data (EASEᴿ)](cornac/models/ease), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.ease.recom_ease), [paper](https://arxiv.org/pdf/1905.03375.pdf) | Collaborative Filtering | CPU | [quick-start](examples/ease_movielens.py)
 |      | [Neural Graph Collaborative Filtering (NGCF)](cornac/models/ngcf), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.ngcf.recom_ngcf), [paper](https://arxiv.org/pdf/1905.08108.pdf) | Collaborative Filtering | [requirements](cornac/models/ngcf/requirements.txt), CPU / GPU | [quick-start](examples/ngcf_example.py)
 |      | [Sampler Design for Bayesian Personalized Ranking by Leveraging View Data (VEBPR)](cornac/models/bpr), [paper](https://arxiv.org/pdf/1809.08162) | Collaborative Filtering | CPU | [quick-start](examples/vebpr_example.py)
 | 2018 | [Collaborative Context Poisson Factorization (C2PF)](cornac/models/c2pf), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.c2pf.recom_c2pf), [paper](https://www.ijcai.org/proceedings/2018/0370.pdf) | Content-Based / Graph | CPU | [quick-start](examples/c2pf_example.py)
+|      | [Self-Attentive Sequential Recommendation (SASRec)](cornac/models/sasrec), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.sasrec.recom_sasrec), [paper](https://arxiv.org/pdf/1808.09781.pdf) | Next-Item | [requirements](cornac/models/sasrec/requirements.txt), CPU / GPU | [quick-start](examples/transformer_rec_diginetica.py)
 |      | [Graph Convolutional Matrix Completion (GCMC)](cornac/models/gcmc), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.gcmc.recom_gcmc), [paper](https://www.kdd.org/kdd2018/files/deep-learning-day/DLDay18_paper_32.pdf) | Collaborative Filtering | [requirements](cornac/models/gcmc/requirements.txt), CPU / GPU | [quick-start](examples/gcmc_example.py)
 |      | [Multi-Task Explainable Recommendation (MTER)](cornac/models/mter), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.mter.recom_mter), [paper](https://arxiv.org/pdf/1806.03568.pdf) | Explainable | CPU | [quick-start](examples/mter_example.py), [deep-dive](https://github.com/PreferredAI/tutorials/blob/master/recommender-systems/07_explanations.ipynb)
 |      | [Neural Attention Rating Regression with Review-level Explanations (NARRE)](cornac/models/narre), [docs](https://cornac.readthedocs.io/en/stable/api_ref/models.html#module-cornac.models.narre.recom_narre), [paper](http://www.thuir.cn/group/~YQLiu/publications/WWW2018_CC.pdf) | Explainable / Content-Based | [requirements](cornac/models/narre/requirements.txt), CPU / GPU | [quick-start](examples/narre_example.py)

cornac/models/__init__.py

@@ -24,6 +24,7 @@
 from .ann import ScaNNANN
 from .baseline_only import BaselineOnly
 from .beacon import Beacon
+from .bert4rec import BERT4Rec
 from .bivaecf import BiVAECF
 from .bpr import BPR
 from .bpr import WBPR
@@ -49,6 +50,7 @@
 from .gcmc import GCMC
 from .global_avg import GlobalAvg
 from .gp_top import GPTop
+from .gpt2rec import GPT2Rec
 from .gru4rec import GRU4Rec
 from .hft import HFT
 from .hpf import HPF
@@ -75,6 +77,7 @@
 from .pmf import PMF
 from .recvae import RecVAE
 from .sansa import SANSA
+from .sasrec import SASRec
 from .sbpr import SBPR
 from .skm import SKMeans
 from .sorec import SoRec

cornac/models/bert4rec/__init__.py

@@ -0,0 +1,16 @@
+# Copyright 2026 The Cornac Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+from .recom_bert4rec import BERT4Rec

cornac/models/bert4rec/bert4rec.py

@@ -0,0 +1,115 @@
+# Copyright 2026 The Cornac Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+import torch
+import torch.nn as nn
+
+
+class BERT4RecModel(nn.Module):
+    """BERT4Rec: bidirectional transformer encoder for sequence rec.
+
+    Uses HuggingFace's :class:`~transformers.BertModel` as the backbone. The
+    sequence-final hidden state is used to score candidate items via the dot
+    product with their output embeddings (separate "head" linear or tied to
+    ``item_emb``).
+
+    Returns ``(B, B+N)`` score matrices when called as
+    ``forward(_, hist_iids, out_iids, return_hidden=False)``.
+    """
+
+    def __init__(
+        self,
+        item_num,
+        embedding_dim=100,
+        maxlen=20,
+        n_layers=2,
+        n_heads=1,
+        dropout=0.1,
+        pad_idx=-1,
+        init_std=0.02,
+        device="cpu",
+    ):
+        super().__init__()
+        from transformers.models.bert import BertConfig, BertModel
+
+        self.item_num = item_num
+        self.pad_idx = pad_idx if pad_idx >= 0 else item_num
+        self.maxlen = maxlen
+        self.dev = device
+        self.init_std = init_std
+
+        config = BertConfig(
+            vocab_size=item_num + 1,
+            hidden_size=embedding_dim,
+            num_hidden_layers=n_layers,
+            num_attention_heads=n_heads,
+            intermediate_size=embedding_dim * 4,
+            hidden_act="gelu",
+            hidden_dropout_prob=dropout,
+            attention_probs_dropout_prob=dropout,
+            max_position_embeddings=maxlen + 1,
+            initializer_range=init_std,
+            pad_token_id=self.pad_idx,
+            layer_norm_eps=1e-12,
+            use_cache=False,
+        )
+
+        self.item_emb = nn.Embedding(
+            num_embeddings=item_num + 1,
+            embedding_dim=embedding_dim,
+            padding_idx=self.pad_idx,
+        )
+        self.transformer_model = BertModel(config)
+        self.item_biases = nn.Embedding(item_num + 1, 1, padding_idx=self.pad_idx)
+        self._init_weights()
+        self.to(device)
+
+    def _init_weights(self):
+        self.item_emb.weight.data.normal_(mean=0.0, std=self.init_std)
+        self.item_emb.weight.data[self.pad_idx].zero_()
+        self.item_biases.weight.data.zero_()
+
+    def _encode(self, hist_iids):
+        attention_mask = (hist_iids != self.pad_idx).long()
+        embeds = self.item_emb(hist_iids)
+        out = self.transformer_model(
+            inputs_embeds=embeds, attention_mask=attention_mask
+        )
+        return out.last_hidden_state[:, -1, :]
+
+    def forward(self, user_ids, hist_iids, out_iids, return_hidden=False):
+        hidden = self._encode(hist_iids)
+        item_e = self.item_emb(out_iids)
+        bias = self.item_biases(out_iids)
+        if return_hidden:
+            return hidden, item_e, bias
+        scores = torch.mm(hidden, item_e.T) + bias.T
+        return scores
+
+    @torch.no_grad()
+    def predict(self, user_ids, log_seqs, item_indices=None):
+        if item_indices is None:
+            item_indices = torch.arange(self.item_num, device=self.dev)
+        else:
+            item_indices = torch.as_tensor(
+                item_indices, dtype=torch.long, device=self.dev
+            )
+        if not isinstance(log_seqs, torch.Tensor):
+            log_seqs = torch.as_tensor(log_seqs, dtype=torch.long, device=self.dev)
+        hidden = self._encode(log_seqs)
+        item_e = self.item_emb(item_indices)
+        bias = self.item_biases(item_indices)
+        scores = torch.mm(hidden, item_e.T) + bias.T
+        return scores.squeeze().detach().cpu().numpy()