Zixuan Wang

, Yuhong Chen

, Yuxuan Zhu

, Guidong Lei

, Zhiluohan Guo

, Yu Zhao

, Kun Wang

, Bangyang Hong

, Kangle Wu

, Yabo Ni

, Anxiang Zeng

, Cong Fu

, Hui Li

williamzixuan@stu.xmu.edu.cn, fc731097343@gmail.com, hui@xmu.edu.cn

Shopee Pte. Ltd.

Abstract.

Industrial recommenders increasingly adopt the pretrain–then-transfer paradigm, yet behavioral distribution drift raises two questions: what to learn from behavior sequences, and how to transfer the learned knowledge while the pretrained model is continually refreshed. To resolve them, we propose Knowledge–Geometry Decoupling (KGD). For what to learn, conventional next-token prediction treats adjacency as dependency and may encode spurious transitions across unrelated sessions. We introduce Behavioral Multi-Token Prediction (BMTP) to retain only collaboratively or semantically related future items as supervision, yielding cleaner and more transferable behavioral knowledge. For how to transfer, pretrained knowledge and task-specific geometry impose conflicting optimization demands on shared parameters. To handle it, KGD assigns them to separate parameter sets: a refreshable encoder owns behavioral knowledge, while a task learner reads contextualized encoder states through read-only cross-attention and writes task-specific geometry through Anchored Calibration Residual (ACR) orthogonal to the pretrained embedding. The decoupled ownership enables continual knowledge refresh without task-gradient interference or invalidating downstream adaptation. KGD improves over strong pretrain–transfer baselines by 4–12% on eight public benchmarks and sustains its advantage over a 90-day production stream where baselines show no gains. KGD has been fully deployed in Shopee. In a live A/B test on Shopee Homepage Search, it increases GMV per user by 1.75% and advertising revenue by 1.53%, demonstrating its high practical value. We provide the core implementation of KGD at https://github.com/FuCongResearchSquad/KGD4REC.

Recommender Systems, Sequential Recommender, Transformer, Pretrain-then-Transfer

Information systems Recommender systems

1. Introduction

Pretraining a large model on abundant data and transferring it to a downstream task has become the default recipe in language and vision (Gan et al., 2022; Min et al., 2024), and industrial recommenders have adopted it in earnest (Liu et al., 2023b). Transformers pretrained on user behavior sequences with next-token objectives now initialize or augment production models for downstream tasks such as CTR estimation and ranking, and recent work shows that this recipe unlocks scaling behavior that discriminative training alone could not reach (Wang et al., 2025). The transfer itself, however, works differently for recommenders. In language and vision, a model is often pretrained once and transferred to tasks thereafter; in recommendation, behavioral distributions shift constantly (Appendix B.1, Fig. 4), so pretraining must be refreshed on incoming data throughout the system’s service lifecycles.

What shifts, and why it matters. Two forms of drift act on a traffic stream, and they compound one another. The item pool turns over continually as new content, listings, and campaigns enter and age out. More consequentially, the audience turns over as well: users arrive unevenly and return at irregular intervals. A user’s behavior sequence is therefore not a continuous trajectory of intent, but a concatenation of separate interest sessions. This segmentation corrupts the supervision on which pretraining relies: across a session boundary, adjacency no longer implies dependency, and an objective that treats every adjacent pair as a training signal injects noise into the learned representation. Drift also renders any fixed representation stale, as it encodes behavioral patterns that the system no longer observes; pretraining must therefore be refreshed. Yet refreshing pretrained parameters while a task optimizes the same weights forces two conflicting objectives onto a single parameter set. These two effects raise two questions for any pretrained recommender under drift: what knowledge to learn, and how to transfer it while the representation keeps refreshing.

In this paper, we focus on addressing two central questions for streaming recommendation:

Q1: What to learn. Sequential pretraining adopts the autoregressive objective of language modeling with little change, predicting the next item (or next a few) from every position (Kang and McAuley, 2018; Chen et al., 2025), thereby treating adjacency as dependency. However, adjacency does not imply dependency in recommender. As depicted in Fig. 1(a), a user sequence in Shopee recommender consists of two consecutive sub-sequences of “bicycle gear” and “pants”, and modeling the dependency between such unrelated items can lead to noisy representations for the downstream task. This noise is recognized, but existing remedies address it only partially. Attention-based denoising (Yuan et al., 2021) down-weights irrelevant items in the representation, yet its objective still supervises the model on the raw transitions, forcing it to fit false dependencies. LLM-based relabeling (Wang et al., 2026a) instead rewrites noisy interactions, but requires per-sequence LLM inference that is impractical to rerun over billion-scale logs at each refresh.

Q2: How to transfer. As shown in Fig. 1(b), our study on Shopee recommender shows that pretrained knowledge and task-specific geometry pull the same parameters in conflicting directions evidenced by the nearly zero cosine similarities of gradients between two tasks, and existing methods satisfy one objective only by compromising the other: (1) Full fine-tuning grants the task full control over geometry, but its gradients overwrite the pretrained parameters. Consequently, refreshing makes pretrained knowledge steadily erode. (2) Frozen transfer, as in the transfer-and-freeze configuration (Wang et al., 2025), takes the opposite extreme: it forbids writes to protect pretrained knowledge, so the task can only re-project a fixed embedding in downstream modules, never adjust the representation geometry or inject task-required knowledge at the input level. (3) Adapters and low-rank tuning (Hu et al., 2022) inherit the same limitation: a frozen backbone with a small trainable residual gives too little freedom to reshape geometry and knowledge encoded at the source.

To resolve these questions, we propose Knowledge–Geometry Decoupling (KGD), a refresh-native framework that separates pretrained behavioral knowledge from task-specific geometry. For what to learn, Behavioral Multi-Token Prediction (BMTP) filters future transitions by collaborative or semantic relevance, avoiding noisy supervision from raw adjacency. For how to transfer, KGD assigns the pretrained encoder and task learner disjoint parameter ownership. The task learner reads contextualized encoder states through read-only cross-attention, while Anchored Calibration Residual (ACR) adds a task-owned low-rank residual orthogonal to the pretrained embedding. This mechanism preserves the behavioral geometry while enabling task-specific deformation from the input level. Because task geometry is anchored to the pretrained embedding as an orthogonal residual, not fixed in absolute coordinates, the encoder can refresh daily beneath a stable task residual at marginal cost. Refresh and adaptation never invalidate each other, exactly what a drifting stream demands.

• A denoised refresh-native pretrain-transfer framework. We propose KGD, which denoises the pretraining objective with BMTP and lets the resulting pretrained behavioral knowledge and task-specific geometry coexist as two orthogonal layers under separate read-write ownership, rather than compete for one representation, answering what to learn and how to transfer. This makes knowledge refresh and task adaptation independent operations: the encoder can refresh daily at marginal cost without invalidating what the task has learned, a missing primitive for pretrained transfer under continuous drift.

• A 90-day study of what makes streaming transfer work. Over 90 days on an industry-scale stream under matched budgets, eight transfer modes each fail differently: no pretraining lacks clean structural knowledge learned; frozen knowledge erodes; entangled refresh is cancelled by objective conflict; partial parameter separation lacks geometric freedom; data replay overfits stale noise. KGD escapes them all by removing each cause: it pretrains a clean geometry, refreshes it, avoids conflict, writes task geometry end-to-end, and needs no history data replay.

• Comprehensive validation and full deployment. KGD outperforms the strongest pretrain–transfer baselines by 4–12% across eight public benchmarks, and extensive ablations confirm that each component contributes to this gain. Deployed to full traffic in Shopee Homepage Search at manageable training cost, it lifts GMV per user by 1.75% and advertising revenue by 1.53% in a live A/B test.

Discussion on our research focus. This work centers on acquiring high-quality pretraining knowledge from behavior sequences and transferring it to real recommendation services, with the scope specified at two levels. At the model level, we study the classic deep-learning recommenders that form the industrial mainline rather than LLM-based recommenders (Wu et al., 2024; Wang et al., 2024), whose pretrained knowledge is distilled from text and adapted via prompting or parameter-efficient tuning (Bao et al., 2023; Liao et al., 2024); the knowledge we study instead consists of collaborative relations and sequential transition patterns learned directly from accumulated user behavior, so LLM4Rec techniques may fall outside our scope. At the experimental-setup level, our streaming evaluation resembles incremental learning but differs in the problem studied: unlike invariant learning that seeks an environment-invariant mechanism (He et al., 2022; Wang et al., 2022) or incremental methods that maintain a discriminative predictor over incoming data (Wang et al., 2020, 2023), we investigate whether knowledge from autoregressive sequence pretraining transfers across the pretrain-to-task objective gap and stays effective under dynamically shifting data.

2. Preliminaries

Problem Setup. We study knowledge transfer under the generative-pretraining-to-discriminative-transfer paradigm for sequential recommendation, first formalized by GPSD (Wang et al., 2025), where a transformer is autoregressively pretrained on user behavior sequences and transferred to downstream tasks like recall and ranking. GPSD shows that, under this setting, transferred pretraining unlocks scaling behavior beyond discriminative training alone.

GPSD frames transfer along two orthogonal design axes: which parameters to carry across: the transformer’s dense parts or only the embeddings; and whether to freeze them. We argue that a streaming recommender raises dimensions these axes do not cover: (1) how to pretrain well on ever-growing industrial behavior streams, where adjacency does not imply dependency; and (2) the pretraining and task objectives conflict in the geometry they demand of the shared representation, and, under continual updates, both knowledge acquisition and transfer must remain stable across the refresh schedule. These are the dimensions this paper addresses.

Sequential recommendation. Let be the user set and the item catalog. Each user is represented by a chronologically ordered behavior sequence with . A transformer encoder maps a prefix to a contextual representation , and each item carries an embedding from a table .

Autoregressive pretraining. The encoder is pretrained by next-token prediction over user sequences, scoring items by inner product against the contextual state and normalizing over the catalog:

(1)

This yields item embeddings whose geometry reflects behavioral co-occurrence, which the downstream task then transfers and adapts.

Downstream transfer. A downstream task is trained on curated supervision. For example, CTR estimation uses exposed-but-unclicked items as negatives against clicked positives, typically under a calibration loss such as binary cross-entropy, to learn a discriminative geometry over the representation. Because this supervision differs from pretraining in both its negative distribution and its objective, the geometry it demands departs from the pretrained one, a gap we characterize in Sec. 3.2.

Streaming refresh. In deployment, the data arrives as a non-stationary stream. Rather than pretraining once, the encoder needs to be refreshed on the incoming window at update steps , producing (in production, daily). Transfer must therefore remain valid as the encoder moves from to : the central requirement our method targets.

3. Our Method KGD

Fig. 2 depicts Knowledge–Geometry Decoupling (KGD). Its premise is that pretrained knowledge and task-specific geometry need not compete for one representation. They can coexist as two layers under separate ownership: a behavioral geometry the encoder keeps refreshing, and a task geometry written orthogonally on top of it. KGD realizes this by answering the two questions in Sec. 1. For what to learn, the encoder is pretrained with Behavioral Multi-Token Prediction (BMTP), which supervises it only on transitions that generalize, building a clean base geometry (Sec. 3.1). For how to transfer, the task learner is linked to the encoder under decoupled read–write ownership through two one-way interfaces (Sec. 3.2). These read-only interfaces further expose a design space we instantiate per task and per deployment scenario (Sec. 3.3).

3.1. Behavioral Multi-Token Prediction (BMTP)

Pretraining by next-token prediction (Eq. 1) drives item embeddings whose inner products approximate the log-co-occurrence of adjacent items (Levy and Goldberg, 2014): the geometry it learns is exactly the geometry of what tends to follow. The question specific to recommendation is not how to predict the next item, but which co-occurrences are worth encoding into that geometry. As argued in Sec. 1, adjacency in behavior is a far weaker signal than in language: a sequence concatenates separate interest sessions (Fig. 1(a)), so a pair straddling a session boundary carries no dependency, and supervising it merely writes noise into the embedding geometry.

Two kinds of transition, by contrast, carry knowledge worth keeping: collaborative ones, where two items are frequently co-consumed even when dissimilar (the classic beer-and-diapers pairing), and semantic ones, where two items are close in content (a user drawn from one red dress to another). BMTP supervises the encoder on these, and only these. Concretely, for each position and each axis, we retain the nearest subsequent items whose similarity to exceeds a threshold: collaboratively, by proximity on a pretrained item co-occurrence graph (Yang et al., 2020); semantically, by cosine similarity between item embeddings precomputed once with a pretrained text-embedding model. Both similarities are computed offline and cached, adding no per-sequence model inference to pretraining. We predict these filtered items:

where and denote the collaborative and semantic similarities with thresholds ; contains the nearest qualifying item on axis , and any empty term is skipped. For efficiency, we supervise only the next qualifying item per axis: the catalog-wide softmax makes the cost scale with the number of positives, and predicting every qualifying future item would dominate training time at production scale, while the nearest few already carry the strongest dependency signal.

BMTP thus supervises the encoder on collaborative and semantic dependency rather than raw adjacency. The base geometry it produces is clean in a specific sense: its structure is set by collaborative and semantic relations that persist, not by the transient noise of session boundaries, so as the stream shifts, refreshing the encoder re-estimates this geometry on new data without accumulating that noise. This is the layer the task learner will read from and build on, and the reason it remains safe to refresh continually.

3.2. Decoupled Read–Write Ownership

A clean pretrained geometry is valuable only if a downstream task can adapt it into discriminative structure without destroying it. The two objectives, however, are not aligned: as Appendix B shows, their gradients on the shared embedding run from negatively correlated to near-orthogonal, so a single parameter set cannot serve both, and fine-tuning overwrites the pretrained geometry, whereas freezing forbids the task from reshaping it at all. KGD addresses this by assigning behavior-transition knowledge and task geometry to two separate owners—the pretrained encoder and the task learner—each holding its own parameters and coupled through two interfaces that leave the encoder’s weights untouched: at embedding level, the task learner superposes its own residual onto the pretrained embedding (i.e., Anchored Calibration Residual); at representation level, it attends to the encoder’s hidden states through read-only cross-attention. In both, gradients from the task never reach the encoder, so the pretrained geometry and the task geometry occupy separate, non-interfering layers: the encoder keeps refreshing the former while the task writes and reads the latter on top.

Anchored Calibration Residual (ACR). At the embedding level, the task learner does not alter the pretrained embedding but forms its own task-adapted embedding by superposing a task-owned residual onto a detached copy of it. We first characterize what this residual must be, relative to the pretrained geometry, and then parameterize exactly that.

Observation 1: in the idealized case, the residual reduces to a global rescaling. Consider a simple case: the task objective differs from pretraining solely by a temperature , scoring items against the same catalog-wide negatives. Its posterior is a temperature sharpening of the pretraining posterior and preserves its ranking. Equating the stationarity conditions of the two objectives,

shows that both are met by a global rescaling with . In this regime the task asks pretraining only for sharper contrast, and the pretrained geometry is preserved up to scale.

Observation 2: real tasks deform the geometry rather than rescale it. Two properties of practical training break this equivalence, and both are structural. First, the objective differs: downstream tasks optimize calibrated or discriminative losses—binary cross-entropy for CTR, for instance—whose stationary geometry is not a temperature scaling of a softmax ranking objective. Second, the negative distribution differs: it is deliberately not the catalog-wide distribution implicit in pretraining. Even retrieval, the task closest to pretraining, is trained with hard-negative sampling rather than a globally sampled softmax, while CTR and CVR draw negatives from exposed-but-unclicked and clicked-but-unconverted items respectively. Each choice displaces the task-optimal direction away from the pretrained behavioral one (the displacement quantified by the gradient-cosine trajectory in Fig. 1(b) and Appendix B), and does so differently for each task. The correction a task requires is therefore a task-conditioned, direction-dependent deformation of the embedding space, which no global scalar can express.

Observation 3: the deformation must not cancel the pretrained geometry. This freedom cannot be unconstrained. The pretrained embedding carries structure that is itself valuable for transfer (evidenced by the strength of embedding-only transfer in GPSD (Wang et al., 2025) and reproduced in our own experiments (Sec. 4)), and in KGD the task learner reads this knowledge back through the very embedding that ACR adapts. A deformation that rotated the embedding against its pretrained direction would cancel that signal at read time. The task therefore needs geometric freedom that is non-conflicting: it must add discriminative structure without reversing or overwriting the pretrained direction.

The above observations jointly fix the form of the correction. Freedom beyond a scalar (Observation 2) together with preservation of the pretrained direction (Observation 3) is attained exactly when the added component is orthogonal to : a task-wise scalar preserves and sharpens the pretrained direction, while a low-rank residual writes the task-conditioned deformation into its orthogonal complement,

(2)

where indexes downstream tasks, is the stop-gradient, and , are task-owned low-rank factors () kept orthogonal to by orthogonality regularization. The orthogonality constraint is what allows the two geometries to coexist: introduces discriminative directions in the complement of the pretrained subspace, so the task attains enough geometric freedom at the input level while (detached and shared across tasks) remains intact and refreshable.

Read-only cross-attention. ACR corrects geometry at the granularity of individual items, but a downstream task needs more than a sharper item space: it must project the user’s contextualized interest (behavior sequence distilled into intent) into a task-discriminative space where positives and negatives are maximally separated and the scores are calibrated. This contextual signal is produced only by the pretrained transformer’s intermediate layers and cannot be recovered from item embeddings alone. Sharing that transformer with the task, however, is not a good option: its parameters are specialized for modeling transitions, and re-optimizing for discrimination reintroduces exactly the conflicts. KGD therefore decouples the two backbones (the task owns a separate transformer) and bridges them with read-only cross-attention, through which the task transformer selectively extracts and re-projects the pretrained model’s contextualized representations without altering them. Concretely, let be the pretrained encoder’s hidden states over the ACR-adjusted embeddings. The task transformer reads them through

where is a task-side reader token and are owned by the task transformer; denotes the encoder’s final-layer hidden states, which the task reads. The stop-gradient on keys and values makes the interface strictly one-way: updates only the task transformer and ACR parameters.

Why this refreshes cleanly. Two properties, established above, make refresh and adaptation independent operations. First, the interface is strictly one-way: read-only cross-attention isolates gradients, so refreshing the transition backbone never disturbs the task backbone, and task optimization never pollutes pretrained knowledge. Second, the task geometry is written as a residual anchored relative to the pretrained embedding rather than at absolute coordinates, so a refresh moves the base beneath a residual that stays valid rather than invalidating it. Because BMTP keeps that base free of transient noise (Sec. 3.1), re-estimating it on new data accumulates knowledge instead of drift. The encoder can therefore refresh on the stream—as often as daily, at the cost of a single encoder update—while the task adaptation it carries remains intact. This is precisely the property a continually drifting stream demands, and Sec. 4 tests it directly under daily streaming schedules.

3.3. Instantiating, Training, and Serving KGD

We now realize the two owners as concrete transformers and describe how KGD is trained and served under a daily stream. More details are provided in Appendix C.4.

Architecture. The encoder is a standard self-attention transformer trained with BMTP (Sec. 3.1), which maps the raw item-embedding sequence to hidden states during its training. The task learner is a transformer with both self- and read-only cross-attention, and it consumes the encoder in two steps. First, ACR reshapes the input embeddings, and the encoder re-encodes them, giving task-adapted states . Second, the learner attends to these states, , where the task-side reader token forms the query and supplies the keys and values. The learner output is then passed to a task head under the task loss. During learner training, the encoder-owned parameters are frozen and run in inference only.

Task-specific query. The instantiation of the reader token follows the task’s demand. For retrieval, is a user-side vector encoding user-profile features or a learned user latent vector (Yang et al., 2026), producing a single user embedding for nearest-neighbor search. For ranking, is a candidate-side token that aggregates the candidate’s features (Zhang et al., 2026; Tang et al., 2026), so each candidate reads task-relevant signal from one shared user history encoding and is scored independently.

Training under the daily stream. Data arrives daily. Each day we first refresh the encoder on the new data for one pass; we then freeze the encoder-owned parameters and run the combined encoder-learner graph, updating only the learner while the encoder only does inference. The training time cost is therefore roughly twice an encoder-only pass, about two hours in our deployment on A100, on par with the incremental training cost of GPSD (Wang et al., 2025), which is unavoidable due to the objective conflicts.

Serving. The learner owns separate attention, FFN parameters, and item-catalog embedding in low dimensions, so KGD roughly doubles the dense parameters and 20% sparse embeddings, which remains manageable at our scale. Latency does not grow: if the query token does not go through our separate learner, it still goes through the shared encoder in a non-pretraining setting (Tang et al., 2026; Zhang et al., 2026), thus our per-request latency aligns with single-backbone baselines.

4. Offline Experiments

We evaluate KGD on reproducible benchmarks isolating what to learn and how to transfer, and an industrial deployment testing refreshability. Complete implementation details are in Appendix C.

4.1. Setup

Strategy	Pretrain	Arts		Beauty		CDs		Phones		Office		Software		Toys		Games
		N@50	R@50	N@50	R@50	N@50	R@50	N@50	R@50	N@50	R@50	N@50	R@50	N@50	R@50	N@50	R@50
Scratch	-	0.0302	0.0807	0.0243	0.0595	0.0618	0.1552	0.0242	0.0635	0.0279	0.0640	0.1257	0.3374	0.0315	0.0755	0.0670	0.1800
TE&FT	NTP	0.0305	0.0834	0.0229	0.0632	0.0636	0.1555	0.0292	0.0808	0.0279	0.0658	0.1279	0.3426	0.0308	0.0764	0.0666	0.1903
	MTP	0.0308	0.0878	0.0229	0.0604	0.0632	0.1527	0.0256	0.0682	0.0274	0.0657	0.1295	0.3463	0.0313	0.0766	0.0664	0.1865
	BMTP	0.0304	0.0901	0.0229	0.0639	0.0618	0.166	0.0268	0.0786	0.0271	0.0735	0.1293	0.3430	0.0314	0.0866	0.0669	0.1914
\rowcolorgainrowcolor	BMTP vs. NTP gain	-0.30%	8.00%	0.00%	1.10%	-2.80%	6.80%	-8.20%	-2.70%	-2.90%	11.70%	1.10%	0.10%	1.90%	13.40%	0.50%	0.60%
TA&FT	NTP (Song et al., 2025)	0.0299	0.0825	0.0220	0.0604	0.0625	0.1594	0.0267	0.0751	0.0277	0.0671	0.1275	0.3380	0.0323	0.0772	0.0675	0.1935
	MTP (Chen et al., 2025; Guo et al., 2026; Lin et al., 2025)	0.0314	0.0877	0.0221	0.0596	0.0645	0.1512	0.0245	0.0640	0.0274	0.0659	0.1276	0.3408	0.0304	0.0769	0.0670	0.1880
	BMTP	0.0304	0.0805	0.0246	0.0625	0.0634	0.1715	0.0254	0.0667	0.0272	0.0658	0.1291	0.3438	0.0322	0.0761	0.0669	0.1927
\rowcolorgainrowcolor	BMTP vs. NTP gain	1.70%	-2.40%	11.80%	3.50%	1.40%	7.60%	-4.90%	-11.20%	-1.80%	-1.90%	1.30%	1.70%	-0.30%	-1.40%	-0.90%	-0.40%
TE&FE	NTP (Wang et al., 2025, 2026b)	0.0353	0.1011	0.0261	0.0725	0.0637	0.1734	0.0321	0.0879	0.0309	0.0826	0.1320	0.3696	0.0325	0.0888	0.0754	0.2127
	MTP	0.0339	0.0956	0.0249	0.0678	0.0620	0.1674	0.0277	0.0741	0.0287	0.0744	0.1306	0.3660	0.0318	0.0844	0.0735	0.2068
	BMTP	0.0376	0.1069	0.0270	0.0755	0.0690	0.1893	0.0330	0.0933	0.0334	0.0895	0.1349	0.3732	0.0354	0.0955	0.0782	0.2208
\rowcolorgainrowcolor	BMTP vs. NTP gain	6.50%	5.70%	3.40%	4.10%	8.30%	9.20%	2.80%	6.10%	8.10%	8.40%	2.20%	1.00%	8.90%	7.50%	3.70%	3.80%
TA&FE	NTP (Wang et al., 2025)	0.0340	0.0987	0.0256	0.0709	0.0649	0.1780	0.0308	0.0855	0.0306	0.0826	0.1325	0.3720	0.0326	0.0892	0.0754	0.2149
	MTP	0.0327	0.0927	0.0237	0.0649	0.0619	0.1685	0.0247	0.0716	0.0287	0.0745	0.1304	0.3638	0.0314	0.0841	0.0728	0.2041
	BMTP	0.0364	0.1045	0.0273	0.0763	0.0691	0.1901	0.0327	0.0947	0.0332	0.0894	0.1345	0.3694	0.0351	0.0951	0.0772	0.2187
\rowcolorgainrowcolor	BMTP vs. NTP gain	7.10%	5.90%	6.60%	7.60%	6.50%	6.80%	6.20%	10.80%	8.50%	8.20%	1.50%	-0.70%	7.70%	6.60%	2.40%	1.80%
TA&FD	NTP	0.0229	0.0724	0.0160	0.0494	0.0531	0.1562	0.0212	0.0653	0.0197	0.0605	0.0887	0.2796	0.0216	0.0644	0.0524	0.1632
	MTP	0.0275	0.0857	0.0181	0.0549	0.0522	0.1530	0.0174	0.0534	0.0230	0.0658	0.0859	0.2735	0.0246	0.0715	0.0556	0.1705
	BMTP	0.0268	0.0860	0.0178	0.0539	0.0531	0.1575	0.0229	0.0704	0.0238	0.0713	0.1005	0.3141	0.0246	0.0727	0.0605	0.1813
\rowcolorgainrowcolor	BMTP vs. NTP gain	17.00%	18.80%	11.30%	9.10%	0.00%	0.80%	8.00%	7.80%	20.80%	17.90%	13.30%	12.30%	13.90%	12.90%	15.50%	11.10%
TA&FA	NTP (Yuan et al., 2020)	0.0266	0.0809	0.0192	0.0582	0.0412	0.1232	0.0257	0.0760	0.0253	0.0726	0.1196	0.3436	0.0199	0.0582	0.0535	0.1611
	MTP	0.0236	0.0727	0.0171	0.0525	0.0388	0.1172	0.0186	0.0561	0.0226	0.0655	0.1179	0.3446	0.0179	0.0533	0.0506	0.1502
	BMTP	0.0300	0.0894	0.0222	0.0655	0.0515	0.1477	0.0292	0.0842	0.0277	0.0754	0.1202	0.3438	0.0266	0.0752	0.0653	0.1895
\rowcolorgainrowcolor	BMTP vs. NTP gain	12.80%	10.50%	15.60%	12.50%	25.00%	19.90%	13.60%	10.80%	9.50%	3.90%	0.50%	0.10%	33.70%	29.20%	22.10%	17.60%
KGD	NTP	0.0354	0.1024	0.0262	0.0734	0.0666	0.1844	0.0317	0.0865	0.0317	0.0857	0.1347	0.3662	0.0339	0.0910	0.0758	0.2189
	MTP	0.0344	0.0979	0.0266	0.0731	0.0635	0.1751	0.0271	0.0725	0.0304	0.0784	0.1340	0.3618	0.0312	0.0843	0.0739	0.2071
	BMTP	0.0380	0.1099	0.0292	0.0803	0.0701	0.1930	0.0337	0.0951	0.0352	0.0938	0.1381	0.3775	0.0365	0.0975	0.0788	0.2244
\rowcolorgainrowcolor	BMTP vs. NTP gain	7.30%	7.30%	11.50%	9.40%	5.30%	4.70%	6.30%	9.90%	11.00%	9.50%	2.50%	3.10%	7.70%	7.10%	4.00%	2.50%
\rowcolorgreen!8     gain vs. best published baseline		7.60%	8.70%	11.90%	10.80%	8.00%	8.40%	5.00%	8.20%	13.90%	13.60%	4.20%	1.50%	12.00%	9.30%	4.50%	4.40%
\rowcolorgreen!8     gain vs. best baseline w/ BMTP		1.06%	2.81%	6.96%	5.24%	1.45%	1.53%	2.12%	0.42%	5.39%	4.8%	2.37%	1.15%	3.11%	2.09%	0.77%	1.63%

Public benchmark setting. We use 8 datasets of Amazon-2023 Reviews (Hou et al., 2024) under single-task modeling, with a reasoning-enhanced SOTA sequential recommender, ManCAR (Yang et al., 2026), as the backbone. On each training split we first pretrain the encoder autoregressively on the full user behavior sequences with BMTP, then reorganize the data into a reasoning-recommendation task to adapt the task learner while it reads the encoder through the read-only interface. The official static split is not oriented toward test refreshability, but it cleanly separates the quality of the pretrained knowledge (BMTP) from the way it is transferred (decoupled ownership) under a strong shared backbone that rules out any weak-backbone confound. We report NDCG@ and Recall@ with as metrics.

Industrial streaming setting. The industrial data comes from Shopee Homepage Search scenario, a billion-scale e-commerce platform with multi-task ranking (click + order) over a daily traffic stream, using a strong Transformer-based industrial ranker, OneRank (Tang et al., 2026), as the backbone. We evaluate each task using AUC and GAUC over two horizons. The 28-day stream is used for the main comparison: a broad sweep of pretraining-objective transfer-method combinations, each averaged over 5 runs and evaluated by AUC under a test-before-train protocol (the model cannot be trained on a new-day’s data before being tested on this day); We report the last-day’s test metrics of each method. The 90-day trajectory then tracks a selected subset—the strongest performers from the 28-day sweep together with representative baselines (e.g., TA&FE, defined below, which corresponds to GPSD (Wang et al., 2025))—to observe how each method holds up over time.

Because streaming data cannot be assessed on a fixed split, we evaluate under three schedules that couple pretraining and refresh differently, which jointly isolate the two axes of our claim:

• S1 — no pretraining. The standard industrial incremental-refresh recipe: each day loads the previous day’s checkpoint and continues training on the new day’s data. This is the shared base recipe of OneRank.

• S2 — pretrain then freeze. After pretraining for the first 14 days, the encoder-produced parameters are then frozen, and only the task-specific module is trained on subsequent days. The encoder is pretrained once and never refreshed afterwards.

• S3 — pretrain–adapt alternation. Each day first refreshes the encoder for one pass over the day’s data, then, depending on the transfer method, freezes, migrates, fine-tunes, or applies KGD’s decoupled training. S3 is what lets us disentangle the value of refresh (S2 vs. S3) from the value of transfer mechanisms.

Baselines. For the pretraining objective, we compare BMTP against Next-Token Prediction (NTP) (Yuan et al., 2020; Wang et al., 2026b; Song et al., 2025; Wang et al., 2025) and Multi-Token Prediction (MTP) (Chen et al., 2025; Guo et al., 2026; Lin et al., 2025). For transfer, we cover the paradigms that, as argued in Sec. 1, each fail differently under drift:

• No pretraining (S1) — Scratch denotes the standard OneRank baseline; IncCTR (Wang et al., 2020) treats the previous-day checkpoint as teacher and distill its score to today’s student; and buffer replay (Rolnick et al., 2019), which rehearses historical samples. None draws structural knowledge from pretraining.

• Fine-tuning (S2) — transfer Embeddings then fine-tune all (TE&FT), and transfer All modules then fully fine-tune (TA&FT) (Song et al., 2025; Chen et al., 2025; Guo et al., 2026; Lin et al., 2025); task gradients overwrite pretrained knowledge.

• Frozen transfer (S2) — the pretrained parameters are frozen and then copied as transfer: transferring only the embeddings and freezing them (TE&FE) (Wang et al., 2025, 2026b); transferring all parameters and freezing the embeddings (TA&FE) (Wang et al., 2025); transferring all and freezing the dense (Transformer) parameters (TA&FD); and transferring all and freezing everything except the task head (TA&FA) (Yuan et al., 2020).

• Entangled refresh (S3) — finetune the same backbone under S3 schedule; the encoder is refreshed daily, then we finetune the task on the same parameters.

• Partial separation (S3) — LoRA/adapter refresh (Hu et al., 2022). Under S3, each day the base is first trained with BMTP, and low-rank adapters on the attention layers are then updated under the task loss with the base frozen.

• Decoupled ownership (S3) — KGD.

媒体内容 · 前往原文查看

Strategy	Pretrain	click AUC	click GAUC	order AUC	order GAUC
Scratch (S1)	–	0.7806	0.7759	0.8941	0.8410
TE&FT (S2)	NTP	0.7826	0.7786	0.8981	0.8434
	MTP	0.7827	0.7785	0.8982	0.8440
	BMTP	0.7839	0.7795	0.8985	0.8444
TA&FT (S2)	NTP	0.7824	0.7787	0.8982	0.8437
	MTP	0.7830	0.7789	0.8980	0.8436
	BMTP	0.7852	0.7801	0.8993	0.8447
TE&FE (S2)	NTP	0.7821	0.7779	0.8977	0.8421
	MTP	0.7820	0.7778	0.8980	0.8427
	BMTP	0.7834	0.7786	0.8985	0.8448
TA&FE (S2)	NTP	0.7813	0.7780	0.8975	0.8421
	MTP	0.7825	0.7781	0.8980	0.8423
	BMTP	0.7835	0.7789	0.8991	0.8434
TA&FD (S2)	NTP	0.7813	0.7765	0.8981	0.8428
	MTP	0.7815	0.7768	0.8981	0.8429
	BMTP	0.7831	0.7779	0.8997	0.8449
TA&FA (S2)	NTP	0.7816	0.7761	0.8970	0.8424
	MTP	0.7812	0.7762	0.8971	0.8423
	BMTP	0.7821	0.7772	0.8984	0.8431
TA&FT (S3)	NTP	0.7819	0.7781	0.8978	0.8432
	MTP	0.7825	0.7781	0.8975	0.8429
	BMTP	0.7837	0.7793	0.8979	0.8433
TA&FE (S3)	NTP	0.7822	0.7783	0.8979	0.8432
	MTP	0.7825	0.7785	0.8978	0.8433
	BMTP	0.7841	0.7792	0.8989	0.8446
KGD (S3)	NTP	0.7837	0.7804	0.9003	0.8460
	MTP	0.7825	0.7785	0.8989	0.8459
	BMTP	0.7867	0.7826	0.9015	0.8477
KGD w/o ACR and RO	BMTP	0.7785	0.7737	0.8919	0.8382
IncCTR	–	0.7810	0.7744	0.8936	0.8397
LoRA	BMTP	0.7818	0.7773	0.8967	0.8423
Buffer replay	–	0.7732	0.7664	0.8813	0.8289

4.2. Public Benchmark Results

Tab. 4.1 reports results on eight datasets. KGD with BMTP attains the best score on every dataset, improving over the strongest published baseline by 4–12%. This margin mixes two gains, from BMTP and from the architecture; we separate them below.

BMTP supplies better behavioral knowledge, which is only realizable if it is not overwritten. Holding the transfer strategy fixed, replacing NTP with BMTP helps when architectures and datasets vary, so the gain comes from the pretraining objective rather than from KGD alone: filtering adjacency-based supervision along collaborative and semantic axes gives the encoder cleaner knowledge. The size of that gain, however, depends sharply on how the knowledge is transferred. Under frozen transfer the BMTP-over-NTP gain is large and consistent (e.g., / on Arts under TE&FE/TA&FE), and it grows as more of the pretrained model is frozen, reaching double digits under TA&FD and TA&FA. Under full fine-tuning, by contrast, the same advantage shrinks or reverses (e.g., on Phones under TE&FT). The pattern is telling: when task gradients are allowed to rewrite the pretrained parameters, they erase precisely the structure BMTP installed. This evidences both that BMTP encodes real transferable knowledge and that preserving it requires a transfer that does not overwrite it.

Decoupled ownership transfers that knowledge best. Controlling for the pretraining objective isolates the architectural contribution: with the same BMTP-pretrained encoder, KGD still exceeds the best shared-parameter baseline on every dataset (by 0.4–7.0%, the “gain vs. best baseline w/ BMTP” row). Three properties of KGD explain the margin, none available to shared parameters. Read-only cross-attention lets the task learner draw on the encoder’s contextualized intermediate representations without any gradient reaching them, so the knowledge is never overwritten. ACR writes the task’s discriminative directions as a residual orthogonal to the pretrained embedding, adding separation in the complement of the pretrained subspace rather than rotating against it, so calibration does not cancel the transferred knowledge. And the decoupled backbones let the task learner own its discriminative geometry end-to-end instead of borrowing a transition-specialized one. Frozen transfer preserves knowledge but cannot reshape geometry; fine-tuning reshapes geometry but destroys knowledge; only decoupled ownership does both without conflicts.

媒体内容 · 前往原文查看

Setting	Office		Software		Industrial Data
	N@50	R@50	N@50	R@50	click AUC	click GAUC	order AUC	order GAUC
w/o semantic BMTP	0.0337	0.0886	0.1374	0.3748	0.7859	0.7818	0.9002	0.8469
w/o collaborative BMTP	0.0338	0.0908	0.1363	0.3731	0.7854	0.7810	0.8986	0.8459
w/o ACR	0.0350	0.0932	0.1369	0.3753	0.7829	0.7790	0.8950	0.8426
ACR + share-param + freeze dense	0.0279	0.0771	0.1259	0.3568	0.7833	0.7797	0.8972	0.8435
ACR + share-param + finetune all	0.0331	0.0884	0.1344	0.3659	0.7847	0.7811	0.9000	0.8451
Align capacity with KGD	0.0267	0.0608	0.1224	0.3240	0.7839	0.7802	0.8962	0.8439
KGD full	0.0352	0.0938	0.1381	0.3775	0.7867	0.7826	0.9015	0.8477

4.3. Industrial Data Results

Tab. 2 reports the 28-day comparison and Fig. 3 provides the 90-day trajectory. KGD attains the best score on all metrics; the controlled schedules attribute this to its two ingredients.

Ownership, not schedule, is the decisive factor. Comparing S2 (pretrain once, then freeze) with S3 (refresh daily) isolates continual refresh’s gain. On shared parameters, refresh does not help: TA&FT even drops from 0.7852 (S2) to 0.7837 (S3), and TA&FE barely moves to 0.7841. Refreshing and adapting the same parameters cancel out each other iteratively. Under the same S3 schedule, KGD reaches 0.7867 by decoupling the two, and removing its interfaces (no ACR or read-only encoder) collapses S3 to entangled refresh at 0.7785, below Scratch (0.7806). Refresh thus pays off only under decoupled ownership. Under entangled parameters, the alternating updates repeatedly overwrite the geometry, preventing it from stabilizing and thereby degrading performance.

Every alternative fails in its own way. The remaining baselines trace distinct failure modes. Without pretraining, continual methods gain little: IncCTR (0.7810) barely exceeds Scratch, distilling a previous-day checkpoint that adds no structural signal; buffer replay is worse (0.7732, below Scratch), as rehearsing history amounts to multi-epoch training that triggers the overfit-beyond-one-epoch issue of sparse embeddings (Zhang et al., 2022). LoRA (0.7818), low-rank adapters on attention, leaves too little freedom to reshape geometry.

The 90-day trajectory reveals what a snapshot hides. Frozen transfer looks competitive within a short time window, which is why refreshability and stability cannot be judged from a static table. Over 90 days (Fig. 3), frozen transfer (TA&FE, i.e., GPSD) erodes as time extends, while KGD holds throughout. Buffer replay degrades monotonically: the stale patterns it memorizes never recur, echoing our measurement (details in Appendix B.1 Fig. 4) that the item distribution’s JSD keeps rising rather than stabilizing. Whereas KGD, refreshing on the live stream, tracks the drift.

Scaling ability. Because ownership is decoupled, capacity can be allocated to the two sides independently (in Appendix D, Tab. D): enlarging the encoder refines the refreshed behavioral knowledge, enlarging the task learner expresses richer geometry, and per-task learner parameters add a further gain. Unlike a shared backbone, where added capacity is split between two contending objectives, decoupling turns scaling into a targeted allocation—encoder capacity for knowledge, task-learner capacity for geometry—aligned with how cost is budgeted in deployment (Sec. 3.3).

4.4. Ablation Study

Tab. 3 reports the ablation study of KGD’s three components, and we provide analyses as follows:

BMTP (what to learn). The two filters play complementary roles, most evident at industrial scale: removing the collaborative (graph) filter costs AUC on head-user requests (top by activity), while removing the semantic filter costs AUC on tail users (bottom ). Head users’ behavior are well captured by the co-occurrence graph, whereas sparse tail users rely on semantic coherence. On Amazon no such separation appears, as its aggressive sampling distorts the real user distribution.

ACR (writing geometry). Removing ACR and reading the pretrained embedding directly hurts across the board: the task cannot obtain its discriminative geometry from a read-only encoder alone.

Read-only decoupling (backbone separation). Keeping ACR but placing it on a shared backbone, whether frozen or fully fine-tuned, is still inferior to full KGD: item-level calibration alone is not enough. On shared parameters, knowledge and geometry keep competing, and the task head cannot obtain a contextualized representation re-projected for its own objective, which only a separate task backbone reading the encoder can provide.

Capacity is not the cause. Matching a shared backbone to KGD’s parameter count still trails KGD, and even regresses on sparse public datasets where the enlarged shared space overfits. The gain comes from how capacity is owned—knowledge on the encoder, geometry on the task learner—not from its amount.

5. Online A/B Testing

We deploy KGD at the ranking stage of the Shopee Homepage Search scenario, the primary search entry serving billion-scale users. Traffic is split by user-ID hash into control and treatment buckets of each (over ten million users per bucket), with an A/A bucket for validation. The control is the production OneRank model trained from scratch under continual updates for over six months; the treatment is KGD, brought online after months of offline alternating training and updated on the same daily stream. The test ran for two weeks in the first half of 2026.

The A/A bucket showed no significant difference, while KGD improved GMV per user by , advertising revenue per user by , CTR by , and CVR by , with GMV and revenue gains significant at (two-sample -test on the per-user distributions). The smaller order-side gains are expected, as order labels are sparse and harder for pretraining to capture, and human evaluation found no increase in the per-query irrelevant-item rate.

A one-week reversal after rollout reduced GMV by and revenue by ; the forward–reversal asymmetry reflects the high variance of GMV rather than an unstable gain. Costs are modest: daily offline training rises from one to two hours on the same A100 count, on par with GPSD, and serving latency stays at  ms on A30 GPUs, with memory doubled by the read-only learner.

6. Conclusion

In this paper, we revisited the pretrain-then-transfer recipe for recommendation under streaming production updates, where pretraining must be refreshed rather than handed off once, and identified two limitations: the next-token objective compresses noise, and pretrained knowledge conflicts with task-specific geometry on shared parameters. We proposed Knowledge–Geometry Decoupling, i.e., KGD. In KGD, Behavioral Multi-Token Prediction (BMTP) builds a clean, refreshable base knowledge, while a task learner reads it through read-only cross-attention and writes its own geometry via the Anchored Calibration Residual (ACR), making daily refresh and task adaptation independent. KGD improves over the strongest baselines by 4–12% on eight public benchmarks and lifts GMV per user by 1.75% in a live A/B test with full deployment on Shopee.

References

• K. Bao, J. Zhang, Y. Zhang, W. Wang, F. Feng, and X. He (2023) TALLRec: an effective and efficient tuning framework to align large language model with recommendation. In RecSys, pp. 1007–1014. Cited by: §1.
• X. Chen, K. Rajesh, M. Lawhon, Z. Wang, H. Li, H. Li, S. V. Joshi, P. Eksombatchai, J. Yang, Y. Hsu, J. Xu, and C. Rosenberg (2025) PinFM: foundation model for user activity sequences at a billion-scale visual discovery platform. In RecSys, pp. 381–390. Cited by: Appendix A, Appendix A, Appendix D, §1, 2nd item, §4.1, §4.1.
• C. Fu, K. Wang, J. Wu, Y. Chen, G. Huzhang, Y. Ni, A. Zeng, and Z. Zhou (2024) Residual multi-task learner for applied ranking. In KDD, pp. 4974–4985. Cited by: §C.1.
• Z. Gan, L. Li, C. Li, L. Wang, Z. Liu, and J. Gao (2022) Vision-language pre-training: basics, recent advances, and future trends. Found. Trends Comput. Graph. Vis. 14 (3-4), pp. 163–352. Cited by: §1.
• C. Gao, S. Li, Y. Zhang, J. Chen, B. Li, W. Lei, P. Jiang, and X. He (2022) Kuairand: an unbiased sequential recommendation dataset with randomly exposed videos. In CIKM, pp. 3953–3957. Cited by: §C.1.
• D. Guo, S. Wang, Q. Xiao, Y. Ren, W. Li, S. Xu, M. Yue, B. Huang, G. Wu, and C. Luo (2026) Climber-pilot: A non-myopic generative recommendation model towards better instruction-following. arXiv Preprint. External Links: Link Cited by: Appendix A, Appendix A, Appendix D, 2nd item, §4.1, §4.1.
• R. Han, Q. Li, H. Jiang, R. Li, Y. Zhao, X. Li, and W. Lin (2024) Enhancing ctr prediction through sequential recommendation pre-training: introducing the srp4ctr framework. In CIKM, pp. 3777–3781. Cited by: Appendix A.
• X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang (2020) Lightgcn: simplifying and powering graph convolution network for recommendation. In SIGIR, pp. 639–648. Cited by: §C.4.
• Y. He, Z. Wang, P. Cui, H. Zou, Y. Zhang, Q. Cui, and Y. Jiang (2022) Causpref: causal preference learning for out-of-distribution recommendation. In WWW, pp. 410–421. Cited by: §1.
• Y. Hou, J. Li, Z. He, A. Yan, X. Chen, and J. J. McAuley (2024) Bridging language and items for retrieval and recommendation. arXiv Preprint. External Links: Link Cited by: §4.1.
• Y. Hou, S. Mu, W. X. Zhao, Y. Li, B. Ding, and J. Wen (2022) Towards universal sequence representation learning for recommender systems. In KDD, pp. 585–593. Cited by: Appendix A.
• E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In ICLR, Cited by: §1, 5th item.
• Y. Huang, Y. Chen, X. Cao, R. Yang, M. Qi, Y. Zhu, Q. Han, Y. Liu, Z. Liu, X. Yao, Y. Jia, L. Ma, Y. Zhang, T. Zhu, L. Zhang, L. Chen, W. Chen, M. Zhu, R. Xu, and L. Zhang (2025) Towards large-scale generative ranking. arXiv Preprint. External Links: Link Cited by: Appendix A.
• W. Kang and J. McAuley (2018) Self-attentive sequential recommendation. In ICDM, pp. 197–206. Cited by: Appendix A, §1.
• O. Levy and Y. Goldberg (2014) Neural word embedding as implicit matrix factorization. In NIPS, Vol. 27, pp. 2177–2185. Cited by: §3.1.
• Y. Li, B. Chen, M. Cheng, Z. Liu, X. Zhang, C. Lei, and W. Ou (2026) KuaiSearch: a large-scale e-commerce search dataset for recall, ranking, and relevance. arXiv preprint arXiv:2602.11518. Cited by: §C.1.
• J. Liao, S. Li, Z. Yang, J. Wu, Y. Yuan, X. Wang, and X. He (2024) LLaRA: large language-recommendation assistant. In SIGIR, pp. 1785–1795. Cited by: §1.
• Z. Lin, Z. Li, C. Dai, W. Bao, S. Lin, E. Yu, H. Zhang, and L. Zhao (2025) GReF: a unified generative framework for efficient reranking via ordered multi-token prediction. In CIKM, pp. 5879–5887. Cited by: Appendix A, Appendix A, Appendix D, 2nd item, §4.1, §4.1.
• J. Liu, Z. Dou, G. Tang, and S. Xu (2023a) Jdsearch: a personalized product search dataset with real queries and full interactions. In SIGIR, pp. 2945–2952. Cited by: §C.1.
• P. Liu, L. Zhang, and J. A. Gulla (2023b) Pre-train, prompt, and recommendation: A comprehensive survey of language modeling paradigm adaptations in recommender systems. Trans. Assoc. Comput. Linguistics 11, pp. 1553–1571. Cited by: §1.
• B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth (2024) Recent advances in natural language processing via large pre-trained language models: A survey. ACM Comput. Surv. 56 (2), pp. 30:1–30:40. Cited by: §1.
• S. Rajput, N. Mehta, A. Singh, R. H. Keshavan, T. Vu, L. Heldt, L. Hong, Y. Tay, V. Q. Tran, J. Samost, M. Kula, E. H. Chi, and M. Sathiamoorthy (2023) Recommender systems with generative retrieval. In NeurIPS, Vol. 36, pp. 10299–10315. Cited by: Appendix A.
• D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne (2019) Experience replay for continual learning. NIPS 32. Cited by: 1st item.
• C. Song, C. Shen, H. Gu, Y. Wu, L. Yi, J. Wen, and C. Chen (2025) PRECISE: pre-training and fine-tuning sequential recommenders with collaborative and semantic information. In CIKM, pp. 6029–6037. Cited by: Appendix A, Appendix A, Appendix D, 2nd item, §4.1, §4.1.
• H. Tang, J. Liu, M. Zhao, and X. Gong (2020) Progressive layered extraction (ple): a novel multi-task learning (mtl) model for personalized recommendations. In SIGIR, pp. 269–278. Cited by: Appendix A.
• J. Tang, S. Dai, K. Wang, Z. Guo, Y. Zhao, C. Fu, K. Wu, Y. Ni, A. Zeng, X. Chen, and J. Xu (2026) OneRank: unified transformer-native ranking architecture for multi-task recommendation. CoRR abs/2606.16838. Cited by: §C.3, §3.3, §3.3, §4.1.
• B. Wang, F. Liu, C. Zhang, J. Chen, Y. Wu, S. Zhou, X. Lou, J. Wang, Y. Feng, C. Chen, and C. Wang (2026a) LLM4DSR: leveraging large language model for denoising sequential recommendation. ACM Trans. Inf. Syst. 44 (1), pp. 6:1–6:32. Cited by: §1.
• C. Wang, B. Wu, Z. Chen, L. Shen, B. Wang, and X. Zeng (2025) Scaling transformers for discriminative recommendation via generative pretraining. In KDD, pp. 2893–2903. Cited by: Appendix A, Appendix A, Appendix D, Appendix D, §1, §1, §2, §3.2, §3.3, 3rd item, §4.1, §4.1, §4.1, §4.1.
• C. Wang, B. Wu, T. Pang, J. Wang, J. Yang, J. Liu, H. Zhang, H. Zhu, L. Shen, S. Wang, et al. (2026b) SORT: a systematically optimized ranking transformer for industrial-scale recommenders. arXiv Preprint. External Links: Link Cited by: Appendix A, Appendix A, Appendix D, 3rd item, §4.1, §4.1.
• Q. Wang, J. Li, S. Wang, Q. Xing, R. Niu, H. Kong, R. Li, G. Long, Y. Chang, and C. Zhang (2024) Towards next-generation llm-based recommender systems: A survey and beyond. arXiv Preprint https://arxiv.org/abs/2410.19744. Cited by: §1.
• Q. Wang, H. Lu, Y. Chen, D. Zhou, D. Zhan, M. Chen, and H. Ye (2023) Streaming CTR prediction: rethinking recommendation task for real-world streaming data. CoRR abs/2307.07509. Cited by: §1.
• R. Wang, R. Shivanna, D. Cheng, S. Jain, D. Lin, L. Hong, and E. Chi (2021) Dcn v2: improved deep & cross network and practical lessons for web-scale learning to rank systems. In WWW, pp. 1785–1797. Cited by: Appendix A.
• Y. Wang, H. Guo, R. Tang, Z. Liu, and X. He (2020) A practical incremental method to train deep CTR models. CoRR abs/2009.02147. Cited by: §1, 1st item.
• Z. Wang, Y. He, J. Liu, W. Zou, P. S. Yu, and P. Cui (2022) Invariant preference learning for general debiasing in recommendation. In KDD, pp. 1969–1978. Cited by: §1.
• B. Wu, F. Yang, Z. Chan, Y. Gu, J. Feng, C. Yi, X. Sheng, H. Zhu, J. Xu, M. Ye, et al. (2025) MUSE: a simple yet effective multimodal search-based framework for lifelong user interest modeling. arXiv preprint arXiv:2512.07216. Cited by: §C.1.
• L. Wu, Z. Zheng, Z. Qiu, H. Wang, H. Gu, T. Shen, C. Qin, C. Zhu, H. Zhu, Q. Liu, H. Xiong, and E. Chen (2024) A survey on large language models for recommendation. World Wide Web (WWW) 27 (5), pp. 60. Cited by: §1.
• K. Yang, Y. Zhu, Y. Chen, S. Zheng, B. Hong, K. Wu, Y. Ni, A. Zeng, C. Fu, and H. Li (2026) ManCAR: manifold-constrained latent reasoning with adaptive test-time computation for sequential recommendation. In KDD, Cited by: §C.3, §3.3, §4.1.
• X. Yang, Y. Zhu, Y. Zhang, X. Wang, and Q. Yuan (2020) Large scale product graph construction for recommendation in e-commerce. arXiv Preprint. External Links: Link Cited by: §3.1.
• F. Yuan, X. He, A. Karatzoglou, and L. Zhang (2020) Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. In SIGIR, pp. 1469–1478. Cited by: Appendix A, Appendix A, Appendix D, 3rd item, §4.1, §4.1.
• G. Yuan, F. Yuan, Y. Li, B. Kong, S. Li, L. Chen, M. Yang, C. Yu, B. Hu, Z. Li, et al. (2022) Tenrec: a large-scale multipurpose benchmark dataset for recommender systems. NIPS 35, pp. 11480–11493. Cited by: §C.1.
• J. Yuan, Z. Song, M. Sun, X. Wang, and W. X. Zhao (2021) Dual sparse attention network for session-based recommendation. In AAAI, pp. 4635–4643. Cited by: §1.
• J. Zhai, L. Liao, X. Liu, Y. Wang, R. Li, X. Cao, L. Gao, Z. Gong, F. Gu, M. He, et al. (2024) Actions speak louder than words: trillion-parameter sequential transducers for generative recommendations. arXiv Preprint. External Links: Link Cited by: Appendix A.
• Z. Zhang, X. Sheng, Y. Zhang, B. Jiang, S. Han, H. Deng, and B. Zheng (2022) Towards understanding the overfitting phenomenon of deep click-through rate models. In CIKM, pp. 2671–2680. Cited by: §4.3.
• Z. Zhang, H. Pei, J. Guo, T. Wang, Y. Feng, H. Sun, S. Liu, and A. Sun (2026) Onetrans: unified feature interaction and sequence modeling with one transformer in industrial recommender. In WWW, pp. 8162–8170. Cited by: §3.3, §3.3.
• G. Zhou, X. Zhu, C. Song, Y. Fan, H. Zhu, X. Ma, Y. Yan, J. Jin, H. Li, and K. Gai (2018) Deep interest network for click-through rate prediction. In KDD, pp. 1059–1068. Cited by: Appendix A.

Appendix A Related Work

Autoregressive pretraining on user behavior. Inspired by the success of LLM pretraining, self-attention-based autoregressive modeling of user behavior sequences has become a widely used practice in recommender systems (Kang and McAuley, 2018; Rajput et al., 2023; Hou et al., 2022; Zhai et al., 2024; Huang et al., 2025). Recent study further defines this as Generative Pretraining (Wang et al., 2025), which pretrains recommenders on large-scale behavior sequences via Next-Token Prediction (Yuan et al., 2020; Wang et al., 2026b; Song et al., 2025; Wang et al., 2025) and Multi-Token Prediction (Chen et al., 2025; Guo et al., 2026; Lin et al., 2025) to capture reusable behavioral knowledge. Such pretrained models provide useful initialization for downstream tasks (Wang et al., 2021; Zhou et al., 2018; Tang et al., 2020).

Task-specific adaptation of pretrained knowledge. Such knowledge adapts to diverse downstream scenarios, including instruction-aware retrieval (Guo et al., 2026), CTR prediction (Han et al., 2024), reranking (Lin et al., 2025), and cross-domain recommendation (Song et al., 2025), mainly differing in whether pretrained parameters are updated during task-specific training. Direct fine-tuning transfers the pretrained modules to the target scenario and keeps optimizing them under downstream supervision (Song et al., 2025; Chen et al., 2025; Lin et al., 2025; Guo et al., 2026; Han et al., 2024), fully reshaping the model for the task. Frozen or partially frozen transfer instead fixes pretrained embeddings or encoders and trains only task-specific modules: PeterRec (Yuan et al., 2020) inserts lightweight task patches for parameter-efficient transfer, GPSD (Wang et al., 2025) freezes transferred embeddings to alleviate the one-epoch phenomenon, and SORT (Wang et al., 2026b) adopts this in industrial deployment.

Appendix B Empirical Motivation

B.1. Behavioral Distribution Drift in Production Streams

Fig. 4 provides auxiliary evidence for the continual drift (Sec. 1). On the industrial stream, we compute the Jensen–Shannon divergence (JSD) between the item interaction distribution of each day and that of a fixed reference day, and track it across the full horizon. The divergence keeps rising rather than saturating, confirming that both the item pool and the audience turn over continually, so no fixed snapshot of the behavioral distribution stays representative. This is precisely the regime our streaming study targets (Sec. 4.3): a once-pretrained frozen representation grows stale and buffer replay degrades over time—the stale patterns it memorizes never recur—whereas continually refreshing the encoder tracks the drift. The measurement thus grounds the paper’s premise that, unlike in language or vision, pretraining for recommendation must be refreshed rather than handed off once.

B.2. Adjacency Does Not Imply Dependency

A motivating example. Treating adjacency as dependency injects noise into pretraining because real user behavior sequences often concatenate several separate interest sessions rather than following a single continuous intent. Fig. 1(a) illustrates this with a sequence that moves from shopping for “bicycle gear” to shopping for “pants”; we compute the pairwise cosine similarity between the item semantic embeddings of its interacted items and visualize it as the heatmap. Within each session the similarity is high (the dark diagonal blocks), because co-consumed items share strong semantic structure; across the session boundary it collapses, because the two intents are essentially unrelated. Next-token prediction nonetheless supervises the model on every adjacent pair, so the “bicycle gear”  “pants” transition straddling the boundary is compressed into the embedding geometry as if it were a genuine dependency.

BMTP resolves the noise. Fig. 1(a) already depicts a relatively idealized case; in our business scenarios, one interest may persist across days and many logins—a user re-searching one category over successive visits—so a natural cut severs it, while a single session id splices unrelated intents, as when a snack browser is shown a dress and chases it. Mechanical segmentation therefore injects noise in both directions, and clean sessions cannot reliably be carved at all. BMTP avoids segmentation altogether. It denoises through supervision, keeping a transition only when the pair is close on either collaborative or semantic axis. Testing relations rather than positions, it flexibly locates valid links anywhere in the sequence, keeping genuine dependencies even across an apparent boundary and dropping pairs like “bicycle gear”  “pants”.

B.3. Conflict Between Pretraining and Task Objectives

Full gradient-conflict evidence. Fig. 5 extends the gradient-conflict observation in Sec. 3.2 (illustrated in Fig. 1(b)) to more Amazon benchmarks, all on the same ManCAR backbone as our main experiments. During joint pretrain–task optimization, we compute the cosine similarity between and for the same parameter group . Across both embedding and Transformer parameters, the cosine remains weak and sometimes becomes negative, showing that the two objectives are far from consistently cooperative under shared parameters. This supports the main-text conclusion that knowledge and geometry should be owned by separate parameter sets: under shared parameters, writing task gradients pollutes the pretrained knowledge, which is exactly what KGD’s knowledge–geometry decoupling avoids.

Objective conflict under shared training. Fig. 6 illustrates the conflict between pretraining and task training under the same setup as Fig. 5. Before the dashed line, only the pretraining loss is back-propagated, while the task loss is monitored. As pretraining converges, the task loss does not consistently improve and even increases on several datasets. After task gradients are enabled, the task loss drops, but the pretraining loss rises again. This bidirectional interference suggests that the two objectives impose different geometric requirements on the shared representation space, motivating KGD to split them: the encoder owns knowledge while the task learner reads it through a read-only attention and carves its own geometry via ACR, instead of frozen transfer that only re-projects the pretrained space and never carves its own geometry.

Geometric discrepancy between pretraining and task training. Fig. 7 visualizes the item embedding spaces learned by denoised pretraining and by task training from scratch in our public benchmark setting. Each point denotes an item, and colors indicate ground-truth category labels. The two embedding spaces exhibit clearly different geometric structures. The denoised pretrained embedding forms more coherent category-aware neighborhoods, suggesting that it captures broad behavior-distribution structure across the item catalog. In contrast, the task embedding presents a substantially different geometry, where category-level neighborhoods are less globally organized and the space is more strongly shaped by task-specific optimization. This qualitative discrepancy supports our central argument that pretraining and task training do not naturally induce a consistent geometry space.

Appendix C Implementation Details

C.1. Data Statistics and Preprocessing

Public benchmark. For Amazon, we use the 5-core setting and evaluate on 8 benchmarks. Tab. 4 reports detailed statistics. We adopt the official leave-one-out strategy for dataset splitting, and use item IDs as the only model input feature.

We have also surveyed public datasets closer to our industrial setting, but by their design they do not fit what KGD studies. First, some are collected by user tracking over highly active users who return to the app almost daily (Gao et al., 2022), so they cannot reflect the large-scale user-ID distribution drift of real traffic and are unsuitable for validating streaming refresh. Second, others miss the inputs KGD requires: most features are encrypted (Liu et al., 2023a; Yuan et al., 2022), precluding semantic BMTP; the interaction-sequence features are insufficient (e.g. Criteo¹¹1https://www.kaggle.com/c/criteo-display-ad-challenge/data); or reliable timestamps are unavailable (Li et al., 2026; Wu et al., 2025). The value of KGD is most fully realized in real industrial scenarios. The eight benchmarks used here are instead widely recognized, classic public datasets, and our primary purpose in adopting them is to first obtain a reproducible result that verifies whether KGD effectively answers what to learn and how to transfer. Only with such a reproducible verification in place are we justified in advancing to the industrial evaluation and the subsequent online A/B test.

媒体内容 · 前往原文查看

Dataset	Users	Items	Interactions	Density
Software (Software)	146,396	17,591	1,130,444	0.044%
Video Games (Games)	94,762	25,612	719,824	0.030%
Office Products (Office)	223,308	77,551	1,577,570	0.009%
CDs and Vinyl (CDs)	123,876	89,370	142,888	0.001%
Arts Crafts and Sewing (Arts)	197,286	89,958	1,589,151	0.009%
Cell Phones and Accessories (Phones)	380,999	111,480	2,371,786	0.006%
Toys and Games (Toys)	432,264	162,035	3,429,622	0.005%
Beauty and Personal Care (Beauty)	729,576	207,649	5,894,865	0.004%

Industrial data. For industrial deployment, we conduct experiments on a large-scale industrial recommendation dataset collected from 28 consecutive days of real-world traffic logs after down-sampling for training feasibility. Tab. 5 reports the statistics of the industrial data. The dataset contains approximately 13 billion samples and is organized at the session level with negative sampling, where each session consists of a sequence of items interacted with by a user. The click rate is approximately 7.5%, while the order rate is approximately 0.41%, indicating a highly sparse conversion signal. At the corpus level, the user vocabulary size is at a ten-million scale, and the item vocabulary size is at nearly a billion scale, which cannot be fully disclosed due to the business requirement. We adopt the same data curation strategy as in (Fu et al., 2024), which was shown to improve generalization to previously unseen items. Since the dataset is constructed via extreme down-sampling and negative sampling, it cannot reflect the actual statistics of the complete production traffic. For the long-term decay study, we further extend training continuously over an additional 62 days of traffic, where the total 90-day dataset contains around 42B samples.

媒体内容 · 前往原文查看

Dataset	#Samples	Click Pos. Rate	Order Pos. Rate	#Items	#Users	Time Span
Industrial	13B	0.075	0.0041	billion scale	10-million scale	28 days

C.2. Evaluation Metrics

Public benchmark metrics. For Amazon, we report Normalized Discounted Cumulative Gain (NDCG) and Recall. NDCG evaluates whether relevant items are ranked at higher positions, reflecting the model’s ranking ability, while Recall measures whether the target item is retrieved within the top- candidates, reflecting retrieval effectiveness. Therefore we choose both metrics to provide a comprehensive evaluation of a recommender.

Industrial deployment metrics. For offline evaluation on the industrial stream, we adopt AUC to measure point-wise ranking quality and Group-AUC (GAUC), averaged over users, to measure group-wise ranking quality; both are reported per task, i.e., for click and order prediction. For the online A/B test (Sec. 5), we report business-impact metrics—GMV per user (GMV/User) and advertising revenue per user—together with CTR and CVR, and we additionally track the per-query irrelevant-item rate via human evaluation as a safeguard against quality regressions.

C.3. Backbone Model

We instantiate KGD on two strong backbones, one per evaluation setting. For public benchmarks, we use ManCAR (Yang et al., 2026) as the sequential recommender backbone, a strong publicly evaluable model that keeps the public experiments reproducible while avoiding a weak-backbone confound. For industrial deployment, we use OneRank (Tang et al., 2026) as the production ranking backbone, which is already used for large-scale multi-task ranking and is our strongest prior online baseline. KGD is applied as an ownership and training framework on top of these backbones and is not tied to either architecture.

C.4. KGD Implementation

BMTP & Encoder. The encoder is obtained by pretraining the backbone’s encoder module with the BMTP objective. For collaborative denoising, item similarity is computed from item graph embeddings learned by LightGCN (He et al., 2020), and the collaborative threshold is set to . For semantic denoising, item text embeddings are generated by Qwen3-Embedding²²2https://huggingface.co/collections/Qwen/qwen3-embedding, and the semantic threshold is set to . The text input is a simple prompt to concatenate the metadata of the item: "Item title: title text, Item category: category text, Item shop: shop title text".

Task Learner. The task learner adapts the read-only encoder representations to downstream tasks, writing its own geometry end-to-end. At the embedding level, it carves this geometry into the item space through the Anchored Calibration Residual (ACR), where the low-rank dimension is set to one quarter of the embedding size. To keep the newly carved geometry from overwriting pretrained knowledge, we encourage by adding an orthogonality regularization term to the task objective:

(3)

where

(4)

Here, denotes the original discriminative task loss, is the set of items involved in the current mini-batch.

Instantiation of the Read-Only Attention. The read-only attention in KGD is instantiated differently for the public ManCAR backbone and the industrial OneRank ranking backbone, but both follow the same three design dimensions introduced in Sec. 3: reader token choice, projector structure, and reading depth. Let denote the hidden states produced by the encoder, and let denote stop-gradient. During task training, all cross-attention keys and values from the encoder are detached, so task gradients update only the task-learner parameters, keeping knowledge transfer strictly read-only.

Public benchmark–ManCAR instantiation. For the public sequential recommender setting, the reader token is initialized from the encoder’s final user representation. Concretely, we take the last position hidden state from the final encoder layer,

and use its adaptation as a reader token,

where denotes the task-learner-owned reader token adapter module. This choice matches the downstream task: the final user state summarizes the historical sequence and provides the starting point for latent reasoning.

The public task learner adopts a reasoning-reader projector structure. It first refines the reader tokens by self-attention (SA),

then reads the final-layer encoder states through cross-attention (CA),

The resulting reader states then cross-attend to the ManCAR context sequence , which contains the graph-neighbor context of recent items:

followed by a feed-forward block (FFN):

Industrial deployment instantiation. For the industrial multi-task ranking setting, the reader tokens are the task-side non-sequential tokens used by the production ranking backbone, including candidate-item tokens and user-feature tokens:

where denotes candidate-side feature embeddings, denotes non-sequential user-feature embeddings, and optionally denotes task-specific tokens for click/order prediction. This choice matches the ranking setting: each candidate and task-side feature token queries the sequential behavioral states relevant to its own prediction.

The industrial task learner instantiates the read-only attention as a Perceiver-style reader. At each task-learner layer , the reader tokens are first updated by self-attention,

then cross-attend to the final-layer encoder states:

and are finally refined by a feed-forward block:

Summary. The two instantiations differ in the reader-token choice, yet both follow KGD’s parameter-decoupling paradigm: ACR writes a task-owned orthogonal residual onto the pretrained embedding, while a read-only cross-attention exposes the encoder’s contextualized behavior knowledge to the task learner via stop-gradient keys and values. Task gradients thus never reach the encoder, which keeps refreshing the knowledge while the task learner reads it and writes its own geometry.

C.5. Hyperparameter Configuration

Public benchmark setting. For the public benchmark setting (ManCAR), we use a 2-layer standard Transformer encoder with a causal attention mask. The hidden size and item embedding dimension are both 256, with 2 attention heads and a maximum input sequence length of 50. During task-specific training, we use Adam with learning rate , batch size 1024, and early stopping patience 3.

Industrial evaluation setting. For the industrial setting (OneRank), we use a 2-layer standard Transformer-based ranking backbone with hidden size 256 and 4 attention heads. The sparse feature embedding dimension is 32 per feature, and the maximum sequence length is 200. We use over 500 well-engineered features to deploy OneRank.

C.6. Compute Resources

The public benchmark experiments are conducted on a machine with NVIDIA A800 GPU with 80GB memory. The hardware environment of the industrial deployment cannot be fully disclosed due to business requirements.

Appendix D More Discussions

媒体内容 · 前往原文查看

Dataset	Method	N@5	N@10	N@20	N@50	R@5	R@10	R@20	R@50
Arts	NTP	0.0131	0.0179	0.0229	0.0302	0.0247	0.0398	0.0596	0.0963
	MTP	0.0112	0.0154	0.0202	0.0269	0.0193	0.0324	0.0514	0.0855
	BMTP	0.0149	0.0203	0.0258	0.0334	0.0268	0.0438	0.0656	0.1041
Beauty	NTP	0.0108	0.0144	0.0181	0.0237	0.0195	0.0307	0.0454	0.0737
	MTP	0.0089	0.0119	0.0152	0.0203	0.0145	0.0239	0.0371	0.0631
	BMTP	0.0125	0.0163	0.0202	0.0259	0.0217	0.0337	0.0492	0.0776
CDs	NTP	0.0275	0.0365	0.0446	0.0549	0.0518	0.0796	0.1117	0.1638
	MTP	0.0215	0.0296	0.0372	0.0474	0.0400	0.0650	0.0953	0.1469
	BMTP	0.0326	0.0431	0.0524	0.0638	0.0592	0.0916	0.1286	0.1863
Phones	NTP	0.0139	0.0182	0.0226	0.0290	0.0240	0.0374	0.0547	0.0871
	MTP	0.0111	0.0143	0.0179	0.0236	0.0170	0.0268	0.0411	0.0701
	BMTP	0.0141	0.0190	0.0236	0.0302	0.0253	0.0404	0.0588	0.0920
Office	NTP	0.0139	0.0177	0.0215	0.0268	0.0248	0.0368	0.0518	0.0786
	MTP	0.0118	0.0151	0.0185	0.0234	0.0195	0.0298	0.0431	0.0681
	BMTP	0.0168	0.0212	0.0253	0.0311	0.0275	0.0411	0.0572	0.0867
Software	NTP	0.0625	0.0826	0.1035	0.1300	0.1061	0.1686	0.2512	0.3851
	MTP	0.0629	0.0820	0.1024	0.1284	0.1025	0.1618	0.2428	0.3742
	BMTP	0.0588	0.0762	0.0946	0.1201	0.0917	0.1456	0.2189	0.3480
Toys	NTP	0.0146	0.0188	0.0227	0.0279	0.0264	0.0395	0.0549	0.0810
	MTP	0.0110	0.0147	0.0182	0.0232	0.0196	0.0311	0.0451	0.0705
	BMTP	0.0174	0.0223	0.0267	0.0325	0.0301	0.0453	0.0627	0.0919
Games	NTP	0.0300	0.0412	0.0526	0.0683	0.0562	0.0910	0.1363	0.2160
	MTP	0.0275	0.0375	0.0480	0.0626	0.0477	0.0789	0.1206	0.1944
	BMTP	0.0333	0.0450	0.0566	0.0725	0.0605	0.0967	0.1431	0.2231

媒体内容 · 前往原文查看

Method	Arts	Beauty	CDs	Software	Games
From Scratch	0.2194	0.2732	0.2040	0.1862	0.4720
Pretrained	0.5074	0.3192	0.4022	0.3941	0.7274

More evidence on the effectiveness of BMTP. Before downstream adaptation, the encoder should already encode catalog-level behavioral structure rather than merely serve as an initialization for the task model. We therefore probe the pretrained item space from two views. First, direct next-item prediction evaluates whether the encoder captures the behavioral transition structure. As shown in Tab. 6, BMTP achieves the best NDCG and Recall on seven of eight Amazon benchmarks, indicating that filtering future-item supervision by collaborative and semantic consistency improves the learned behavior-distribution structure. Second, category purity measures whether the learned item space aligns with catalog-level semantic organization. In Tab. 7, pretrained embeddings exhibit substantially higher purity than embeddings learned from scratch across all evaluated benchmarks, showing that the encoder recovers meaningful category-level structure in the item space.

The Software benchmark is the only direct-prediction exception: BMTP underperforms NTP/MTP before downstream adaptation. This is consistent with Software’s small catalog and high interaction density, where adjacent transitions are already comparatively reliable and denoising may remove useful local signal. Importantly, this exception does not undermine the role of BMTP in the full KGD pipeline: after task adaptation, KGD with BMTP still achieves the best downstream performance on Software. Thus, the pretraining probes support the intended role of the encoder: BMTP improves catalog-level behavioral structure in most sparse recommendation regimes, while the read-only task learner determines how that structure is converted into task-specific retrieval geometry.

Parameter sensitivity analysis. We further study the sensitivity of the two BMTP denoising thresholds in the encoder, i.e., the collaborative threshold and the semantic threshold . Fig. 8 and Fig. 9 report their effects from two perspectives: the direct performance of the pretrained module and the performance after task training.

Overall, the proposed denoised pretraining objective is sensitive to both thresholds, but exhibits clear and stable optimal regions across most datasets. For the collaborative threshold , the best performance is typically achieved within . This suggests that the collaborative signal induced by the item graph should be neither too weak nor too restrictive: a small threshold retains noisy transitions, while an overly large threshold removes useful co-occurrence structure. For the semantic threshold , the optimal range is generally higher, around . This indicates that in user behavior sequences, items with highly similar content tend to carry strong behavioral relevance, which aligns with the intuitive assumption that users often exhibit consistent interests toward content-similar items. Modeling such high-confidence semantic associations during pretraining therefore provides useful supervision for both direct retrieval and downstream adaptation.

Furthermore, the pretraining-stage and task-training-stage curves exhibit a consistent trend. This indicates that the direct performance of the pretrained module can serve as an effective proxy for downstream performance. Therefore, the optimal threshold range can be identified by evaluating the pretrained model alone, which substantially simplifies the hyperparameter optimization process.

Setting	click AUC	click GAUC	order AUC	order GAUC
\rowcolorgray!15     Scaling in task-specific parameters
tasks share parameters	0.7867	0.7826	0.9015	0.8477
tasks independent parameters	0.7883	0.7839	0.9033	0.8479
\rowcolorgray!15     Scaling in each side
2-layer encoder + 2-layer task learner	0.7867	0.7826	0.9015	0.8477
4-layer encoder + 2-layer task learner	0.7895	0.7853	0.9048	0.8518
2-layer encoder + 4-layer task learner	0.7891	0.7848	0.9053	0.8525
4-layer encoder + 4-layer task learner	0.7907	0.7859	0.9061	0.8529

Strategy	Pretrain	Arts		Beauty		CDs		Phones		Office		Software		Toys		Games
		N@20	R@20	N@20	R@20	N@20	R@20	N@20	R@20	N@20	R@20	N@20	R@20	N@20	R@20	N@20	R@20
Scratch	-	0.0251	0.0548	0.0208	0.0417	0.0544	0.1180	0.0204	0.0443	0.0245	0.0467	0.1034	0.2248	0.0278	0.0568	0.0556	0.1222
TE&FT	NTP	0.0251	0.0559	0.0190	0.0436	0.0560	0.1173	0.0239	0.0539	0.0243	0.0476	0.1056	0.2302	0.0270	0.0569	0.0537	0.1253
	MTP	0.0247	0.0570	0.0193	0.0421	0.0559	0.1160	0.0213	0.0463	0.0238	0.0477	0.1068	0.2321	0.0274	0.0569	0.0539	0.1235
	BMTP	0.0241	0.0585	0.0190	0.0441	0.0529	0.1210	0.0215	0.0519	0.0228	0.0517	0.1071	0.2309	0.0266	0.0620	0.0537	0.1252
\rowcolorgainrowcolor	BMTP vs. NTP gain	-4.00%	4.70%	0.00%	1.10%	-5.50%	3.20%	-10.00%	-3.70%	-6.20%	8.60%	1.40%	0.30%	-1.50%	9.00%	0.00%	-0.10%
TA&FT	NTP (Song et al., 2025)	0.0245	0.0555	0.0183	0.0418	0.0544	0.1185	0.0219	0.0505	0.0241	0.0488	0.1056	0.2275	0.0284	0.0576	0.0541	0.1258
	MTP (Chen et al., 2025; Guo et al., 2026; Lin et al., 2025)	0.0257	0.0586	0.0185	0.0415	0.0573	0.1153	0.0204	0.0429	0.0238	0.0477	0.1054	0.2285	0.0264	0.0569	0.0542	0.1232
	BMTP	0.0254	0.0548	0.0209	0.0433	0.0542	0.1251	0.0214	0.0463	0.0236	0.0476	0.1070	0.2323	0.0285	0.0575	0.0537	0.1262
\rowcolorgainrowcolor	BMTP vs. NTP gain	3.70%	-1.30%	14.20%	3.60%	-0.40%	5.60%	-2.30%	-8.30%	-2.10%	-2.50%	1.30%	2.10%	0.40%	-0.20%	-0.70%	0.30%
TE&FE	NTP (Wang et al., 2025, 2026b)	0.0281	0.0645	0.0212	0.0477	0.0538	0.1238	0.0260	0.0573	0.0257	0.0563	0.1070	0.2429	0.0274	0.0629	0.0604	0.1370
	MTP	0.0271	0.0615	0.0203	0.0443	0.0524	0.1190	0.0225	0.0481	0.0240	0.0509	0.1058	0.2409	0.0268	0.0592	0.0589	0.1331
	BMTP	0.0301	0.0692	0.0220	0.0499	0.0581	0.1342	0.0266	0.0609	0.0278	0.0602	0.1092	0.2436	0.0298	0.0673	0.0629	0.1434
\rowcolorgainrowcolor	BMTP vs. NTP gain	7.10%	7.30%	3.80%	4.60%	8.00%	8.40%	2.30%	6.30%	8.20%	6.90%	2.10%	0.30%	8.80%	7.00%	4.10%	4.70%
TA&FE	NTP (Wang et al., 2025)	0.0271	0.0636	0.0209	0.0469	0.0546	0.1260	0.0249	0.0557	0.0255	0.0566	0.1071	0.2435	0.0274	0.0631	0.0601	0.1377
	MTP	0.0261	0.0591	0.0193	0.0427	0.0522	0.1194	0.0192	0.0434	0.0242	0.0515	0.1059	0.2399	0.0265	0.0590	0.0585	0.1318
	BMTP	0.0290	0.0672	0.0221	0.0502	0.0580	0.1341	0.0261	0.0614	0.0275	0.0604	0.1094	0.2429	0.0295	0.0667	0.0618	0.1405
\rowcolorgainrowcolor	BMTP vs. NTP gain	7.00%	5.70%	5.70%	7.00%	6.20%	6.40%	4.80%	10.20%	7.80%	6.70%	2.10%	-0.20%	7.70%	5.70%	2.80%	2.00%
TA&FD	NTP	0.0166	0.0408	0.0119	0.0289	0.0432	0.1059	0.0160	0.0391	0.0149	0.0361	0.0654	0.1616	0.0169	0.0407	0.0396	0.0982
	MTP	0.0207	0.0510	0.0137	0.0327	0.0424	0.1038	0.0129	0.0303	0.0184	0.0423	0.0642	0.1639	0.0199	0.0474	0.0425	0.1040
	BMTP	0.0197	0.0497	0.0134	0.0319	0.0424	0.1036	0.0173	0.0423	0.0184	0.0440	0.0758	0.1894	0.0193	0.0460	0.0472	0.1139
\rowcolorgainrowcolor	BMTP vs. NTP gain	18.70%	21.80%	12.60%	10.40%	-1.90%	-2.20%	8.10%	8.20%	23.50%	21.90%	15.90%	17.20%	14.20%	13.00%	19.20%	16.00%
TA&FA	NTP (Yuan et al., 2020)	0.0201	0.0481	0.0145	0.0345	0.0319	0.0760	0.0197	0.0455	0.0200	0.0456	0.0948	0.2180	0.0155	0.0359	0.0408	0.0968
	MTP	0.0177	0.0428	0.0127	0.0303	0.0299	0.0724	0.0139	0.0321	0.0176	0.0403	0.0931	0.2195	0.0138	0.0322	0.0386	0.0895
	BMTP	0.0232	0.0546	0.0171	0.0397	0.0414	0.0964	0.0228	0.0515	0.0226	0.0494	0.0954	0.2186	0.0212	0.0483	0.0509	0.1168
\rowcolorgainrowcolor	BMTP vs. NTP gain	15.40%	13.50%	17.90%	15.10%	29.80%	26.80%	15.70%	13.20%	13.00%	8.30%	0.60%	0.30%	36.80%	34.50%	24.80%	20.70%
KGD	NTP	0.0278	0.0640	0.0212	0.0482	0.0558	0.1302	0.0257	0.0563	0.0262	0.0576	0.1099	0.2411	0.0285	0.0637	0.0601	0.1396
	MTP	0.0274	0.0625	0.0215	0.0474	0.0530	0.1224	0.0222	0.0476	0.0255	0.0536	0.1097	0.2394	0.0261	0.0584	0.0595	0.1338
	BMTP	0.0302	0.0701	0.0238	0.0528	0.0588	0.1359	0.0272	0.0618	0.0292	0.0631	0.1126	0.2484	0.0305	0.0675	0.0631	0.1452
\rowcolorgainrowcolor	BMTP vs. NTP gain	8.60%	9.50%	12.30%	9.50%	5.40%	4.40%	5.80%	9.80%	11.50%	9.50%	2.50%	3.00%	7.00%	6.00%	5.00%	4.00%
\rowcolorgreen!8     gain vs. best published baseline		7.50%	8.70%	12.30%	10.70%	2.60%	7.90%	4.60%	7.90%	13.60%	11.50%	5.10%	2.00%	7.40%	7.00%	4.50%	5.40%
\rowcolorgreen!8     gain vs. best baseline w/ BMTP		0.30%	1.30%	8.20%	5.20%	1.20%	1.30%	2.30%	0.70%	5.00%	4.50%	2.90%	2.00%	2.30%	0.30%	0.30%	1.30%

Scaling. Tab. D reports the full industrial scaling results, covering task-shared vs. task-independent parameterization and per-component capacity scaling of the encoder and task learner. Both bring consistent gains, with the best performance when both sides are enlarged; see the analysis in Sec. 4.4.

Appendix E Extended Experimental Results

Tab. D reports results of 8 public benchmarks under the NDCG@20 and Recall@20, complementing the @50 metrics in the main table. The conclusions in Tab. 4.1 still hold: KGD consistently outperforms the published baselines, showing that the advantage of decoupled ownership of knowledge vs. geometry is not specific to a larger cutoff, and replacing standard pretraining objectives with BMTP brings further gains. Fig. 10 presents the unsmoothed 90-day trajectories corresponding to Fig. 3; the same trends hold without smoothing.

Appendix F Ethics and Privacy Statement

The industrial data used in this study were collected and processed in accordance with the platform’s internal data-governance policies. All user identifiers were anonymized, and only aggregated experimental results are reported. The study involved no direct researcher interaction with users. The online experiment modified ranking results as part of routine platform experimentation and was conducted under the platform’s internal experimentation, privacy, and data-governance policies. No personally identifiable information is released.