How Do Decoder-Only LLMs Perceive Users? Rethinking Attention Masking for User Representation Learning

🍞 Hook: Imagine you’re trying to guess what your friend will do next at a theme park. If you only remember the last ride they took, you’ll miss patterns like “she always eats before a roller coaster.” You need the whole story, not just the latest bit.

🥬 The Concept (User Representation Learning): It’s a way for computers to build a compact profile (an embedding) of a person from many clues—purchases, app taps, searches—so the system can understand and help them better.

• How it works: (1) Collect a user’s many activities over time and across apps; (2) Turn each activity into numbers; (3) Mix them into one vector that captures habits and preferences; (4) Use that vector to make predictions (like which features they’ll use or what they’ll like next).
• Why it matters: Without a good user embedding, apps make poor guesses, like recommending bus tickets to someone who always takes the subway.

🍞 Anchor: Your music app remembering you love upbeat pop on weekdays and calm jazz on Sundays is user representation learning at work.

🍞 Hook: You know how a storyteller makes up a tale one sentence at a time, always using what was just said to decide what comes next?

🥬 The Concept (Decoder-Only LLMs): These are language models that generate or read sequences one token at a time, mainly using what came before.

• How it works: (1) Read the previous tokens; (2) Predict the next one; (3) Repeat; (4) Build understanding step by step.
• Why it matters: They’re great for interactive systems where new signals come in continuously (like real-time app activity).

🍞 Anchor: Chatbots that keep answering as you keep typing rely on decoder-only LLMs to understand and respond on the fly.

🍞 Hook: Think of using a flashlight in a dark room. You choose where to shine it and where not to.

🥬 The Concept (Attention Masking): It tells the model where it’s allowed to “look” in the sequence and where it must “look away.”

• How it works: (1) Mark which tokens each position can see; (2) Hide some tokens (mask them); (3) Let attention focus only on visible parts; (4) Compute an output using those focused views.
• Why it matters: Without the right mask, the model might cheat during training or miss vital context during understanding.

🍞 Anchor: In a quiz, covering the answer with your hand keeps you honest; that’s attention masking preventing peeking at the future.

🍞 Hook: Reading a mystery book only page by page without peeking ahead keeps things fair, but you might miss big-picture hints.

🥬 The Concept (Causal Attention): The model can only use tokens to the left (past) and can’t see the future.

• How it works: (1) For each word, allow attention only to earlier words; (2) Block all future words; (3) Predict or embed using past-only info; (4) Repeat for every position.
• Why it matters: It matches how decoder-only models were pretrained and keeps generation stable—no future peeking.

🍞 Anchor: When composing a message, you can’t read the reply yet; you write using only what has been said.

The world before: People used encoder models (which see both past and future) to learn rich user embeddings, but those models need the full sequence upfront. That’s hard in real apps where data arrives little by little. Decoder-only LLMs can work in streams, but they were trained with causal attention, which might limit how well they combine far-away clues into one powerful user profile.

The problem: No one had carefully compared how different masks (causal, hybrid, and bidirectional) affect user embeddings in one fair setup. Also, suddenly switching a decoder-only model from “past-only” to “see everything” can make training bumpy and the results worse.

Failed attempts: People tried (1) staying causal—safe but misses future context; (2) going fully bidirectional right away—rich context but unstable training; (3) hybrid tricks—better but complicated and not consistently strong; (4) simple schedulers that slowly open the mask—helps, but still not smooth enough.

The gap: We needed a stable, data-driven way to transition from causal to bidirectional so decoder-only LLMs could learn truly strong user embeddings without losing their pretraining benefits.

Real stakes: Better embeddings mean apps can predict churn, tailor recommendations, plan marketing fairly, and avoid irrelevant nudges. That makes experiences smoother for users and decisions smarter for businesses.

🍞 Hook: Picture teaching a new puppy to cross a busy street. You don’t drop the leash and say “run anywhere.” You loosen it bit by bit, paying attention to where the puppy feels safest.

🥬 The Concept (Bidirectional Attention): Letting the model see both past and future tokens to build a fuller understanding.

• How it works: (1) For each position, allow attention to all tokens; (2) Weigh what matters most from both sides; (3) Form a richer, context-aware representation.
• Why it matters: Without looking both ways, the model misses future clues—like a later search or purchase—that make a user’s pattern clearer.

🍞 Anchor: When guessing what a story is really about, you look at both earlier and later chapters, not just what came before one page.

🍞 Hook: Think of mixing hot and cold water to get the perfect bath temperature—you don’t turn the faucets from cold to hot instantly.

🥬 The Concept (Hybrid Attention): A blend—often bidirectional in the user-history block but causal after, so some tokens see more, others stay sequential.

• How it works: (1) Pick a segment (like history) to see both ways; (2) Keep future tokens causal; (3) Let a helper (like an MLP or a global token) guide attention; (4) Train embeddings with this mixed view.
• Why it matters: It tries to balance rich understanding and compatibility with generation, but tuning it can be tricky.

🍞 Anchor: It’s like letting a class discuss freely within their table group (history) while still following turn-taking rules for the rest of the room (future).

🍞 Hook: Imagine dimmer switches on ceiling lights. Instead of flipping everything to full brightness, you brighten the bulbs that help you see best first.

🥬 The Concept (Gradient-Guided Soft Masking, GG-SM): A warm-up that opens future attention softly, guided by gradient signals about which tokens matter most, before smoothly moving to full bidirectional.

• How it works: (1) Start with causal attention; (2) Measure which future tokens push learning the most (using gradient strength); (3) Give those tokens partial visibility first (soft masks); (4) After warm-up, linearly open to full bidirectionality.
• Why it matters: Jumping straight to full future visibility can confuse a model trained to look only left. GG-SM teaches it “where to look” and “how much,” making training steady and embeddings stronger.

🍞 Anchor: When learning a new song, you first focus on the bars where you make the most mistakes. Then, once those are comfy, you play the whole song smoothly.

Aha! moment in one sentence: Don’t just decide which tokens the model can see—decide how to open that visibility over time using the model’s own gradients as a guide.

Multiple analogies:

• Traffic cop: Causal is a one-way street; bidirectional is a two-way boulevard; GG-SM is timing the green lights so traffic starts flowing safely and faster.
• Puzzle building: Instead of dumping all puzzle pieces at once, GG-SM lets in the most helpful future pieces first, so the picture forms without overwhelm.
• Flashlight dimmer: You brighten the spots that matter (high-gradient tokens) before turning on all the lights.

Before vs. after:

• Before: Causal is safe but incomplete; immediate bidirectional can be shaky; hybrids are fussy; schedulers help but aren’t smart about which tokens to open first.
• After: GG-SM gives a smart, stable path to full context, consistently yielding better user embeddings across many tasks.

Why it works (intuition): Gradients act like a pressure map showing which hidden tokens would most reduce the training error. By letting those spots become visible first, the model learns to rely on the most informative future clues, preventing chaos from a sudden context flood.

Building blocks:

• Datasets that pair user histories with future behaviors or QA alignments.
• A dual-tower encoder (same backbone) to embed users and answers.
• A contrastive objective that pulls matched pairs together and pushes others apart.
• Mask recipes: causal, hybrid, bidirectional.
• GG-SM: gradient-guided warm-up, then a linear schedule to full bidirectionality.

At a high level: Multimodal user history (and optional query) → modality encoders + adapters → decoder-only LLM tower to get a user embedding at the <USER> token; answer text → same LLM tower to get an answer embedding; apply attention masking (causal/hybrid/bidirectional) with GG-SM warm-up; train with contrastive loss; output: robust user embeddings.

🍞 Hook: Think of packing a suitcase. You group clothes (shirts, pants, socks) before zipping the bag.

🥬 The Concept (Modality-Specific Encoding): Different data types (bills, app taps, searches, tables) are encoded by small specialists, then mapped into the LLM’s space.

• How it works: (1) Each modality goes through its own encoder; (2) Lightweight adapters align them to the LLM’s token space; (3) Concatenate in a standard template with tags; (4) Feed into the LLM.
• Why it matters: Without this, the LLM would see a jumble of unrelated formats and learn a messy embedding.

🍞 Anchor: It’s like labeling boxes “kitchen,” “books,” and “toys” before loading the moving truck so everything fits the new house.

Step-by-step recipe:

1. Standardized input template

• What happens: Wrap each modality with clear tags: <bill>...</bill>, <search>...</search>, etc., append optional user query, then a special <USER> token where we read out the embedding.
• Why: Tags teach the model structure; <USER> marks the “collect everything here” spot.
• Example: <bill>[grocery:$30, 2024-11-20]</bill><search>[‘movie tickets’]</search> ... <USER>

1. Dual-tower encoding with a shared LLM backbone

• What happens: The left tower reads user+query up to <USER> and outputs the user embedding; the right tower reads the answer text and outputs the answer embedding. Both towers share weights but process inputs separately.
• Why: Contrastive learning needs two views—user side and answer side—and the shared backbone keeps them compatible.
• Example: Left produces a 1024-d user vector; right produces a 1024-d answer vector.

1. Attention masking strategies

• What happens: Choose a mask: causal (left-only), hybrid (mixed), or bidirectional (see all). With GG-SM, start causal, softly open future tokens by gradient importance, then linearly reach full bidirectionality.
• Why: The mask controls how context is combined; GG-SM avoids a rough jump that can hurt learning.
• Example: Early in training, only a few high-gradient future tokens near <USER> get partial visibility; later, all tokens are visible.

🍞 Hook: Learning which questions to peek at on a test can make you overconfident unless you peek the right way.

🥬 The Concept (Contrastive Learning): Learn by pulling matching pairs close and pushing mismatched ones apart.

• How it works: (1) Compute similarity between the user embedding and its correct answer (positive); (2) Compare against other answers and other users (negatives); (3) Increase the positive score, decrease negatives; (4) Repeat across the batch.
• Why it matters: Without contrast, embeddings collapse or fail to capture what makes users uniquely similar or different.

🍞 Anchor: Sorting socks: you find the right pair (pull together) and keep different socks apart (push away).

1. InfoNCE loss with smart negatives

• What happens: Use InfoNCE to compute a softmax over similarities, including same-side negatives (user–user, answer–answer) and mask out false negatives with a margin.
• Why: Same-side negatives make embeddings sharper; masking false negatives prevents punishing true similarities.
• Example: If two users are extremely similar and pass the margin, they aren’t forced apart.

🍞 Hook: If you try to sprint from zero to top speed, you stumble; a warm-up helps.

🥬 The Concept (Scheduler vs. Gradient-Guided Soft Masking): A scheduler opens attention on a timer; GG-SM opens it where gradients say it helps most.

• How it works: (1) During warm-up, compute gradients to see which future tokens would reduce loss most; (2) Assign soft weights (not all-or-nothing); (3) Freeze those weights at warm-up end; (4) Interpolate to full open with a simple linear schedule.
• Why it matters: Time-based opening can expose noisy or unhelpful tokens too early; gradient-guided opening focuses learning where it counts.

🍞 Anchor: A coach watches which muscles are tight and designs stretches for those first, then expands to a full workout.

1. Training details that keep things stable

• What happens: Large batch contrastive training, AdamW optimizer, cosine decay, LoRA for efficient fine-tuning; identical settings across masks for a fair comparison.
• Why: Fair apples-to-apples tests show differences come from masking, not from extra tricks.
• Example: Same backbone, same steps, different masking recipes.

Concrete walk-through with data

• Input: A user’s 90-day history: paid utilities, rode public transit, searched “discount movies,” used a food app; optional query: “Any weekend movie deals?”; then <USER>.
• Encoding: Modality encoders produce embeddings, adapters align them, LLM processes sequence under the current mask.
• GG-SM warm-up: Early on, gradients show “discount movies” and “transit rides” near weekends matter most; those future tokens get soft visibility first.
• Contrastive step: Pair with an answer like “Yes—half-price shows after 6 pm near you.” The model pulls this pair closer than random answers.
• Over time: Visibility opens to all tokens; the user embedding becomes a compact, context-rich vector.

Secret sauce

• GG-SM uses the model’s own learning pressure (gradients) to decide where to look next. That makes the transition smoother than a blind schedule, stabilizes optimization, and yields better final bidirectional embeddings.

🍞 Hook: If two students both get 87%, it means more when the rest of the class got around 75%—context turns numbers into meaning.

🥬 The Concept (AUC – Area Under the ROC Curve): A score from 0.5 (coin flip) to 1.0 (perfect) showing how well a model separates positives from negatives across thresholds.

• How it works: (1) Rank examples by confidence; (2) Sweep a threshold; (3) Plot true-positive vs. false-positive rates; (4) Measure the area under that curve.
• Why it matters: It’s threshold-free and shows overall discriminative power, great for imbalanced tasks.

🍞 Anchor: If you can almost always place the “will click” users above the “won’t click” users in a list, your AUC is high.

The test: 9 real-world, binary user cognition tasks from Alipay in three domains: (1) User Prediction (concert click, login, MAU loss), (2) Behavior Preference (transit, consumption power, food, movie), (3) Marketing Sensitivity (achievement, physical). Models are trained once to make embeddings, then we do linear probing—a simple classifier on top.

🍞 Hook: Imagine you learn to summarize books, and we test your summaries by seeing how well a simple quiz uses them to pick the right answer.

🥬 The Concept (Linear Probing): Train a tiny classifier on frozen embeddings to see how much useful info they hold.

• How it works: (1) Freeze embeddings; (2) Fit a linear model per task; (3) Evaluate AUC; (4) Compare across models.
• Why it matters: It tests representation quality, not just model capacity to memorize.

🍞 Anchor: It’s like checking if good notes help you pass a quiz without letting you rewrite the whole textbook.

Competition: We compare against strong general-purpose embedding models (e.g., Llama-embed-nemotron, KaLM-Embedding, Qwen3-Embedding), classic user models (MSDP, One4all, CPC), and LLM-based user models (FOUND, InstructUE). We also test the same decoder backbone under masking variants: Causal, Hybrid (several flavors), Bidirectional, Bidirectional + Scheduler, and Bidirectional + GG-SM (ours).

Scoreboard with context:

• Our GG-SM reaches an average AUC of 0.7745 across all 9 tasks. Think of that as getting an A while many strong alternatives land more like B/B+.
• It consistently edges out bidirectional without warm-up and scheduler-only methods, showing that how you open future attention matters, not just that you open it.
• It also beats larger, general embedding models on these user tasks, highlighting that domain alignment and transition stability can trump raw parameter count.

Task-level color:

• Behavior Preference: Notable gains on food/movie interest and transit, where long-range context (e.g., weekend searches aligning with purchases) matters.
• Marketing Sensitivity: GG-SM is strong where latent intent is subtle; the gradient-guided opening seems to capture nuanced traits.
• User Prediction: Solid improvements over causal and hybrid masks; bidirectional context helps anticipate near-future actions.

Surprising findings:

• Hybrid masks provide only modest and inconsistent boosts—likely because extra parameters or control tokens add complexity without reliably improving alignment.
• Training stability is visibly better with GG-SM: loss curves are smoother and converge faster than with a plain scheduler.
• More parameters do not automatically win. On sparse, non-linguistic behavioral logs, carefully guided attention opening can beat much larger general models.

Takeaway: The transition path is a first-class design choice. GG-SM’s gradient-informed warm-up turns decoder-only LLMs into stronger bidirectional encoders for user understanding.

Limitations

• Domain scope: Experiments center on Alipay’s ecosystems and tasks; results may shift in domains with very different behavior patterns (e.g., education or health logs).
• Data synthesis reliance: The QA alignment set uses LLM-generated pairs and post-processing; quality depends on prompt design and the base generator.
• Full bidirectionality at inference: This maximizes embedding quality but is not directly generative; hybrid setups are better if you need generation and embeddings at the same time.
• Gradient noise: Early gradients can be noisy; while soft masks smooth things, extremely short sequences or tiny batches may reduce the benefit.
• Compute: Large-batch contrastive training and gradient-based masking add overhead; smaller setups may need lighter approximations.

Required resources

• A capable decoder-only backbone and modality encoders/adapters.
• Substantial GPU time (e.g., multi-GPU training with large batches) for stable contrastive learning.
• Access to diverse user histories with careful privacy and compliance controls.
• Engineering for a standardized input template and dual-tower retrieval pipeline.

When not to use

• Real-time text generation where strict causality must be preserved at inference.
• Ultra-short histories where bidirectional context adds little.
• Settings where privacy prevents assembling multi-modal histories into one view.

Open questions

• Can we estimate token importance without full gradients (e.g., using attention entropy or cheap saliency) to cut costs?
• How does GG-SM interact with very long contexts (128k+) and memory-efficient attention variants?
• Can we co-train for both generation and embeddings with a shared hybrid policy that adapts per task at inference?
• How does the approach generalize to other domains (e.g., healthcare trajectories) and languages with different tokenization patterns?
• Could curriculum learning pair with GG-SM to schedule which data, not just which tokens, to reveal when?

Three-sentence summary

• Decoder-only LLMs can produce excellent user embeddings, but attention masking—and especially how we transition from causal to bidirectional—shapes their ultimate quality.
• Gradient-Guided Soft Masking uses gradients to softly and smartly open future attention before a simple linear schedule completes the move to full bidirectionality.
• This stable transition delivers stronger, more transferable user representations across nine real-world tasks, outperforming larger general embedding models and prior user-modeling baselines.

Main achievement

• Turning the transition path itself into a learnable lever: GG-SM consistently stabilizes training and boosts bidirectional embedding quality without discarding decoder pretraining benefits.

Future directions

• Cheaper importance signals (beyond gradients), better hybrid policies for joint generation+embedding, scaling to ultra-long contexts, and replication across very different behavior domains.

Why remember this

• Not just what the model sees, but when and how it’s allowed to see it, can make the difference between average and best-in-class user understanding. GG-SM shows that gentle, guided visibility creates smarter, steadier learning for decoder-only LLMs used as user encoders.