Measuring Search-Agent Robustness to Poor Evidence, Semantic Planning for Multi-Stakeholder Recommendation, and More!
Vol.165 for Jul 13 - Jul 19, 2026
Stay Ahead of the Curve with the Latest Advancements and Discoveries in Information Retrieval.
This week’s newsletter highlights the following research:
Stress-Testing Search Agents Under Controlled Evidence Degradation, from Orange Research
Semantic Planning for Multi-Stakeholder Recommendation, from Jin et al.
Decoupling Model Depth from User History Length in Personalized Ranking, from Meta
Cost-Aware Tool Routing for Pointwise LLM Reranking, from Liu et al.
Protecting Sparse Gradients in Unified Billion-Scale Retrieval, from Pinterest
Semantic Retrieval for Multilingual Music Search, from Apple
Information Gain as a Prior for Test-Time Compute in Generative Recommendation, from Yang et al.
Hybrid Negative Mining and Legacy-Aware Distillation for Production E-commerce Retrieval, from Walmart
Aligning DocID Design with Business Objectives in Generative Retrieval, from Alibaba
Multiplex Item and User Representations for Multitask Product Ranking, from JD[.com]
[1] DeepStress: Stress-Testing Deep Search Agents
This paper from Orange Research introduces DeepStress, a stress-testing framework that swaps out a search agent’s retrieval module for a synthetic document generator. It lets the authors dial the quality of returned evidence up or down along three axes: whether a source looks credible, whether it actually answers the query, and whether its claims are true. Documents are generated on the fly for each agent query, with per-call labels sampled from scenario probabilities, and a human check confirms the intended trustworthiness and factuality labels mostly land. The authors ran twelve systems (GPT-5 nano and mini at three reasoning efforts, plus open-weight agents like Search-R1, AutoRefine, ASearcher, SpiqaDR, and DR-Tulu) across HotpotQA and BrowseComp-Plus at three quality levels per axis. They scored outcomes in five buckets rather than plain right/wrong, and added a Reliability-Aware Score that gives partial credit for abstaining when the evidence was never going to support an answer. The outcome shows distinct failure patterns. Open-weight RL-trained agents largely ignore credibility cues and keep answering confidently from whatever lands in context, GPT models either abstain or burn through their tool budget searching, and DR-Tulu and AutoRefine start dropping their answer tags entirely under degradation. Irrelevant documents turn out to be easier to reject than relevant-but-false ones. The authors also show that agents still answer plenty of HotpotQA questions correctly when every document is fabricated, which means parametric knowledge is doing the work and HotpotQA needs a no-search baseline when it is used to evaluate search agents.
📚 https://arxiv.org/abs/2607.13920
[2] From Raw IDs to Semantic Planning: How Recommender Systems Utilize Information at Scale
This position paper from Jin et al. surveys two decades of recommender system history in terms of how systems make information usable at industrial scale. Raw IDs won early because they solve two real problems, acting as stable keys that let logging, retrieval, and serving stay synchronized across a messy production stack, and letting models memorize item-specific behavior that semantic features can’t explain. The cost is cold-start and embedding tables that keep growing. Systems further added richer signals like reviews, images, context, and sequences, but identity stayed raw, so semantics sat next to the ID rather than inside it. Semantic IDs change that by folding part of the structure into the identifier itself, which the authors frame as three widening unifications: within a domain, then across domains and modalities, then across search and recommendation. They argue that all three still just do better user-item matching, and none of them handles the fact that industrial recommendation serves users, platforms, and providers at once with conflicting goals. Their proposal, semantic planning, inserts a layer that first decides what the next exposure should accomplish, then instantiates that target as an item, an offer, or a generated creative.
📚 https://arxiv.org/abs/2607.09540
[3] SlimPer: Make Personalization Model Slim and Smart
This paper from Meta introduces SlimPer and argues that recommendation ranking shouldn’t borrow its architecture from language models. LLMs need full token-level representations at every layer because they predict at every position, but a ranking model only outputs a handful of relevance scores per user-item pair, so carrying big intermediate tensors that grow with sequence length doesn’t add anything. Instead, SlimPer keeps a fixed-size knowledge base of roughly 64 slots initialized from item-side features, and each layer runs a Select-Match-Refine cycle over it. Attention pulls from the full set of raw user-side tokens, dot products against learned templates produce explicit relevance scores, and an MLP folds the result back into the knowledge base. As every layer reads the raw tokens directly rather than a compressed hidden state, depth stops scaling with history length, giving O(N) per-layer compute and O(L·K) memory. The design also extends request-only optimization across all user-side modalities, so user tokens get computed once per request and shared across candidates. The authors folded sparse, dense, and sequence features into one backbone rather than the usual late-fusion processing. On Instagram Reels and Feed they beat a mature HSTU plus Wukong baseline, with 8 to 25x fewer FLOPs and roughly 10 to 18 percent lower memory at matched sequence length.
📚 https://arxiv.org/abs/2607.12281
[4] Tool-Adaptive LLM Reranker
This paper from Liu et al. introduces TALRanker (Tool-Adaptive LLM Reranker), a pointwise LLM reranker that decides per query-document pair whether to call a search tool. The motivation is that closed-book reasoning rerankers hallucinate on out-of-distribution and long-tail queries, while calling search on every pair is too slow for systems that score hundreds of pairs per session. The authors model pointwise scoring as an agentic MDP. The model generates a reasoning trace and emits either a tool call, which suspends generation and appends retrieved evidence to the state, or a terminal answer token, at which point the relevance score is read from the yes/no probability split. During training, the warm-up stage applies binary cross-entropy at the final scoring token and a token-wise KL penalty against the reference model on all preceding tokens, which preserves the generative capacity that plain BCE fine-tuning destroys. The RL stage uses GRPO with a reward combining format compliance, a per-call latency penalty of λ·N_hops, and an accuracy term that assigns -1.0 to a wrong zero-hop answer but 0.0 to a wrong answer that invoked a tool. Built on Qwen3 and trained on MS MARCO, the 8B model beats ReasonRank-7B and Rank-R1-14B on BEIR and BRIGHT. Reported tool invocation rates for the 8B model are below 0.3% on every subset and 0.00% on DL19 and DL20, so retrieval is rarely triggered at the scales where the model performs best.
📚 https://arxiv.org/abs/2607.10555
[5] MESH: Scaling Up Retrieval with Heterogeneous Content Unification
This paper from Pinterest found that scaling up a recommendation retrieval model doesn’t help all content equally. When the authors grew model capacity on their billion-scale Related Pins system, popular evergreen content kept improving while fresh and long-tail items barely moved. They called this problem the ‘Scaling Bias of Heterogeneity’, caused by gradient dominance. In a flat architecture where all features get crossed together, high-frequency engagement signals dominate the sparse signals from new items, so extra compute mostly refines what the model already knows. They propose a fix called MESH (Modularized Effective Scaling for Heterogeneity) that splits the feature space into separate user, item, and context sub-towers that each enrich their own signals before anything crosses globally, then merges them through a gated bias correction step that lets context absorb engagement noise rather than contaminate user-item matching. The isolation gives sparse items a protected gradient path. Their online A/B tests show +5.5% fresh impressions, +2.33% long-tail shopping repins, and +0.46% on long sessions without cannibalizing overall traffic.
📚 https://arxiv.org/abs/2607.12392
[6] Multilingual Semantic Retrieval for Apple Music Search
This paper from Apple describes a semantic retrieval system deployed in Apple Music search. Their existing stack uses a token-based index with language-specific analyzers, which works for head queries but fails on the tail, where lexical overlap between query and intended item is weak or absent. The authors’ analysis of poor-quality sessions found most were recall failures rather than ranking errors, with misspellings, transliterations, cross-lingual mismatches, and functional queries as the main modes. Tail queries are 83% of unique queries and roughly a third of sessions. Their proposed model is a 305M-parameter Siamese bi-encoder fine-tuned from GTE-multilingual-base. It trains on query-item pairs from engagement logs, with a secondary query-query objective aligning failed queries to their successful reformulations. Catalog items are encoded from a linearized metadata template that includes bucketed popularity, release strings, lyric snippets, and prior engaged queries. For integration, ANN cosine scores are linearly blended with the existing text-match score, with the blend weight fit against spell-correction pairs, and a quantile map then aligns the blended distribution to the original text-match distribution so downstream rankers need no retraining. Offline Hit@10 improves 69% over the base backbone, and an 18-day worldwide A/B test reports +2.28% conversion rate, an 86% reduction in the no-result rate, positive lift in every storefront, and +7.93% on tail queries against +0.14% on head. CJK storefronts show the weakest gains.
📚 https://arxiv.org/abs/2607.10239
[7] Where Reasoning Matters: Rethinking Latent Reasoning in Semantic ID-based Generative Recommendation
This paper from Yang et al. asks when an item is represented by a short sequence of discrete codes (in semantic ID-based generative recommendation) generated autoregressively, and latent reasoning adds extra hidden-state computation before each token decision, should every position get the same number of refinement steps? The authors measure position-wise information gain offline from training data, treating each code prefix as a candidate set and computing the entropy it removes, and find early positions cut far more uncertainty than later ones, which the RQ-VAE hierarchy and the generation trie have already largely pinned down. Early positions also gain more token-level accuracy from extra refinement, so the expected utility of a step scales with both factors. Their framework, IBA (Information-Gain Budget Allocation), treats refinement steps as a fixed budget and spends it accordingly. A dual-axis refinement module stabilizes multi-step reasoning by aligning intermediate states to the codebook space at each step, a lightweight MLP predicts the gain of each candidate step for a given user, and an allocation objective picks the per-sequence schedule that maximizes predicted gain under a prior biased toward early positions. In their experiments, IBA lifts TIGER and LETTER across Recall and NDCG and edges out reasoning-enhanced baselines.
📚 https://arxiv.org/abs/2607.12425
[8] Scaling and Stabilizing Large-Scale Embedding-Based Retrieval
This paper from Walmart presents a report on upgrading a production embedding-based retrieval system. It targets two problems that plague dual-encoders at scale: the training-inference gap, where a model trained against a narrow batch of negatives must later discriminate across hundreds of millions of items, and the risk of swapping in a bigger backbone. On the signal side, cross-batch sampling all-gathers item embeddings across GPUs online, expanding the effective negative pool from the local batch to batch times device count without extra forward passes. Offline, they extend an ANCE-style mining loop with a BERT cross-encoder that flags candidates the product-type heuristic lets through, cases where the category matches but the brand, size, or resolution is wrong, then sort the rest into heuristic negatives. On the capacity side, they move from DistilBERT to GTE-base, but note that a cold-start from a public checkpoint throws away years of accumulated domain knowledge and regresses precision. So, before fine-tuning, they train the new student to match the legacy production model’s temperature-scaled score distribution via KL divergence, then proceed with the enriched negatives. In their experiments, human raters score +7.34% NDCG@5, and A/B testing shows +0.50% gross revenue.
📚 https://arxiv.org/abs/2607.10096
[9] Beyond Semantic IDs: Encoding Business-Value Ranking into Document Identifiers for Generative Retrieval
This paper from Alibaba tackles a specific problem in generative retrieval. Document identifiers built purely from semantic embeddings optimize for semantic reconstruction rather than what the business actually cares about, and they collide once the corpus runs to hundreds of millions of items. The authors propose Cluster-Ranked Identifier (CRID), which splits the DocID into two parts. The first levels come from ordinary contrastive-learned embeddings quantized with RQ-KMeans, giving coarse semantic clusters. The last level, instead of another semantic codebook, is the item’s ordinal rank by business value within its cluster, typically 30-day conversion count. That single change makes identifiers collision-free by construction and allows absorbing new items by reranking within affected clusters daily without codebook retraining. On a 300M-item Taobao pool with Qwen2.5-0.5B as the backbone, CRID beats OPQ, Sinkhorn-Knopp balancing, Tiger, and FORGE, and holds up regardless of whether the ranking signal is conversions, clicks, or an internal quality score. Deployed as an extra retrieval channel, it delivered +1.06% GMV on full traffic.
📚 https://arxiv.org/abs/2607.11392
[10] MMRM: A Multiplex Multimodal Representation Model for Product Ranking in E-commerce Search
This paper from JD[.com] explores how e-commerce search ranking uses multimodal data. Existing work fine-tune a multimodal LLM on one collaborative signal, usually query-to-item clicks, and then drop the resulting embedding into the ranking model as just another item feature. The authors argue that e-commerce signals are heterogeneous, since q2i clicks capture broad relevance while item-to-item cart and order behaviors encode much finer similarity and complementarity, and that a single embedding table entangles tasks in a multitask setting. Their answer is MMRM (Multiplex Multimodal Representation Model), which builds four triplet datasets (q2i click plus i2i click, cart, and order, each 0.3B triplets). The model starts from Qwen3-VL-4B-Instruct and appends four special tokens, SEARCH, CLICK, CART, and ORDER, projecting each token’s last hidden state through its own MLP head. Four contrastive losses share the backbone, so one forward pass yields four disentangled item embeddings instead of four separate passes. Downstream, the ranking model uses each task’s embedding table for soft-search behavior retrieval, builds a task-specific user representation via multi-head target attention, then feeds everything through MMoE into CTR, ACR, and CVR towers. A week-long A/B test showed roughly 0.35% to 0.42% lifts on user-level CTR, ACR, and CVR.
📚 https://arxiv.org/abs/2607.11030
I hope this weekly roundup of top papers has provided you with valuable insights and a glimpse into the exciting advancements taking place in the field. Remember to look deeper into the papers that pique your interest.
I also blog about Machine Learning, Deep Learning, MLOps, and Software Engineering domains. I explore diverse topics, such as Natural Language Processing, Large Language Models, Recommendation Systems, etc., and conduct in-depth analyses, drawing insights from the latest research papers.