Skip to main content
Search

# Sparsity Penalty

2 related articles

Subscribe to updates
  1. arXiv

    Zero-shot RL stops losing the goal in a dense task code: one sparsity penalty lifts DMC average return by 53 points

    Adding a sparsity penalty to TD-JEPA's task encoder concentrates the inferred reward near the goal, raising average return on DMC by 53 points and average success rate on OGBench by 10.4 percentage points in zero-shot reinforcement learning.

    University of California, Merced
    Duaa Nakshbandi（First author）
    Sparse-TD-JEPAZero-Shot Reinforcement LearningTemporal-Difference Joint-Embedding Predictive ArchitectureSparsity PenaltySuccessor Features
  2. arXiv

    FactorEngram factorizes n-gram memory over a shared dictionary, lifting 8K long-context retrieval by up to 43.3 points at 340M and 1B backbones

    The work proposes FactorEngram, which replaces monolithic per-pattern n-gram embeddings with sparsity-regularized coefficients over a dictionary shared across patterns and reuses that same dictionary for basis-level contextual gating, improving language modeling, downstream tasks, and long-context retrieval on 340M- and 1B-parameter Transformer backbones.

    Nanyang Technological UniversityBaidu (China)
    Bowen Yang（First author）
    FactorEngramLookup-based memoryBasis-level contextual gatingSparsity PenaltyLanguage modeling

You’re all caught up