Sihan Ren 🥳
Sihan Ren

Junior Student

ShanghaiTech University
UC Berkeley

About Me

Hi! I’m Sihan(Tabi) Ren, a computer science student. My main research interests are in human motion animation and LLM Agents. I enjoy working on cool things and don’t mind getting my hands dirty to build stuff! On this website, I’ve listed some of the interesting projects I’ve worked on—hope you find them exciting too!

Download CV
Interests
  • Human Motion Perception
  • Human Motion Synthesis
  • Large Language Model Agents
  • Computer Vision
Education

  • University of California, Berkeley

    Computer Science, Visiting Student

  • ShanghaiTech University

    Computer Science & Interactive Design, Undergrad

Education

  1. University of California, Berkeley

    Computer Science, Visiting Student

    GPA: 4.0/4.0

    Highlight Courses: LLM Agents, Computer Vision, Computer Graphics, Artificial Intelligence

    Logo

  2. ShanghaiTech University

    Computer Science & Interactive Design, Undergrad

    GPA: 3.74/4.0, Merit Student (for top 15% of students)

    Highlight Courses: Machine Learning, 3D Visual Concept Design(Blender), Unity Game Development, Critical Studies of VR Animation Expression

    Logo
Featured Publications
SCOPE: Sign Language Contextual Processing with Embedding from LLMs
Sign Language Recognition & Translation

SCOPE: Sign Language Contextual Processing with Embedding from LLMs

Abstract: Sign languages, used by around 70 million Deaf individuals globally, are visual languages that convey visual and contextual information. Current methods in vision-based sign language recognition (SLR) and translation (SLT) struggle with dialogue scenes due to limited dataset diversity and the neglect of contextually relevant information. To address these challenges, we introduce SCOPE (Sign language Contextual Processing with Embedding from LLMs), a novel context-aware vision-based SLR and SLT framework. For SLR, we utilize dialogue contexts through a multi-modal encoder to enhance gloss-level recognition. For subsequent SLT, we further fine-tune a Large Language Model (LLM) by incorporating prior conversational context. We also contribute a new sign language dataset that contains 72 hours of Chinese sign language videos in contextual dialogues across various scenarios. Experimental results demonstrate that our SCOPE framework achieves state-of-the-art performance on multiple datasets, including Phoenix-2014T, CSL-Daily, and our SCOPE dataset. Moreover, surveys conducted with participants from the Deaf community further validate the robustness and effectiveness of our approach in real-world applications. Both our dataset and code will be open-sourced to facilitate further research.

OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers
Human Motion Synthesis

OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers

Abstract: We have recently seen tremendous progress in realistic text-to-motion generation. Yet, the existing methods often fail or produce implausible motions with unseen text inputs, which limits the applications. In this paper, we present OMG, a novel framework, which enables compelling motion generation from zero-shot open-vocabulary text prompts. Our key idea is to carefully tailor the pretrain-then-finetune paradigm into the text-to-motion generation. At the pre-training stage, our model improves the generation ability by learning the rich out-of-domain inherent motion traits. To this end, we scale up a large unconditional diffusion model up to 1B parameters, so as to utilize the massive unlabeled motion data up to over 20M motion instances. At the subsequent fine-tuning stage, we introduce motion ControlNet, which incorporates text prompts as conditioning information, through a trainable copy of the pre-trained model and the proposed novel Mixture-of-Controllers (MoC) block. MoC block adaptively recognizes various ranges of the sub-motions with a cross-attention mechanism and processes them separately with the text-token-specific experts. Such a design effectively aligns the CLIP token embeddings of text prompts to various ranges of compact and expressive motion features. Extensive experiments demonstrate that our OMG achieves significant improvements over the state-of-the-art methods on zero-shot text-to-motion generation.

Projects
DREAMoR - Diffusion-based REconstruction And Motion prioR
Mesh Any Gaussians
Facial Keypoint Detection
Computer Graphics Projects
SPARK - A Scavenger Hunt Game for LLM Agents
LLM Agents

SPARK - A Scavenger Hunt Game for LLM Agents

We propose a novel open-source testing framework and benchmark in the field of Vision-Language Navigation (VLN) to evaluate the goal-seeking capabilities of Large Language Model (LLM) agents in real-world environments. To this end, we designed a QA agent that operates without relying on human supervision or data annotations, serving as a semantic heuristic function to provide navi- gational cues to the agent under evaluation. Additionally, we leveraged techniques such as Rein-forcement Learning with AI Feedback (RLAIF) to develop new metrics for detailed analysis of the agent’s progressive information acquisition, multimodal cross-inference, and spatial reasoning abilities. Experimental results demonstrate significant room for improvement in current LLM agents across these dimensions. Future work may explore enhancing LLMs’ visual perception capabilities and their alignment of spatial information with semantic understanding.

See all
Publications
Yuqi Liu, Wenqian Zhang, Sihan Ren, Chengyu Huang, Jingyi Yu, Lan Xu (2024). SCOPE: Sign Language Contextual Processing with Embedding from LLMs. AAAI 2025, 2024.
PDF Cite Code Dataset Project
Han Liang, Jiacheng Bao, Ruichi Zhang, Sihan Ren, Yuecheng Xu, Sibei Yang, Xin Chen, Jingyi Yu, Lan Xu (2023). OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers. CVPR 2024, 2023.
PDF Cite Code Project Video