Animesh Gupta

PhD student at the University of Central Florida, advised by Dr. Mubarak Shah. I build multimodal models for retrieval and grounded understanding across images and videos.

Email Scholar GitHub Twitter CV

My research centers on retrieval with models that integrate vision and language, with an emphasis on modeling efficiency and fine-grained temporal reasoning.

In summer 2026 I was an Applied Researcher Intern at eBay in San Jose, building multimodal pivot retrieval for product search. Before my PhD, I was a Machine Learning Engineer Intern at MVisionAI, working on medical image registration for radiotherapy planning. I received my Bachelor's in Electronics and Computer Engineering from Thapar Institute of Engineering & Technology, India, and interned at UiT – The Arctic University of Norway (coreset-based data selection for efficient training) and the SketchX Lab, University of Surrey (sketch-based visual understanding).

What's new

2026

Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs accepted at ECCV 2026.

May 2026

Interned at eBay (San Jose) as Applied Researcher through August on multimodal pivot retrieval.

May 2026

Enhancing Box and Block Test with Computer Vision for Post-Stroke Upper Extremity Motor Evaluation accepted at IEEE EMBC 2026.

Sept 2025

From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos accepted at NeurIPS 2025.

June 2025

StrokeVision-Bench: A Multimodal Video and 2D Pose Benchmark for Tracking Stroke Recovery accepted at IEEE MLSP 2025.

Aug 2024

Started PhD at University of Central Florida, supervised by Dr. Mubarak Shah.

May 2023

Big-Bench accepted at TMLR 2023.

Feb 2023

Started internship at MVisionAI.

July 2022

Adaptive Fine-Grained Sketch-Based Image Retrieval accepted at ECCV 2022.

May 2022

Research internship at UiT – The Arctic University of Norway.

Mar 2022

Research internship at NVIDIA, India.

July 2021

Research internship at SketchX, University of Surrey.

Publications

VISAGE pipeline: attention-based grounding multiplier re-ranks proposed tokens

ECCV 2026

Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs

Vishal Narnaware*, Animesh Gupta*, Kevin Zhai, Zhenyi Wang, Mubarak Shah

* equal contribution

Website Paper

Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.

@inproceedings{narnaware2026seeing,
  title={Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs},
  author={Narnaware, Vishal and Gupta, Animesh and Zhai, Kevin and Wang, Zhenyi and Shah, Mubarak},
  booktitle={European Conference on Computer Vision},
  pages={413--430},
  year={2026},
  organization={Springer}
}

Pipeline: BBT video frames to world-aligned 3D joint angles

IEEE EMBC 2026

Enhancing Box and Block Test with Computer Vision for Post-Stroke Upper Extremity Motor Evaluation

David Robinson, Animesh Gupta, Elizabeth Clark, Olga Melnik, Qiushi Fu, Mubarak Shah

Paper

Standard clinical assessments of upper-extremity motor function after stroke either rely on ordinal scoring, which lacks sensitivity, or time-based task metrics, which do not capture movement quality. In this work, we present a computer vision-based framework for analysis of upper-extremity movement during the Box and Block Test (BBT) through world-aligned joint angles of fingers, arm, and trunk without depth sensors or calibration objects. We apply this framework to a dataset of 136 BBT recordings collected from 48 healthy individuals and 7 individuals post stroke. Using unsupervised dimensionality reduction of joint-angle features, we analyze movement patterns without relying on expert clinical labels. The resulting embeddings show separation between healthy movement patterns and stroke-related movement deviations. Importantly, some patients with the same BBT scores can be separated with different postural patterns. These results show that world-aligned joint angles can capture meaningful information of upper-extremity functions beyond standard time-based BBT scores, with no effort from the clinician other than monocular video recordings of the patient using a phone or camera. This work highlights the potential of a camera-based, calibration-free framework to measure movement quality in clinical assessments without changing the widely adopted clinical routine.

@article{robinson2026enhancing,
  title={Enhancing Box and Block Test with Computer Vision for Post-Stroke Upper Extremity Motor Evaluation},
  author={Robinson, David and Gupta, Animesh and Clark, Elizabeth and Melnik, Olga and Fu, Qiushi and Shah, Mubarak},
  journal={arXiv preprint arXiv:2603.29101},
  year={2026}
}

NeurIPS 2025 · Datasets & Benchmarks

From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos

Animesh Gupta, Jay Parmar, Ishan Rajendrakumar Dave, Mubarak Shah

Website Code Paper

Composed Video Retrieval (CoVR) retrieves a target video given a query video and a modification text describing the intended change. Existing CoVR benchmarks emphasize appearance shifts or coarse event changes and therefore do not test the ability to capture subtle, fast-paced temporal differences. We introduce TF-CoVR, the first large-scale benchmark dedicated to temporally fine-grained CoVR. TF-CoVR focuses on gymnastics and diving and provides 180K triplets drawn from FineGym and FineDiving. Previous CoVR benchmarks focusing on temporal aspect, link each query to a single target segment taken from the same video, limiting practical usefulness. In TF-CoVR, we instead construct each <query, modification> pair by prompting an LLM with the label differences between clips drawn from different videos; every pair is thus associated with multiple valid target videos (3.9 on average), reflecting real-world tasks such as sports-highlight generation. To model these temporal dynamics we propose TF-CoVR-Base, a concise two-stage training framework: (i) pre-train a video encoder on fine-grained action classification to obtain temporally discriminative embeddings; (ii) align the composed query with candidate videos using contrastive learning. We conduct the first comprehensive study of image, video, and general multimodal embedding (GME) models on temporally fine-grained composed retrieval in both zero-shot and fine-tuning regimes. On TF-CoVR, TF-CoVR-Base improves zero-shot mAP@50 from 5.92 (LanguageBind) to 7.51, and after fine-tuning raises the state-of-the-art from 19.83 to 27.13.

@article{gupta2026play,
  title={From play to replay: Composed video retrieval for temporally fine-grained videos},
  author={Gupta, Animesh and Parmar, Jay and Dave, Ishan Rajendrakumar and Shah, Mubarak},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  year={2026}
}

IEEE MLSP 2025

StrokeVision-Bench: A Multimodal Video and 2D Pose Benchmark for Tracking Stroke Recovery

David Robinson, Animesh Gupta, Rizwan Quershi, Qiushi Fu, Mubarak Shah

Paper

Despite advancements in rehabilitation protocols, clinical assessment of upper extremity (UE) function after stroke remains largely subjective. This subjectivity limits the sensitivity of assessments to detect subtle motor improvements, which are critical for personalized rehabilitation planning. StrokeVision-Bench introduces the first clinically structured dataset for stroke rehabilitation benchmarking, consisting of 1,000 annotated videos categorized into clinically meaningful action classes across two modalities: raw video frames and 2D skeletal keypoints. We evaluate multiple state-of-the-art action recognition and skeleton-based classification models to establish baselines for automated stroke rehabilitation assessment.

@inproceedings{robinson2025strokevision,
  title={Strokevision-Bench: A Multimodal Video and 2D Pose Benchmark for Tracking Stroke Recovery},
  author={Robinson, David and Gupta, Animesh and Qureshi, Rizwan and Fu, Qiushi and Shah, Mubarak},
  booktitle={2025 IEEE 35th International Workshop on Machine Learning for Signal Processing (MLSP)},
  pages={1--6},
  year={2025},
  organization={IEEE}
}

arXiv 2023

Data-Efficient Training of CNNs and Transformers with Coresets: A Stability Perspective

Animesh Gupta, Irtiza Hasan, Dilip K Prasad, Deepak K Gupta

Code Paper

Coreset selection is among the most effective ways to reduce the training time of CNNs, however, only limited is known on how the resultant models will behave under variations of the coreset size, and choice of datasets and models. Moreover, given the recent paradigm shift towards transformer-based models, it is still an open question how coreset selection would impact their performance. There are several similar intriguing questions that need to be answered for a wide acceptance of coreset selection methods, and this paper attempts to answer some of these. We present a systematic benchmarking setup and perform a rigorous comparison of different coreset selection methods on CNNs and transformers. Our investigation reveals that under certain circumstances, random selection of subsets is more robust and stable when compared with the SOTA selection methods. We demonstrate that the conventional concept of uniform subset sampling across the various classes of the data is not the appropriate choice. Rather samples should be adaptively chosen based on the complexity of the data distribution for each class. Transformers are generally pretrained on large datasets, and we show that for certain target datasets, it helps to keep their performance stable at even very small coreset sizes. We further show that when no pretraining is done or when the pretrained transformer models are used with non-natural images (e.g. medical data), CNNs tend to generalize better than transformers at even very small coreset sizes. Lastly, we demonstrate that in the absence of the right pretraining, CNNs are better at learning the semantic coherence between spatially distant objects within an image, and these tend to outperform transformers at almost all choices of the coreset size.

@article{gupta2023data,
  title={Data-Efficient Training of CNNs and Transformers with Coresets},
  author={Gupta, Animesh and Hassan, Irtiza and Prasad, Dilip K and Gupta, Deepak K},
  year={2023}
}

TMLR 2023

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

BIG-bench authors · task contribution: dataset collection and LLM evaluation (Crash Blossom)

Code Paper

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

@article{srivastava2022beyond,
  title={Beyond the imitation game: Quantifying and extrapolating the capabilities of language models},
  author={Srivastava, Aarohi and Rastogi, Abhinav and Rao, Abhishek and Shoeb, Abu Awal Md and Abid, Abubakar and Fisch, Adam and Brown, Adam R and Santoro, Adam and Gupta, Aditya and Garriga-Alonso, Adri{\`a} and others},
  journal={arXiv preprint arXiv:2206.04615},
  year={2022}
}

ECCV 2022

Adaptive Fine-Grained Sketch-Based Image Retrieval

Ayan Kumar Bhunia, Aneeshan Sain, Parth Hiren Shah, Animesh Gupta, Pinaki Nath Chowdhury, Tao Xiang, Yi-Zhe Song

Code Paper

The recent focus on Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) has shifted towards generalising a model to new categories without any training data from them. In real-world applications, however, a trained FG-SBIR model is often applied to both new categories and different human sketchers, i.e., different drawing styles. Although this complicates the generalisation problem, fortunately, a handful of examples are typically available, enabling the model to adapt to the new category/style. In this paper, we offer a novel perspective -- instead of asking for a model that generalises, we advocate for one that quickly adapts, with just very few samples during testing (in a few-shot manner). To solve this new problem, we introduce a novel model-agnostic meta-learning (MAML) based framework with several key modifications: (1) As a retrieval task with a margin-based contrastive loss, we simplify the MAML training in the inner loop to make it more stable and tractable. (2) The margin in our contrastive loss is also meta-learned with the rest of the model. (3) Three additional regularisation losses are introduced in the outer loop, to make the meta-learned FG-SBIR model more effective for category/style adaptation. Extensive experiments on public datasets suggest a large gain over generalisation and zero-shot based approaches, and a few strong few-shot baselines.

@inproceedings{bhunia2022adaptive,
  title={Adaptive fine-grained sketch-based image retrieval},
  author={Bhunia, Ayan Kumar and Sain, Aneeshan and Shah, Parth Hiren and Gupta, Animesh and Chowdhury, Pinaki Nath and Xiang, Tao and Song, Yi-Zhe},
  booktitle={European Conference on Computer Vision},
  pages={163--181},
  year={2022},
  organization={Springer}
}

Research experience

Applied Researcher Intern, eBay

May 2026 – August 2026 · San Jose, California

Worked on multimodal pivot retrieval: product search from a reference image plus a natural-language intent modification, beyond image-only or text-only retrieval.
Built a scalable binary-embedding perturbation strategy to mine related product pairs from eBay's 2.3B-product catalog, and used Qwen3-VL to generate intent modifications, constructing a 4M-triplet composed retrieval dataset (3M train / 1M test).
Developed a flow-based multimodal retrieval model on top of eBay's existing search models, improving mAP@25 from 9.45 to 27.06 (2.9×) and Recall@10 from 55.21 to 90.98 (+35.77 points).

Machine Learning Engineer Intern, MVisionAI

February 2023 – July 2024 · Helsinki, Finland · Supervisor: Dr. Saad Ullah Akram

Worked on radiotherapy planning using multi-modal image registration (CT and MRI).
Built an efficient library supporting multiple datasets and algorithms.
Adapted RWCNet and Transmorph for OASIS and NLST datasets; established AbdomenCTCT baselines.

Research Intern, University of Tromsø

May 2022 – November 2022 · Tromsø, Norway · Supervisors: Dr. Deepak Gupta, Dr. Irtiza Hasan, Dr. Dilip Prasad

Created a benchmarking setup for coreset selection on CNNs and Transformers.
Showed class-complexity-driven sampling outperforms uniform sampling.
Findings led to a research publication available on arXiv.

Research Intern, NVIDIA

March 2022 – May 2022

Worked on real-time lane detection and vision transformers for DRIVE-Perceptron.
Focused on optimizing inference and improving real-world performance.

Research Intern, SketchX

July 2021 – March 2022 · Supervisor: Dr. Yi-Zhe Song

Worked on fine-grained and category-level sketch-based image retrieval.
Co-authored ECCV 2022 paper on few-shot FG-SBIR adaptation across categories and styles.

Intern, GirlScript Summer of Code

March 2021 – June 2021

Face-X: Implemented NasNet and Xception for face recognition. [PRs]
Comet.Box: Added YOLOv5-based object detection example. [PRs]

Research Engineer, Minus Zero

October 2020 – March 2021

Worked on road segmentation for autonomous driving in Indian traffic scenarios.
Trained FCHarDNet on a 10k-image dataset annotated with 34 semantic classes.