Tonmoay Deb

I am a Data and Applied Scientist II at Microsoft, exploring machine learning and natural language processing as part of the Windows Group. I received my Ph.D. in Computer Science from Northwestern University, where I was a member of the Northwestern Security and AI Lab (NSAIL), advised by Dr. V.S. Subrahmanian. My research expertise spans cutting-edge AI including multimodal AI, computer vision, natural language processing, generative modeling, responsible AI frameworks, multi-agent systems, and robotics applications.

My doctoral research tackled a pressing real-world challenge: how to defend cities from weaponized drones while respecting the law. I developed autonomous systems for early threat prediction and multi-agent coordination, validated by the Dutch Police and The Hague municipality. A key finding was that incorporating legal constraints can actually improve tactical performance when defenders are outnumbered.

Tonmoay Deb

I spent two summers (2024 and 2025) in machine learning and natural language processing applied research at Microsoft and contributed to multimodal AI research at Zillow Group with Dr. Sing Bing Kang. I also collaborated on aerial perception at Carnegie Mellon's Robotics Institute with Dr. Sebastian Scherer. Before Northwestern, I spent Summer 2021 at the University of New Hampshire, working on unsupervised coral reef segmentation at the Center for Coastal and Ocean Mapping with Dr. Yuri Rzhanov and Dr. Kim Lowell.

My research interest on autonomous drone defense began at North South University with PATRON [demo], a multi-faceted research initiative combining crowd behavior analysis, surveillance, and anomaly detection. I was glad to be supervised by excellent advisors Dr. Mohammad Rashedur Rahman, Mr. Adnan Firoze, and Dr. Mohammad Ashrafuzzaman Khan. I graduated with Summa Cum Laude from North South University. I then explored video captioning and Bengali NLP as a Research Associate at Independent University Bangladesh's AGenCy Lab, supervised by Dr. Amin Ahsan Ali, Dr. A K M Mahbubur Rahman, and Dr. Iftekhar Tanveer.

My overarching dream is to give dreams to machines: the ability to perceive, reason, and act on real-world problems.

News

  • October 2024: Paper on Pareto-Optimal Agent Decision Making published in IEEE Transactions on Cybernetics.
  • October 2024: DEWS Invention Disclosure submitted to Northwestern INVO.
  • June 2024: Data and Applied Scientist Intern at Microsoft, Windows Update Platforms.
  • April 2024: Paper "ZInD-Tell" accepted at CVPR 2024 Workshop (MULA). US Patent submitted.
  • June 2023: Started Summer Research Internship (Computer Vision) at Zillow Inc.
  • December 2022: Received AAAI Conference Travel Grant.
  • December 2022: Received MS in Computer Science at Northwestern University.
  • December 2022: Featured in Northwestern McCormick School of Engineering News.
  • October 2022: DUCK Demo paper accepted at AAAI 2023.
  • October 2022: Presented DUCK at Conference on AI and National Security. [Tweet]
  • October 2021: One paper accepted at WACV 2022.
  • September 2021: Joined Northwestern University for Ph.D. in Computer Science.
  • May 2021: Started Summer Research Internship at CCOM, University of New Hampshire.
  • April 2021: One paper from PATRON accepted at IJCNN 2021.
  • February 2021: Starting Masters in Computer Science (thesis-based) at University of New Hampshire.

Ph.D. Dissertation

Responsible Defense from Multi-Drone Attacks
Tonmoay Deb
Ph.D. Dissertation, Northwestern University, 2025
Advisor: Dr. V.S. Subrahmanian (Chair)
Committee: Dr. Larry Birnbaum (Northwestern University), Dr. Nabil Alshurafa (Northwestern University), Dr. Alberto Quattrini Li (Dartmouth College)

Commercial drones weaponized by non-state actors achieve attack success rates above 70%, yet existing counter-drone systems focus on detection rather than threat assessment and legally compliant response. This dissertation develops four interconnected systems for autonomous urban drone defense:

DEWS (Drone Early Warning System) classifies drone trajectories as threatening or benign based on initial flight segments. Evaluated on 349 real-world trajectories provided by the Dutch Police over eight months in The Hague, DEWS achieves F1-scores exceeding 0.80 within 30 seconds of initial observation.

STATE (Safe and Threatening Adversarial Trajectory Engine) addresses data scarcity through conditional generative adversarial networks that synthesize threat-conditioned drone trajectories based on geographic context.

POSS (Pareto-Optimal Status Sets) formalizes multi-objective decision-making under legal constraints using deontic logic and Pareto optimization, ensuring autonomous agents never select legally impermissible actions.

GUARDIAN (Governance-Unified Aerial Reinforcement-Defense In Accordance with Norms) coordinates defensive drone swarms using multi-agent reinforcement learning while maintaining strict legal compliance through POSS-based action masking. A key finding is that incorporating legal constraints can actually improve defensive performance when defenders are outnumbered, challenging assumptions that compliance necessarily compromises tactical effectiveness.

Publications

Journal Papers / Book Chapters

Stackelberg drone defense
D. Mutzari, Tonmoay Deb, C. Molinaro, A. Pugliese, V.S. Subrahmanian, and S. Kraus
Artificial Intelligence Journal (Elsevier), Volume 349, 2026. Accepted for publication.
Extends Sequential Stackelberg Security Games to handle multiple attack/defense drones with payload and battery constraints. Experiments on 80 real-world cities.
DEWS system
Tonmoay Deb, S. de Laaf, V. La Gatta, O. Lemmens, R. Lindelauf, M. van Meerten, H. Meerveld, A. Neeleman, M. Postiglione, and V.S. Subrahmanian
IEEE Intelligent Systems, 2025.
Framework to solve the Drone Threat Prediction Problem. DEWS makes accurate predictions within 30 seconds of flight with an F1-score of over 80% on data from a major European city.
POSS framework
Tonmoay Deb, M. Jeong, C. Molinaro, A. Pugliese, A. Quattrini Li, E. Santos, V.S. Subrahmanian, and Y. Zhang
IEEE Transactions on Cybernetics, 54(12): 7147-7162, 2024.
Framework for multi-objective decision-making under legal constraints using deontic logic and Pareto-optimal status sets.
Color harmony
T. Osman, S.S. Psyche, Tonmoay Deb, A. Firoze, and R.M. Rahman
In Systems Simulation and Modeling for Cloud Computing and Big Data Applications, Academic Press/Elsevier, 2020.
Bengali image captioning
Tonmoay Deb, M.Z.A. Ali, S. Bhowmik, A. Firoze, S.S. Ahmed, M.A. Tahmeed, N. Rahman, and R.M. Rahman
Journal of Intelligent and Fuzzy Systems, vol. 37, pp. 7427-7439, 2019.
First work addressing Bengali image caption generation via subsampling machine-translated captions, with CNN-LSTM architectures.
Cognitive aesthetics
T. Osman, S.S. Psyche, Tonmoay Deb, A. Firoze, and R.M. Rahman
Journal of Information and Telecommunication, vol. 3, no. 2, 2019.
Learning model estimating cognitive perception of aesthetics by incorporating human psychology with computational metrics across 5000 images.

Conference Papers

Multi-object tracking
M. Jeong, C. Molinaro, Tonmoay Deb, Y. Zhang, A. Pugliese, E. Santos Jr., V.S. Subrahmanian, and A. Quattrini Li
arXiv preprint arXiv:2502.01041, 2025.
Comprehensive multi-agent pipeline with LSTM trajectory prediction and information-theoretic optimization. 1.3x to 3.2x faster mission completion versus baselines.
ZInD-Tell indoor panorama
Tonmoay Deb, L. Wang, Z. Bessinger, N. Khosravan, E. Penner, and S.B. Kang
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024.
Novel extension to the Zillow Indoor Dataset for multimodal indoor environment understanding. Introduces language-based home retrieval and indoor description generation tasks.
DUCK Demo
Tonmoay Deb, J. Dix, M. Jeong, C. Molinaro, A. Pugliese, A. Quattrini Li, E. Santos, V.S. Subrahmanian, S. Yang, and Y. Zhang
Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI), 2023 (Demonstration Track).
Multi-agent testbed for simulating drone-based attacks and developing surveillance camera, drone, and cyber defenses. Over 10,100 lines of code in C++ and Python.
VSLAN output
Tonmoay Deb, A. Sadmanee, K.K. Bhaumik, A.A. Ali, M.A. Amin, and A.K.M.M. Rahman
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022.
Novel approach for diverse video captioning with CIDEr improvement of 7.8% on MSVD and 4.5% on MSR-VTT over state-of-the-art.
UAV-CROWD simulator
M. Rahmun, Tonmoay Deb, S.A. Bijoy, and M.H. Raha
arXiv preprint arXiv:2208.06702, 2022.
Photo-realistic synthetic crowd simulator using Unreal Engine and AirSim. Augmenting synthetic data improves binary video classification accuracy by 2-8%.
HyMP tracking
Tonmoay Deb, M. Rahmun, M.H. Raha, S.A. Bijoy, and M.A. Khan
International Joint Conference on Neural Networks (IJCNN), IEEE, 2021.
Novel crowd group tracking benchmark dataset (70 sequences, 161.6K frames) and HyMP algorithm with graph convolutional networks. Outperforms state-of-the-art by 7.5%.
Violence detection
Tonmoay Deb, A. Arman, and A. Firoze
17th IEEE International Conference on Machine Learning and Applications (ICMLA), 2018.
Outlier-Resistant VLAD encoding using median and interquartile range for robust video violence detection. Near-perfect classification on standard benchmarks.

Patents and Intellectual Property

DEWS system
DEWS: Drone Early Warning System
Tonmoay Deb et al.
Invention Disclosure (Disc-ID-24-10-23-001), Northwestern University INVO, October 2024. Under review.
Invention disclosure for the DEWS framework submitted to the Northwestern University Innovation and New Ventures Office (INVO).
ZInD-Tell
Indoor Panorama Description Generation and Language-Based Home Retrieval
Tonmoay Deb, L. Wang, Z. Bessinger, N. Khosravan, E. Penner, and S.B. Kang
US Patent Application, filed via Zillow Group, 2024. Under review.
Patent application covering the ZInD-Tell system for translating indoor panoramic images into natural language descriptions and enabling language-based home retrieval.

Technical Papers

Generative deep models reviews
Generative Deep Models: Course Paper Reviews
Tonmoay Deb
Northwestern University (Generative Deep Models), Fall 2022.
Comprehensive reviews of foundational papers including Attention Is All You Need (Transformer), BERT, and GPT-3.
Federated learning
Modeling Robustness to Distribution Diversity Among Participants in Federated Learning
Tonmoay Deb, N. Franzese, M. Chen, and A. Vijayaraghavan
Northwestern University (Reliability and Robustness in Machine Learning), Fall 2021.
Addresses federated learning with non-malicious but distribution-shifted participants, proposing Byzantine-resilient aggregation for medical domain applications.
Coral reef segmentation
Coral Reef Image Segmentation and Classification: NOAA-UNH CCOM Summer Internship Report
Tonmoay Deb
Center for Coastal and Ocean Mapping / NOAA-UNH Joint Hydrographic Center, University of New Hampshire, Summer 2021.
Internship research report exploring unsupervised and few-shot learning methods for coral reef image segmentation and classification on the MLC dataset. Developed a refined patch extraction pipeline that improved ProtoNet few-shot accuracy from ~21% to ~42%. Also applied YOLOv5 for coral class detection and IIC for unsupervised semantic segmentation, with image cleaning pipelines to remove calibration artifacts.
Language-conditioned controller
An Approach to Design Language-Conditioned Low-Level Visual Controller
Tonmoay Deb and M. Begum
University of New Hampshire (Human-Robot Interaction), Spring 2021.
Language-modulated low-level controller for robot manipulation using visual data and natural language commands, with a full-scale HRI study involving 7 participants.
COVID-19 reinforcement learning
Contributions on Solving COVID-19 Crisis with Reinforcement Learning
Tonmoay Deb and F. Mikulis-Borsoi
University of New Hampshire (Advanced Reinforcement Learning), Spring 2021.
Formulates COVID-19 lockdown policy as a Markov Decision Process, using Value Iteration and Soft-Robust Optimization for handling uncertain transition probabilities.

Service

Peer Review

Teaching

MSAI 371: Representation, Reasoning, and Language (Northwestern University, Winter 2026, MS in AI Program)
COMP SCI 348: Introduction to Artificial Intelligence (Northwestern University, Spring 2025 and Fall 2025, ~150 students per section)
COMP SCI 349: Machine Learning (Northwestern University, Fall 2024, ~160 students)
CS415/CS416: Introduction to Computer Science I and II (University of New Hampshire, Spring 2021)
CSE115: Fundamentals of Programming and CSE225: Data Structures and Algorithms (North South University, 2020-2021, Lab Instructor, 312 students total)
CSE445: Machine Learning (North South University, Fall 2020, Graduate TA, 40 students)
CSE311: Database Systems (North South University, Spring-Fall 2019, Undergraduate TA, 240 students in six sections)

Honors, Awards, and Talks

AAAI-2026 Program Committee Member
August 2025
Selected for the 40th AAAI Conference on Artificial Intelligence, Singapore.
Salesforce AI Research Future Forum
May 2025
Invited presenter at Salesforce Tower Chicago. Lightning poster presentation on ZInD-Tell research.
ICMLA 2023 Program Committee Member
December 2023
Selected for the 22nd IEEE International Conference on Machine Learning and Applications, Florida.
AAAI-2023 Travel Grant
December 2022
Competitive travel grant scholarship to attend the 37th AAAI Conference, Washington, D.C.
Conference on AI and National Security
October 2022
Northwestern Departmental Fellowship
September 2021
Ph.D. enrollment with departmental fellowship at Northwestern University.
North South University Innovation Challenge Season 8
August 2019
Secured 2nd Runner-up position in the competitive capstone project showcase event.
The Undergraduate Awards (Dublin, Ireland)
November 2018
Highly Commended Entrant in Computer Science (Top 10% out of 4,887 worldwide submissions), under the patronage of the President of Ireland. Project: "Machine Cognition of Violence in Videos with Median-IQR VLAD Encoding."
Merit Scholarship, North South University
2017 - 2019
100% tuition fee waiver based scholarship in Merit Quota (awarded to top 0.5% of NSU students).