publications

2024

1. FEABench: Evaluating Large Language Models on Real World Physics Reasoning Ability

   Nayantara Mudur, Hao Cui, Subhashini Venugopalan, Paul Raccuglia, Michael Brenner, and Peter Norgaard

   Submitted, under review Workshops: NeurIPS Math-AI, NeurIPS Open World Agents , 2024

   Abs Bib

   Building precise simulations of the real world and invoking numerical solvers to answer quantitative problems is an essential requirement in engineering and science. We present FEABench, a benchmark to evaluate the ability of large language models (LLMs) and LLM agents to simulate and solve physics, mathematics and engineering problems using finite element analysis (FEA). We introduce a multipronged evaluation scheme to investigate the ability of LLMs to solve these problems by reasoning over natural language problem descriptions and operating COMSOL Multiphysics®, an FEA software, to compute the answers. In addition to testing state-of-the art-LLMs, we further design a language model agent equipped with the ability to interact with the software through its Application Programming Interface (API), examine its outputs and use tools to improve its solutions over multiple iterations. Our best performing strategy generates executable API calls 88% of the time. However, this benchmark still proves to be challenging enough that the LLMs and agents we tested were not able to completely and correctly solve any problem. LLMs that can successfully interact with and operate FEA software to solve problems such as those in our benchmark would significantly push the frontiers of their utility. Acquiring this capability would augment LLMs’ reasoning skills with the precision of numerical solvers and advance the development of autonomous systems that can tackle complex problems in the real world.

   @article{mudur2024feabench,
     title = {FEABench: Evaluating Large Language Models on Real World Physics Reasoning Ability},
     author = {Mudur, Nayantara and Cui, Hao and Venugopalan, Subhashini and Raccuglia, Paul and Brenner, Michael and Norgaard, Peter},
     journal = {Submitted, under review},
     year = {2024},
   }

2. Diffusion-HMC: Parameter Inference with Diffusion Model driven Hamiltonian Monte Carlo

   Nayantara Mudur, Carolina Cuesta-Lazaro, and Douglas P Finkbeiner

   Accepted, The Astrophysical Journal , 2024

   Abs Bib Code

   Diffusion generative models have excelled at diverse image generation and reconstruction tasks across fields. A less explored avenue is their application to discriminative tasks involving regression or classification problems. The cornerstone of modern cosmology is the ability to generate predictions for observed astrophysical fields from theory and constrain physical models from observations using these predictions. This work uses a single diffusion generative model to address these interlinked objectives – as a surrogate model or emulator for cold dark matter density fields conditional on input cosmological parameters, and as a parameter inference model that solves the inverse problem of constraining the cosmological parameters of an input field. The model is able to emulate fields with summary statistics consistent with those of the simulated target distribution. We then leverage the approximate likelihood of the diffusion generative model to derive tight constraints on cosmology by using the Hamiltonian Monte Carlo method to sample the posterior on cosmological parameters for a given test image. Finally, we demonstrate that this parameter inference approach is more robust to the addition of noise than baseline parameter inference networks.

   @article{mudur2024diffusion,
     title = {Diffusion-HMC: Parameter Inference with Diffusion Model driven Hamiltonian Monte Carlo},
     author = {Mudur, Nayantara and Cuesta-Lazaro, Carolina and Finkbeiner, Douglas P},
     journal = {Accepted, The Astrophysical Journal },
     year = {2024},
   }

3. Debiasing with Diffusion: Probabilistic reconstruction of Dark Matter fields from galaxies with CAMELS

   Victoria Ono, Core Francisco Park, Nayantara Mudur, Yueying Ni, Carolina Cuesta-Lazaro, and Francisco Villaescusa-Navarro

   The Astrophysical Journal , 2024

   Bib

   @article{ono2024debiasing,
     title = {Debiasing with Diffusion: Probabilistic reconstruction of Dark Matter fields from galaxies with CAMELS},
     author = {Ono, Victoria and Park, Core Francisco and Mudur, Nayantara and Ni, Yueying and Cuesta-Lazaro, Carolina and Villaescusa-Navarro, Francisco},
     journal = {The Astrophysical Journal },
     volume = {970},
     number = {2},
     pages = {174},
     year = {2024},
     publisher = {IOP Publishing},
   }

4. Quantum Many-Body Physics Calculations with Large Language Models

   Haining Pan, Nayantara Mudur, Will Taranto, Maria Tikhanovskaya, Subhashini Venugopalan, Yasaman Bahri, Michael P Brenner, and Eun-Ah Kim

   Accepted, Communications Physics , 2024

   Abs Bib

   Large language models (LLMs) have demonstrated an unprecedented ability to perform complex tasks in multiple domains, including mathematical and scientific reasoning. We demonstrate that with carefully designed prompts, LLMs can accurately carry out key calculations in research papers in theoretical physics. We focus on a broadly used approximation method in quantum physics: the Hartree-Fock method, requiring an analytic multi-step calculation deriving approximate Hamiltonian and corresponding self-consistency equations. To carry out the calculations using LLMs, we design multi-step prompt templates that break down the analytic calculation into standardized steps with placeholders for problem-specific information. We evaluate GPT-4’s performance in executing the calculation for 15 research papers from the past decade, demonstrating that, with correction of intermediate steps, it can correctly derive the final Hartree-Fock Hamiltonian in 13 cases and makes minor errors in 2 cases. Aggregating across all research papers, we find an average score of 87.5 (out of 100) on the execution of individual calculation steps. Overall, the requisite skill for doing these calculations is at the graduate level in quantum condensed matter theory. We further use LLMs to mitigate the two primary bottlenecks in this evaluation process: (i) extracting information from papers to fill in templates and (ii) automatic scoring of the calculation steps, demonstrating good results in both cases. The strong performance is the first step for developing algorithms that automatically explore theoretical hypotheses at an unprecedented scale.

   @article{pan2024quantum,
     title = {Quantum Many-Body Physics Calculations with Large Language Models},
     author = {Pan, Haining and Mudur, Nayantara and Taranto, Will and Tikhanovskaya, Maria and Venugopalan, Subhashini and Bahri, Yasaman and Brenner, Michael P and Kim, Eun-Ah},
     journal = {Accepted, Communications Physics },
     year = {2024},
   }

2023

1. Stellar-reddening-based Extinction Maps for Cosmological Applications

   Nayantara Mudur, Core Francisco Park, and Douglas P Finkbeiner

   The Astrophysical Journal Data Products: Mouse-Over Map Animation , 2023

   Abs Bib HTML Code

   Cosmological surveys must correct their observations for the reddening of extragalactic objects by Galactic dust. Existing dust maps, however, have been found to have spatial correlations with the large-scale structure of the Universe. Errors in extinction maps can propagate systematic biases into samples of dereddened extragalactic objects and into cosmological measurements such as correlation functions between foreground lenses and background objects and the primordial non-gaussianity parameter fNL. Emission-based maps are contaminated by the cosmic infrared background, while maps inferred from stellar-reddenings suffer from imperfect removal of quasars and galaxies from stellar catalogs. Thus, stellar-reddening based maps using catalogs without extragalactic objects offer a promising path to making dust maps with minimal correlations with large-scale structure. We present two high-latitude integrated extinction maps based on stellar reddenings, with a point spread function of full-width half-maximum 6.1’ and 15’. We employ a strict selection of catalog objects to filter out galaxies and quasars and measure the spatial correlation of our extinction maps with extragalactic structure. Our galactic extinction maps have reduced spatial correlation with large scale structure relative to most existing stellar-reddening based and emission-based extinction maps.

   @article{mudur2023stellar,
     title = {Stellar-reddening-based Extinction Maps for Cosmological Applications},
     author = {Mudur, Nayantara and Park, Core Francisco and Finkbeiner, Douglas P},
     journal = {The Astrophysical Journal },
     volume = {949},
     number = {2},
     pages = {47},
     year = {2023},
     publisher = {IOP Publishing},
   }

2022

1. Can denoising diffusion probabilistic models generate realistic astrophysical fields?

   Nayantara Mudur, and Douglas P Finkbeiner

   2022

   Bib Code

   @article{mudur2022can,
     title = {Can denoising diffusion probabilistic models generate realistic astrophysical fields?},
     author = {Mudur, Nayantara and Finkbeiner, Douglas P},
     year = {2022},
   }

2021

1. Contrastive similarity matching for supervised learning

   Shanshan Qin, Nayantara Mudur, and Cengiz Pehlevan

   Neural computation , 2021

   Bib HTML

   @article{qin2021contrastive,
     title = {Contrastive similarity matching for supervised learning},
     author = {Qin, Shanshan and Mudur, Nayantara and Pehlevan, Cengiz},
     journal = {Neural computation },
     volume = {33},
     number = {5},
     pages = {1300--1328},
     year = {2021},
     publisher = {MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA},
   }