This started as a final project for CS066 and did not stop when the semester did. I've spent most of the summer reading PINN papers, reimplementing them, and running sweeps on a 96GB GPU that is dramatically overqualified for the job.
the idea
A physics-informed neural network is trained to satisfy a differential equation, not just to fit points.
You take the PDE, move everything to one side so a correct solution makes it zero, and then evaluate that expression on the network's output at a bunch of scattered points in the domain. However far from zero you land is the residual, and you add it to the loss. Do the same for the boundary and initial conditions and you have a network that gets pulled toward being a solution rather than toward memorizing samples.
The reason this is interesting is that the physics is doing the job data usually does. You can fit with very few observations, and noisy ones, because the equation rules out most of the functions that would otherwise explain your data.
Autodiff is what makes it practical. The derivatives in the PDE are derivatives of the network output with respect to its inputs, and torch.autograd.grad gives you those directly. Nothing gets discretized. There's no mesh. The network is the solution and its derivatives come for free, which is the part that felt like a trick the first time I wrote it out.
raissi
Raissi, Perdikaris, and Karniadakis (2019) is the paper everyone starts from. It names the thing and sets up both problems: forward, where you have the equation and want the solution, and inverse, where you have messy observations of the solution and want the equation's parameters.
The inverse case is what I keep coming back to. You're recovering a physical constant from data because measuring it directly is expensive or impossible, and the structure of the equation is what makes that recoverable at all.
I implemented it from scratch, and the training loop is genuinely short. Sample points, compute the residual through autograd, add the boundary loss, step.
What isn't short is getting it to converge. The weight on the physics term relative to the data term changes everything, and equal weighting almost never works. Too much weight on the residual and the network happily satisfies the PDE while ignoring your boundary conditions, which gives you a beautiful solution to a different problem. I lost a while to that before I understood what I was looking at.
deepxde
After doing it by hand I moved to DeepXDE, Lu Lu's library, which handles the geometry, the sampling, and the loss assembly so you can describe a problem in twenty lines.
It's very good for trying things. It also hides enough that I stopped understanding my own results, so I ended up in a pattern where I prototype in DeepXDE and then reimplement anything I want to make a claim about. Reading its source was worth more to me than using it, honestly. A lot of the sampling and weighting tricks in the literature are in there, written down properly.
stiffness
Vanilla PINNs fall apart on stiff systems, where some components move on microsecond timescales and others on hours. Chemical kinetics is the standard example.
The failure is in the conditioning. Gradients from the fast components are enormous compared to the slow ones, so the optimizer spends everything it has on the fast dynamics and never makes progress on the rest. You watch the loss go down and the solution stay wrong.
Ji et al.'s Stiff-PINN handles this with a curriculum: train on a short time window where the problem hasn't gotten stiff yet, then extend it. The network learns the fast behavior on a domain where that's most of what there is, and carries it forward.
I built a simplified version for a three-reaction combustion ODE. It clearly works, in that it solves problems the vanilla version doesn't. It's also much fussier. How fast you expand the window matters, and I found it by trial and error rather than by any principle, which I'm not thrilled about.
systems biology
Yazdani et al. apply the inverse setup to biological ODE systems, recovering rate parameters in gene regulatory networks and metabolic pathways from noisy time series.
The result that got my attention is how little data it needs. The ODE constrains the space of possible trajectories so hard that a handful of observations pins down the parameters. Something purely data-driven would need orders of magnitude more, and would still have no reason to respect conservation.
I reproduced their main result on a smaller enzyme kinetics model, then started making the observations worse on purpose, which is where I still am.
what the GPU is for
Not what I assumed. These networks are tiny, a few thousand parameters, and nothing is memory-bound. The bottleneck is the optimizer. L-BFGS converges better than Adam on most of these problems and parallelizes badly; Adam is fast per step and needs a lot of steps.
What 96GB buys is running thirty experiments at once. I queue a sweep over learning rate, width, depth, sampling strategy, and weighting scheme, and read the results in the morning. That changes what questions you're willing to ask, more than it changes any single run.
A few things the sweeps have been consistent about. Adaptive loss weighting, the kind that rebalances terms by gradient magnitude during training (Wang et al., 2021), beat fixed weights on essentially everything I tried, and the gap widened with more competing loss terms. Fourier feature embeddings fixed the high-frequency problems, where a plain MLP learns the smooth part of the solution quickly and then stalls on the oscillatory part. And collocation sampling mattered more than I would have guessed: uniform random is a fine default, but concentrating points where the residual is large converges faster on anything with a sharp feature.
where this is going
I'm trying to run the systems biology inverse problem on data that looks like real data. Irregular sampling, gaps, correlated noise instead of independent Gaussian. My guess is that the standard method degrades badly and that a better noise model plus putting collocation points near the actual observations recovers a lot of it, but that's a guess I haven't earned yet.
The pattern I keep hitting in the literature is that results are on clean synthetic data. Which is reasonable, that's how you isolate a method. It also means the question of whether any of this survives contact with a real measurement is mostly still open.
Six months in, I think PINNs are narrower than the enthusiasm suggests and genuinely good inside that range: known physics, not much data, and a need to be able to say why the answer looks like it does. Outside it, a solver or a plain neural network usually wins. Figuring out where that boundary sits has taught me more than any individual implementation.