For better or worse, "LLM" or "generative ai" has become roughly synonymous with the current wave of ML.

I know very little about ChatGPT, but Waymo is using an LLM: "Powered by Gemini, a multimodal large language model developed by Google, EMMA employs a unified, end-to-end trained model to generate future trajectories for autonomous vehicles directly from sensor data." (https://waymo.com/blog/2024/10/introducing-emma)

Waymo uses reinforcement learning (what it was before LLMs) (TD3+BC according to one of their blogs)

Emma is something they tried, but further down the article they explain why they don't use it as such yet.

Yep. It's an interesting experiment and really stretched my understanding of what an LLM is and can do.

Huh. I mean it makes sense to train end-to-end on all the interrelated tasks involved in driving but putting a whole-ass language model in the middle of that seems like a stunt. I wonder if it does better than like, any random transformer not trained on language first? Still, I hadn't heard that so I guess I was wrong about that one

No, you were right, this appears to be just research on how applicable LLMs could be to the space. They talk about the improvements their LLM makes, especially in being multimodal vs training multiple independent models, but also the limitations that appear to prevent it from being useable as it is. Maybe some form if it will be used some day (it does seem like it would be useful to have semantic understanding of the world integrated into the system), but at least as of when this was published, it's not actually used.

Vision Language Models are absolutely being trialed for self-driving

https://wayve.ai/thinking/lingo-2-driving-with-language/

Okay so because of the ambiguity of the other reply I'm just gonna say, I don't think we should be surprised that someone is trying to use LLMs to do basically anything. That's basically what prints funding money right now, so long as you're the kind of company or guy the VCs or whoever will believe in. The signal here is "does it do something to appreciably advance the state of the art over previous methods"?

Seems to be the case to me, reading this and waymo's attempts. There's paper on EMMA here - https://arxiv.org/abs/2410.23262

And there are state of the art weather prediction transformers.

https://arxiv.org/abs/2312.03876

Yeah so like, this is a cool result, and it uses a transformer architecture. I actually do think that it's fair to say that transformers have proven widely useful, especially in tasks that look like sequence modeling. It's a step change akin to the now-pervasive use of convolutional neural networks that started in the 2010s, and is deeply significant of course. This is also really different from "this is an LLM"

The reason I want to specifically harp on this is because a lot of people are selling this narrative where "AI is becoming superintelligent" or whatever by making an amorphous blob out of a bunch of separate advances that use machine learning techniques. This has been happening for a while, is a great thing for science, and it's clear that machine learning methods are here to stay in science. I'm a machine learning researcher. I've understood, celebrated, and tried to help with this as best I can manage over the last 9 years of my life. And it's been going on for a lot longer than the general public has been in this AI hype wave. The entire modern field of bioinformatics is arguably built on the backbone of machine learning, and has been since before I went to grad school.

This is really different from "We fed everything into a language model and now it's superintelligent and is making scientific advances all by itself" or even "scientists just ask chatGPT shit and it figures it out for them". The breathless tech press really makes it sound like anything that happens in AI research, which increasingly includes the entire usage of ML toolkits in the sciences (Which is pervasive, and expectedly so! ML is an extension of statistics and statistics has been the basis of science for like a century) is just some amorphous force called "AI" that's suddenly gained this aggregate body of competency. Imagine if we anthropomorphized statistics that way. Or Math for that matter. This kind of narrative gives me the overall impression that this is not being talked about honestly, and it's clear that this is profitable to do. I don't have to use charged words like "con" or "fraud" to think this deceptive framing is not a great thing

What you're saying does happen to some degree and in this instance, if i had linked some advance with a diffusion model then i would get it but about the only difference between this and chatgpt is the data it's been trained on. If Open AI cared, the next version of GPT could be a State of the Art weather predictor.

I mean by the same logic the only difference between a diffusion model and a VLM is that you put the spatial transformer on the other end.

Yes, one of the powerful things about every kind of neural network is that they're a very general class of function approximator. That we can use a similar toolkit of techniques to tackle a wide variety of problems is very cool and useful. Again, the analogy to statistical models is telling. You can model a lot of phenomena decently well with gaussian distributions. Should we report this as "Normal distribution makes yet another groundbreaking discovery!"? Probably this wouldn't have the same impact, because people aren't being sold sci-fi stories about an anthropomorphized bell curve. People who are using LLMs already think of "AI" as a thinking thing they're talking to, because they have been encouraged to do that by marketing. Attributing these discoveries made by scientists using this method to "AI" in the same way that we attribute answers produced by chatGPT to "AI" is a deliberate and misleading conflation

>I mean by the same logic the only difference between a diffusion model and a VLM is that you put the spatial transformer on the other end.

Maybe if that was the only different but it's not. There are diffusion models that have nothing to do with transformers or attention or anything like that and where using them for arbitrary sequence prediction is either not possible or highly non-trivial.

Yes, All Neural Network architectures are function approximators but that doesn't they excel equally for all tasks or that you can even use them for anything other than a single task. This era of the transformer where you can simply use a single architecture for NLP, Computer Vision, Robotics, even reinforcement learning is a very new one. Literally anything a bog standard transformer can do is anything GPT can do if Open AI wished.

Like i said, i don't disagree with your broader point. I just don't think this is an instance of it.

It's clear you're missing what point it is that I'm making from these responses, but I'm unsure how to explain it better and you're not really giving me much to work with in terms of seeming to engage with the substance of it, so I think we gotta leave this an impasse for now