- Welcome to Languish.org.
Recent posts
#21
Off the Record / Re: The AI dooooooom thread
Last post by Baron von Schtinkenbutt - Today at 11:01:08 AMGoogle did move straight from developing the transformer architecture to developing a language model with it. However, they took a fundamentally different approach. Google developed BERT, which is an encoder-only transformer model. OpenAI developed what is arguably a higher-level model architecture based on a decoder-only transformer model.
Google's approach created a language model that was suitable for creating inputs to non-language models. BERT and its derivatives became heavily used as the first stage in ML systems to generate various forms of content embeddings. These embeddings are also useful on their own for search and retrieval systems, which is why Google went this direction. The transformer architecture itself came out of an effort to develop better models for sequence-to-sequence conversion than what was then available with LSTM models fed by simple word embeddings like Word2Vec.
OpenAI intended to go in a different direction and create a language model that could generate language rather than encoded sequences. They introduced the concept of generative pre-training (GPT), which is what gives these models their "knowledge". Basically, an architecture designed to recreate language that looked like what it had been trained on. This approach is not very useful for search and retrieval, but is useful if you want to build a chatbot that uses the "knowledge" encoded in the model to to retrieval and synthesis.
As architectures developed it turned out the GPT architecture had so-called emergent behaviors that made the base models built this way useful for general tasks, provided the right tooling and scaffolding was built around it. Google came around to the generative model party late partly because the value to them wasn't clear until OpenAI rolled out a complete chatbot product using it. Plus, as Joan said, the whole "we're just a research effort" bullshit.
Google's approach created a language model that was suitable for creating inputs to non-language models. BERT and its derivatives became heavily used as the first stage in ML systems to generate various forms of content embeddings. These embeddings are also useful on their own for search and retrieval systems, which is why Google went this direction. The transformer architecture itself came out of an effort to develop better models for sequence-to-sequence conversion than what was then available with LSTM models fed by simple word embeddings like Word2Vec.
OpenAI intended to go in a different direction and create a language model that could generate language rather than encoded sequences. They introduced the concept of generative pre-training (GPT), which is what gives these models their "knowledge". Basically, an architecture designed to recreate language that looked like what it had been trained on. This approach is not very useful for search and retrieval, but is useful if you want to build a chatbot that uses the "knowledge" encoded in the model to to retrieval and synthesis.
As architectures developed it turned out the GPT architecture had so-called emergent behaviors that made the base models built this way useful for general tasks, provided the right tooling and scaffolding was built around it. Google came around to the generative model party late partly because the value to them wasn't clear until OpenAI rolled out a complete chatbot product using it. Plus, as Joan said, the whole "we're just a research effort" bullshit.
#22
Off the Record / Re: The AI dooooooom thread
Last post by crazy canuck - Today at 10:32:43 AMQuote from: DGuller on Today at 10:09:11 AMQuote from: The Minsky Moment on Today at 09:49:50 AMI think the explanation for Google being beat to market is off the mark. In history plenty of companies failed to capitalize on their own inventions, or even appreciate their potential, for reasons other than their focus on quality. I think the far more likely explanation is that Google, being a mature large public company, is just naturally far less nimble than a motivated privately-held startup. Companies like that have way too many stakeholder alignment meetings, risk committee working groups, and quarterly earning targets, to move fast, at least until external factors make them move fast.
- Google launched their new Gemini iteration and appears to have overtaken or at least caught up to GPT-5. Open AI insiders leaked a memo from Altman declaring a "code red". Of course, there is nothing unexpected about this development; Google was the pioneer of these industrial scale LLMs (from their 2017 paper) and was only beat to market because they wouldn't release a half-baked product.
It has nothing to do with nimbleness. When GPT was first released, the developers were very clear that it was still in development. But despite that warning people treated it las if it was a reliable tool with all of the catastrophic consequences that have been widely reported.
GPT is still in development and yet people still take it seriously as if it is a reliable tool.
Google's product may be reliable, or it may have the same defects of all LLMs. We shall see.
#23
Off the Record / Re: Russo-Ukrainian War 2014-2...
Last post by Crazy_Ivan80 - Today at 10:30:49 AMRussia will stay solvent as long as China desires
#24
Off the Record / Re: The AI dooooooom thread
Last post by HisMajestyBOB - Today at 10:25:49 AMI would like to get a new computer next year, please don't drive up the cost of components.
#25
Gaming HQ / Re: Europa Universalis V confi...
Last post by The Minsky Moment - Today at 10:23:50 AMQuote from: HVC on Today at 04:32:00 AMPlus ça change
Wake me up around 1.12 . . .
#26
Off the Record / Re: The AI dooooooom thread
Last post by The Minsky Moment - Today at 10:18:52 AMDG I think we are saying the same thing in different ways, with you putting greater emphasis on the downsides of big enterprises (bureaucracy) and me putting greater emphasis on the downsides of startups (willingness to shove out and hype a half-baked beta and fix later on the fly)
But I would add - Google probably thought they were dealing with something else. OpenAI was supposed to be a non-profit, defined by a precautionary ethos, NOT a commercially "motivated" start-up. That turned out to be a mirage, one that even much of OpenAI's own board was unaware of.
But I would add - Google probably thought they were dealing with something else. OpenAI was supposed to be a non-profit, defined by a precautionary ethos, NOT a commercially "motivated" start-up. That turned out to be a mirage, one that even much of OpenAI's own board was unaware of.
#27
Off the Record / Re: The AI dooooooom thread
Last post by DGuller - Today at 10:09:11 AMQuote from: The Minsky Moment on Today at 09:49:50 AMI think the explanation for Google being beat to market is off the mark. In history plenty of companies failed to capitalize on their own inventions, or even appreciate their potential, for reasons other than their focus on quality. I think the far more likely explanation is that Google, being a mature large public company, is just naturally far less nimble than a motivated privately-held startup. Companies like that have way too many stakeholder alignment meetings, risk committee working groups, and quarterly earning targets, to move fast, at least until external factors make them move fast.
- Google launched their new Gemini iteration and appears to have overtaken or at least caught up to GPT-5. Open AI insiders leaked a memo from Altman declaring a "code red". Of course, there is nothing unexpected about this development; Google was the pioneer of these industrial scale LLMs (from their 2017 paper) and was only beat to market because they wouldn't release a half-baked product.
#28
Gaming HQ / Re: Europa Universalis V confi...
Last post by Josephus - Today at 10:07:02 AMQuote from: crazy canuck on Today at 08:29:11 AMJos, with Castile, at the start of the game, delete a bunch of the unneeded fortifications.
That frees up a lot of budget room.
Thanks
I also keep forgetting about the wonderful "Ask for money" diplomatic function.
#29
Off the Record / Re: The AI dooooooom thread
Last post by Syt - Today at 09:59:39 AMI'm so glad I bought a new PC with RAM a few months ago. 
#30
Off the Record / Re: The AI dooooooom thread
Last post by The Minsky Moment - Today at 09:49:50 AMOn the topic:
Two immediate conclusions:
1) The AI investment boom has reached the phase where it is causing significant distortions in the real economy, with likely adverse impact on consumers and the broader tech industry well into 2026.
2) The probability of a nasty crash over the next 6-12 month window is rising.
- Open AI signed huge procurement deals with both Samsung and SK Hynix, the duopoly that effectively dominates DRAM supply. It was reported that the deals account for 40% of global supply.
- DRAM prices have exploded as everyone else scrambles to grab supply. There are immediate knock on effects for retail consumers as anything that uses RAM is going to go way up in price once stockpiles are exhausted. I don't think we can expect a significant supply response, because "Samnix" has to be concerned what would happen if Open AI implodes (see below) and dumps their massive stockpile back onto the market.
- Within the industry, there are increasing questions being raised about LLM scalability and whether the existing strategy of dumping more and more computing power and data is hitting diminishing returns. That is: the incremental cost of improving the models is rising fast due to supply constraints, but the returns to model effectiveness may be diminishing.
- Google launched their new Gemini iteration and appears to have overtaken or at least caught up to GPT-5. Open AI insiders leaked a memo from Altman declaring a "code red". Of course, there is nothing unexpected about this development; Google was the pioneer of these industrial scale LLMs (from their 2017 paper) and was only beat to market because they wouldn't release a half-baked product.
- Google generates massive profits off of its core business; it can afford take a bath on LLMs. OpenAI has no business other than raising VC capital from supercharged hype and grabbing a few shekels here and there by vending the more advanced iteration of GPT-5. The true state of OpenAI finances is unknown because they are private but it is clear that operating losses are large and capital costs are staggering, on a country level scale.
Two immediate conclusions:
1) The AI investment boom has reached the phase where it is causing significant distortions in the real economy, with likely adverse impact on consumers and the broader tech industry well into 2026.
2) The probability of a nasty crash over the next 6-12 month window is rising.