Big Brother is always looking.
LLMs have made tremendous strides in recent years, with models like ChatGPT, OpenAI, and Google’s Gemini pushing the boundaries of what AI can do. However, there are signs that the current approach to building LLMs may be reaching its limits.
Continue reading “The Evolution of AI: From Large Language Models to Better Models”
Analytics India reports that according to the GPT franchise’s release cycle, the fourth generation is imminent, if not overdue. Last year, Sam Altman, the CEO of OpenAI, discussed the upcoming GPT-4 release during a Q&A session at the AC10 online meetup.
Analytics India reports that according to the GPT franchise’s release cycle, the fourth generation is imminent, if not overdue. Last year, Sam Altman, the CEO of OpenAI, discussed the upcoming GPT-4 release during a Q&A session at the AC10 online meetup. The release is most likely scheduled for July-August of this year. However, OpenAI has kept the release date under wraps, and no definitive information is available in the public domain. GPT-4, on the other hand, will not have 100 trillion parameters.
GPT-3, which will be available in May 2020, has 175 billion parameters. Deep learning is used in the third generation of the GPT-n series to produce human-like text. Microsoft licensed the exclusive use of GPT-3 on September 22, 2020. We’ve compiled a list of GPT-4 improvements based on available information and Sam Altman’s comments during the Q&A session.
Apparently, size doesn’t matter.
Large language models like GPT-3 have achieved outstanding results without much model parameter updating. Though GPT-4 is most likely to be bigger than GPT-3 in terms of parameters, Sam Altman has clarified that size won’t be the differentiator for the next generation of OpenAI’s autoregressive language model. The parameter figures are likely to fall between GPT-3 and Gopher; between 175 billion-280 billion.
Future deep learning will be multimodal. Our multisensory brains reflect our multimodal reality. Only one mode of perception at a time significantly limits AI navigation and comprehension. GPT-4 could be a text-only model to push language models to their boundaries before going on to multimodal AI.
Sparse models that process diverse inputs using conditional computing in different regions of the model have been successful. Without incurring substantial computing expenses, such models can quickly scale above 1 trillion parameters. On very large models, however, the benefits of MoE techniques start to fade. GPT-4 will be a dense model, like GPT-2 and GPT-3. To put it another way, any input will be processed using all parameters.
If GPT-4 is larger than GPT-3, the amount of training tokens necessary to be compute-optimal (according to DeepMind’s findings) might be approximately 5 trillion, which is a factor of a billion more than current datasets. The number of FLOPs needed to train the model with low training loss would be 10–20 times that of GPT-3. Altman stated in the Q&A that GPT-4 would necessitate more compute than GPT-3. OpenAI will prioritize variable optimization over model scaling.
A helpful AGI is OpenAI’s north star. The InstructGPT models, which are trained with people in the loop, are expected to be used by OpenAI. Using strategies gained through their alignment research, InstructGPT was deployed as the default language model on OpenAI’s API and is considerably better at following user intents than GPT-3 while also making them more true and less toxic. However, only OpenAI staff and English-speaking labellers were aligned. In comparison to GPT-3, GPT-4 is more likely to be human-like.
The Wall Street Journal a few years ago ran an article about Swintec, one of the last surviving typewriter companies in the U.S. (Please see the WSJ article for more information.)
The Wall Street Journal a few years ago ran an article about Swintec, one of the last surviving typewriter companies in the U.S. (Please see the WSJ article for more information.)
Edward Michael, who started the company in 1985, is quoted in the article as saying, “We’re typewriters. This is our specialty. This is what we know.”
Down to about 10 employees now from about 85 during the boom years, Swintec continues to sell typewriters at a click-clackety pace: between 3,000 to 5,000 units a year, mostly to universities, senior centers, and prisons. Yes, typewriters are quite popular behind bars — especially now that Swintec came upon the novel idea of a clear typewriter designed to prevent the smuggling of contraband.
From our point of view, typewriters aren’t going anywhere. After all, the vacuum cleaner, as Mr. Michael points out in the above video, failed to replace the broom. Nor did the typewriter replace pens and pencils.
The Kindle Fire is burning up the tablet competition — on the Android side.
Amazon.com Inc.’s tablet computer is catching fire on in a big way, having grabbed 54.4% of the Android tablet market during February, the fourth month it was on the market, according to new data from comScore Inc. That represented almost double of the Fire’s Android market share since December.
The Kindle Fire is in warp drive — far outpacing Samsung’s Galaxy Tab (15.4% of Android), Motorola Xoom (7%), the Asus Transformer (6.3%) and others by Dell, Lenovo and Sony.
But the tablet market leader remains Apple’s iPad, which, according to the market research firm IDC, owned about 55% of the tablet market at the end of 2011. Android tablets accounted for about 45%. This is all great news for Amazon, meaning about 30% of tablets currently shipping are Kindle Fires, making the Fire a close second to the iPad.
Despite efforts by Apple, dismissing the Fire is increasingly difficult to do so. In February, Apple Chief Executive Tim Cook dismissed tablets like the Kindle Fire as inferior:
A cheap product might sell some units … But then [consumers] get it home and use it and the joy is gone. And the joy is gone every day that they use it and they wind up not using it anymore.
No one knows for sure how much use the Kinde Fire is getting, but consumers certainly are buying it.
