Mouth vs Brain

A couple of days ago I read 2 articles about LLM (large language model) and ChatGPT. The first one is How ChatGPT and Other LLMs Work—and Where They Could Go Next on Wired. It mainly discussed the following topic:

  • The main idea behind the LLM: with some user-provided words (usually a quesion), LLM choose the most suitable following without understanding the meaning of the question and the sentence it emits.
  • Technical lineage: neural network to transformer, then to LLM.
  • How human participation promotes the model quality. Here the article introduced the basic idea of RLHF (reinforcement learning on human feedback), the silver bullet of ChatGPT.

The second article How ChatGPT actually works contains far more technical detals. It explains why ChatGPT-4 is superior than its precursors (better alignment), the detailed steps of RLHF, how to evaluate the performance of a LLM, and finally the shortcomings of the methodology used in current model training.

After reading above articles, I tend to believe that LLM nowadays is more like the mouth rather than the brain of AGI (artificial general intelligence), like Blade Runner or agent Smith in The Matrix.
LLM read gigantic amout of human-generated materials, and learn the patterns in them. It mixes and regenerates new texts (instead of ideas) based on the texts (instead of ideas) it learned. However, text is only avatar of idea, not the idea itself.

People argue that there’s no such thing of “idea”, or “intelligence”. These are just imagination of human being. When a system evolves large enough, “intelligence” emerges. According to this interpretation of intelligence, the advent of AGI happens natrually with the increasing of the complexity of the LLM. Well, it’s reasonable. But at least until now, I didn’t see the possibility.

Comments

Post a Comment

Popular posts from this blog

2023: On the Road

There're so many dramas happened in the past 3 years. People say there'll be more in the coming 2023. Basically, I agree with this. But no matter what happens in the physical world, we need to concentrate on something really important and interesting. In the past 2022, my most profound change was, that I started thinking seriously about the future living style. My family and friends mostly live like plants, fixed by our roots, and change ourselves to adapt to the environment. We are where we are, proud of its history and culture, and contained by it. It'll not be the case anymore. In 2022, I learned, read, listened, wrote, and spoke in second languages, especially English and Japanese. They bring me a new world. I also read several books on Chinese and middle east histories. The world before me is fusing as a whole sphere. China and Chinese people are a part, and the most familiar part for me, the starting point of my exploration. It gives me enough confidence as a foreig...
Read more

Yet another advice to kids

Sharon posted some advice to the youth (although she herself is young). She suggested being independent, with no fear of failure, which is excellent advice. I'd like to give some more "concrete" tips to kids or myself in childhood (if I have the chance to tell that me). First things first, the following suggestions can't make you more competitive than your peers or perform better in exams. or win the girl make you fascinated. They make your everyday life more exciting and fancy even if you're not rich or powerful. They're neither expensive (so you can achieve these goals even if not born into a rich family) nor painful. What you need is to know them, and keep doing them. OK, here we go. 1. Find a sport, enjoy it, and be good at it. Sport is a way to talk with your body, with the physical world, express yourself, and communicate with people. 2. Be honest, especially to yourself. Living honestly in Chinese society looks stupid and clumsy. Never mind. Being hones...
Read more

The Joy of Reading in Natural Light

Nowadays I read books in paperbacks much less than ebooks. Not because I don't like reading paper books, but after reading for so many years on screen, my eyes are very sensitive to the brightness of reading materials. It's very hard for me to read characters when the material is not bright enough. Unfortunately, the light in my living room is not bright enough for me. This morning it was cloudy, while in the afternoon it got clear, and the blue sky and white clouds appeared. So I moved the desk near the window. To my surprise, the characters on paper books are much more clear and more comfortable to read than in artificial light. Maybe such a simple fact is not worthy to mention. But to me, a programmer spent too much time with the keyboard and screen, this reminds me of the memories of childhood, reading a book, or writing homework in the yard. The characters blurred at sunset. That's when we were going to have super. The window of my living room is east-forwarding, so to...
Read more