AI Tidbits

[ergophobe]

AI Tidbits

------- Another leader in the field has regrets

'The Godfather of A.I.' Leaves Google and Warns of Danger Ahead

"I console myself with the normal excuse: If I hadn't done it, somebody else would have," Dr. Hinton said

https://www.nytimes.com/2023/05/01/technology/ai-google-chatbot-engineer-quits-hinton.html


------ It's gone circular -----------

GOOGLE'S BARD AND BING AI ALREADY CITING EACH OTHER IN NEURAL HALL OF MIRRORS
WE HATE THIS TIMELINE.
https://futurism.com/the-byte/bard-bing-neural-hall-of-mirrors

Yes, that's correct - Bard generated misinformation based on a joke on Reddit and Bing used that to spread the misinformation.

----------- Analog chips and AI -------------

The Return of Analog computers
https://www.wired.com/story/unbelievable-zombie-comeback-analog-computing/

Interesting in general, but the AI tie-in is that it turns out that analog computers are vastly more efficient for some applications. As one example - the human brain uses 20 watts, while a computer array with a similar number of "synapses" uses megawatts of power.

[ergophobe]

Context Rot May Limit AI Progress
https://www.understandingai.org/p/context-rot-the-emerging-challenge

The article is worth a look if only for the first chart, which shows the incredible progress of AI at programming tasks.

An analogy to the human brain helps to illustrate the problem. As I sit here writing this article, I'm not thinking about what I ate for breakfast in 2019, the acrimonious breakup I had in 2002, or the many episodes of Star Trek I watched in the 1990s. If my brain were constantly thinking about these and thousands of other random topics, I'd be too distracted to write a coherent essay.

But LLMs do get distracted as more tokens are added to their context window — a phenomenon that has been dubbed "context rot."

So when I started to study how large language models work, I was shocked to learn that one of the foundational concepts, the attention mechanism, has terrible scaling properties. Before an LLM generates a new token, it compares the most recent token to every previous token in its context window. This means that an LLM consumes more and more computing power — per token — as its context window grows.

If there are 101 previous tokens, it takes 100 attention operations to generate the next token. If there are 1,001 previous tokens, it takes 1,000 attention operations. And these costs are per token, so a session with 10 times more tokens takes about 100 times more computing power

This is something that has come up in conversations with my nephew who is using AI and working on very large software projects. He has mentioned several times that the context window is the limitation. You simply cannot upload a program with 2,000,000 lines of code into an LLM, but an engineer can hold the whole structure of the program in her head without having to remember every single line, which allows the engineer to figure out what needs to be done next in a way that an LLM cannot. But once the engineer has boiled it down to a problem that has a manageable number of lines in the context window, the LLM can get it done many times faster than the engineer.

The question raised by this article is whether or not, using current methods, it will be possible to scale to very large context windows.

[rcjordan]

> progress of AI at programming tasks

I was going to post that when I Gsearch for perl-ish script help for photo-handler ExifTool, the AI blurb at the top of the serp is cutting out the chase and writing the script. 90%+ of them have worked straight out of the box.

The scripts I'm using look like this:
Prepend exif year to filename   -k -d %Y-%%-c%%f.%%e "-filename<CreateDate"