Can you watch yourself thinking?

Consider how you are reading this book, or any book for that matter. You probably go back and forth across a page, and when met with a complicated part, you may ask yourself: What is the gist of this sentence? What is the main idea here? And that familiar part of your mind springs to life, highlighting and summarizing, making decisions, doing the hard work. You can watch it think, go through the usual paces. You probably did that thousands of times. But the other part, that makes requests, can you watch it think? Where is it? What is it? It is there. Inaccessible.

Clearly, these parts, one working and one directing, are related to each other. They speak the same language, after all. They are almost the same, but different. What if we could figure out how they are related? Then, we could work backward from what we observe to what must be there.

Consider the fundamental constraints on energy, on space, on time that we have. You don’t have infinite energy. Your biological brain is probably about 1.4 liters in volume and uses 20 watts of power. So every one of your thoughts costs calories, but the world is infinite. There are more books than you could ever read in your limited lifetime, as simple as that. So you must select, by necessity. Everyone does that, but not everyone does it the same. What makes our filters different? What do we filter for exactly?

Language models can do work our minds can do. Is this intelligence? What kind of intelligence? How does it compare to mine and yours? Not every intelligence is equal. Say, a toad operates in a separate level of comprehension compared to the insects it eats: a fly has no idea what hit it. What are these levels? That’s not about brain size; there are countless species with larger brains than humans, yet they are not that smart. So there’s something else that makes us smarter. What is the structural reason we are as smart?

But first, we have to establish the ground rules. Surprisingly, there are only a few rules.