When you train an AI model with code, it gets better at reasoning. For example, Mark Zuckerberg revealed that teaching Meta's Llama model with code significantly improved its reasoning abilities. This enabled the smaller Llama 3 model to outperform larger models like Llama 2 in logical and mathematical reasoning capabilities. This is no coincidence. Users think in terms of data, using tools like calculators and spreadsheets to directly manipulate it. Developers, however, think in terms of metadata, writing code that manipulates variables, which in turn manipulates data at runtime. This ability to abstract data as metadata is a step change in intelligence between users and developers. AI improves its intelligence in the same way. One approach, called program induction (where AI writes code to solve problems), allows AI to generate code for complex problems like frontier math. If understanding and generating code can make AI more intelligent, could understanding meta-metadata such as compilers make it even smarter? That's what we will explore in this post.


Intelligence in Math

Imagine you are a maintenance worker asked to hang a painting in an art gallery at 12 feet from the floor. You have a ladder whose length is 13 feet. How would you place the ladder to reach 12 feet? You might first get the help of another person to hold one end of a measuring tape at the bottom of the wall while you climb the ladder with the other end of the tape to mark 12 feet. Then you would place the ladder parallel to the wall and drag it until it hits the 12-foot mark. This approach of working directly with data represents the basic level of intelligence.

If you are an engineer, you would apply the Pythagorean theorem to find the solution in one step: placing the base of the ladder 5 feet away from the wall ensures it hits the wall at exactly 12 feet. Knowing an algebraic equation (i.e., √c² - a²) and applying it to the data is a higher level of intelligence.

Now, imagine you are Pythagoras, and the theorem has not been invented yet. Observing patterns in right-angled triangles (perhaps noticing the 3-4-5 triangle used in Egyptian architecture) you could hypothesize that a² + b² = c². By generalizing and verifying this pattern across different usecases, you formalize it as a theorem. This progression (from data to metadata to the invention of metadata) illustrates how reasoning evolves from worker to engineer to inventor. Intelligence is the ability to abstract and reason at multiple levels.


Intelligence in AI


The first version of ChatGPT was comically bad at math. Even the recent version, ChatGPT 4, is only as good as the maintenance worker. It answers that the ladder has to be placed 3 feet from the wall. It uses the 4:1 safety guideline to minimize ladder slippage. It relies on transduction, which means simply memorizing and recalling data without deep understanding.

Next-generation models like o1 can memorize and recall metadata, such as the Pythagorean theorem, and apply it to data. It correctly calculates that the ladder needs to be exactly 5 feet away using the theorem, much like an engineer would. This ability aligns with program induction, where the AI generates code to solve complex problems.

Future models could resolve enigmas like the Collatz conjecture, a math puzzle where applying two simple rules (if even, divide by 2; if odd, multiply by 3 and add 1) always leads to 1. Like Pythagoras uncovering hidden geometric relationships, proving this for every number would expose universal patterns governing seemingly chaotic systems.


Intelligence in coding

The idea of moving up the metadata stack to become more intelligent isn’t new to developers. Workers use calculators (data tools) to solve specific problems like finding ladder distances. Developers abstract these tasks into code (metadata), using programming languages like python to automate such problems for millions of workers around the world. At the highest level, language creators build meta-tools like compilers that compile and execute code that helps millions of developers around the world. They use meta-metadata tools, like Python's PEG (Parsing Expression Grammar) or ANTLR (a parser generator) to create new programming languages and compilers.
This is true for game engines like AlphaGo and AlphaZero. DeepMind researchers trained AlphaGo on data from Go games. But, AlphaZero trained on all games to become better at Go and it automatically became better at Chess. Now they are exploring if the next version of AlphaZero can invent a new game. This climbing up the ladder of abstraction is how intelligence evolves:
  1. Understanding data from what you read and recalling them is memorization. This is what GPT3 does.
  2. Understanding metadata such as Pythagorean theorem and applying it to data such as ladder problem is reasoning. This is what o1 does.
  3. Understanding meta-metadata such as ANTLR grammar and creating a new programming language is invention. Thats what AGI should do.


Summary

OpenAI recently introduced a levels framework describing how AI evolves from chatbot to reasoning to invention. While it provides a useful measure of capability, it does not fully explain the underlying mechanisms driving the evolution of intelligence. The Llama example shows that teaching AI with code and asking it to generate code improves its intelligence. This mirrors how developers become more intelligent than users by abstracting logic as metadata instead of data. But an inventor thinks in meta-metadata. They recognize patterns and derive new metadata, just as Pythagoras formulated a² + b² = c² . Pythagoras didn’t stop at math; he reimagined music through ratios (meta-metadata), showing how patterns transcend disciplines.
This ability to abstract patterns across disciplines (like linking math ratios to musical harmonies) demonstrates the next frontier of intelligence. There might be 2 ways to get there:
  1. Training the model with metadata of multiple game rules or programming languages or theorems and scaling up to see if figures the meta-metadata by itself and create a new metadata.
  2. Or, teach it meta-metadata (e.g., compilers, cross-domain patterns) and see if it can create a new metadata such as a new game or programming language or theorem.
Maybe it is a combination of both, because intelligence isn’t just about solving problems. It’s about discovering the hidden rules that create them.