Data Is Not All You Need
I think the "data is all you need" paradigm will be proven wrong in the next 4 years.
My example is, even if you give a 5-year-old the Bible and let them read it cover to cover, they will have read it, but unable to speak about it or understand the full implications. It’s not about more data, it’s enabling deeper understanding of existing data.
And so scaling laws have a limit. Even if you give LLM more time to think, it is still limited by associative connections (it’s effectively just casting and combining more vectors). I’m intuiting that the breakthrough will come from enabling LLMs to inherently make recursive abstractions in chains of thought. After it generates a sentence to itself, it needs the ability to ask: “What is this?” "Is this good?" "Is this (any category within an ontology)?"
What I'm reaching it as it's not about more data, but better labeled data. Early AI was very much about manual labeling and tree construction. It's too vast for humans to do it. We need specialized LLMs to do this abstraction process—to reason over tokens, and then bake back those insights into the token itself, to draw upon during reasoning traces... I think.
This is all pretty fuzzy right now, and limited by my technical understanding, but it could be fun to do some research and properly anchor my "beyond LLMs" ideas. In terms of analyzing essay quality, I could see Essay Architecture evaluations feed directly into model training, but, to what end? I'm all for analysis, but less interested in generation.