Token metadata
How mono-dimensional is a token in the model weights of an LLM? Is all of the data in each node only determining words before and after, to determine what is likely to be said next? I wonder if we can embed richer meta-data in each token (ie: definitions, encyclopedia entries, quality scores established by other LLMs). This would mean that during chain-of-thought reasoning, it would be able to use symbolic logic in addition to inference.
Separate thought: for AI to write really well, it needs the time to go through recursion loops. These loops should be legible to the writer, and it should be interactive. The AI will think for 30-60 minutes, and you’re reading and responding to thoughts to help reframe the question.