Newsletter #16: PEARL — A LLM brain for large texts
Planning and execution of queries on large documents using LLMs
More and more system designs are being proposed where the LLMs are used not only for text analysis but also as a pseudo brain. A few weeks ago, I wrote about one such approach where the LLMs were modularized as the eye, the nerve and the brain.
In this post, we take a look at another model that works on a similar principle of using a sophisticated Chain-Of-Thought (COT) prompting approach to plan and execute tasks over long documents.
The model proposes 3 steps: Action Mining, Plan Generation and Plan Execution.
Let’s explore how it works.
Model Design
Action Mining
The model generates all possible steps that might be required for the task. The output of this step is a function definition and it’s a description in natural language.
The module gives a few seed actions along with the task to get task-specific results.
As seen above, the output is a function definition with parameters and a function description in natural language.
A full pass over the training data is required to generate the set of all actions.
Plan Generation
This is the framework for producing the answer. It will choose the actions mined in the previous step. The input arguments to action could be the input document, a string or output from the previous step.
Plan execution
The actions from step 2 are executed in this step. The goal is to execute actions in the right order as well as handle errors where necessary.
A plan parser is used prior to plan execution for error handling and checking the quality and feasibility of the generated plan. In case of an error, the error message is fed back to the LLM and a new plan is generated.
Limitations
Even though this is a novel approach to using LLMs of generating misinformation or hallucination. There were some cases in the evaluation where the plan generated was incorrect or a good plan was executed in an erroneous way. In addition to this, even though the model is designed to work on large documents, it is bottlenecked by the max context size of the LLM.
Conclusion
Auto planners using LLMs are in the nascent stage of development. As LLMs get more powerful, better architecture can be designed to build an efficient planner of difficult tasks. This will have huge applications in education since once the model is tuned to a particular student, a smart planner can plan an action map suited to each individual student.
I’m sure we are in for exciting times ahead!
That’s it for this issue. I hope you found this article interesting. Until next time!
📖Resources
Let’s connect :)
