Newsletter #7: Advanced Prompt Engineering
Learning Zero-shot, Few-shot and Chain-of-thought prompts
In the last post, we looked at some of the basic prompting types. In this post, we take a look at 3 advanced types of prompts.
Zero-Shot Prompting
As the name suggests, zero-shot prompting is using the model to directly reason or analyse a given input. This means you do not provide any example in the input of what the desired output should be like.
For example, if we want to generate a poem about winter, we can give the model a prompt like "Explain AI." The model can then generate a poem based on its pre-existing knowledge about winter.
The model generates an output based on its pre-existing knowledge about the task at hand. This is useful when there is no specific training data available for a particular task.
Few-shot prompting
When zero-shot prompting fails, few-shot prompting can be used to improve the output. This was proposed by Brown et. al. and it involves giving a few examples of what the desired output should look like.
Here is an example given in the paper:
Prompt: A "whatpu" is a small, furry animal native to Tanzania. An example of a sentence that uses the word whatpu is: We were travelling in Africa and we saw these very cute whatpus.
To do a "farduddle" means to jump up and down really fast. An example of a sentence that uses the word farduddle is:
Response: One day when I was playing tag with my little sister, she got really excited and she started doing these crazy farduddles.
Here’s the prompt fed into ChatGPT
Even though the words are not part of the training data, few-shot prompting helped the LLM to reason about how the word should be used based on the context in which it appeared.
Chain-of-thought prompting
CoT prompting was introduced by Wei et. al. It involves prompting a complex series of steps to get to the desired output. Using the logic followed in the prompt, the LLM is able to solve similar problems.
For example,
Prompt:
Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can have 3 tennis balls. How many tennis balls does he have now?
Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.
The cafeteria had 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have?
Output:
The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9.
In the prompt, we provide not only the correct answer but also how we got to it. Using similar reasoning, the LLM will respond to the next question. The early versions of GPT-3 used to fail to solve such problems without explicit CoT.
CoT helps to make the LLM model reason better on various tasks such as arithmetic reasoning, common sense reasoning, instruction and planning, etc.
One note here, the authors in the paper have mentioned,
“Through experiments on arithmetic, symbolic, and commonsense reasoning, we find that chain-of-thought reasoning is an emergent property of model scale that allows sufficiently large language models to perform reasoning tasks that otherwise have flat scaling curves.“
Hence CoT might not work on smaller LLMs.
Conclusion
Advanced prompting techniques like zero-shot prompting, few-shot prompting, and chain-of-thought prompting are powerful tools in the field of machine learning. These techniques enable models to generate outputs for tasks that they have never seen before and to generate complex outputs that require multiple steps.
Learning these will help us to use LLMs efficiently and get the most out of them.
📖Resources
That’s it for this issue. I hope you found this article interesting. Until next time!
Let’s connect :)