Automatic Prompting
Learning Objectives
- You know of the possibility to automatically generate and improve prompts.
As prompt engineering is largely exploratory and manual, there have also been efforts to automate the task. Automatic prompt engineering can involve generating prompts automatically, optimizing prompts, or both.
As an example, the Automatic Prompt Engineer takes an initial prompt, and then creates a set of candidate prompts using an LLM. Then, given a set of reference outputs and a criteria, the system evaluates the candidate prompts and selects the prompt that best matches the reference outputs and the criteria. Similarly, Optimization by Prompting involves creating candidate prompts by providing an LLM with prior prompts and their scores (calculated with some criteria), and then asking an LLM to create new a prompt that achieves a higher score. This can be done iteratively.
There are also software solutions for the task. As an example AutoPrompt is a Python tool that can be used to improve prompts. The system is given the task and an initial prompt, after which the user can provide feedback on the generated outputs. The system then uses the feedback to generate new prompts, and the process is repeated until convergence.
There are also commercial solutions for the task, including PromptPerfect and OctiAI. LLM providers such as Anthropic have also started to provide tools for prompt generation.