Demystifying Prompt Engineering: A Beginner’s Guide
Introduction
Deeplearning.ai released a free course in April 2023 in partnership with OpenAI called “ChatGPT Prompt Engineering for Developers”. It is 1.5 hrs short, very hands-on, and introduces us to all the magic that goes behind getting well written responses to user questions, from Large Language Models (LLMs) like ChatGPT.
In this blog, we will cover the course summary with some high level topics, takeaways and apply those concepts with some prompt examples. Towards the end, get the Colab notebook to try these examples yourself!
What is Prompt Engineering?
A prompt is an instruction provided to a natural language model along with input. Prompt engineering is a skill where engineers help improve the accuracy and efficiency of natural language models by fine-tuning information provided to the model and structure how you request information from it.
Principles for Prompt Engineering
The course[1] highlights two guiding principles for prompt engineering.
- Give the LLM clear and specific instructions. This doesn’t have to be short or concise, but needs to provide clarity and context. Use delimiters to help separate instruction and input which helps avoid prompt injection. Ask for structured outputs and instruct the LLM to fail fast when assumptions aren’t satisfied. A Tip: Give the LLM a few successful results to learn from.
- Give the LLM time to think. Break down instructions, specify steps required to complete the task, and instruct the LLM to work out a solution before producing an output or result for the user’s input.
Inspired by this reddit thread, I asked ChatGPT what writing effective prompts look like and this is what it had to say.
Limitations
Engineers need to understand the limitations of LLMs that directly impact what makes a prompt effective and reliable.
One of the major limitations of LLMs is Hallucinations. This term was first coined by Google AI in the paper Hallucinations in Neural Machine Translation — Agarwal et al. (ICLR 2018). Often when the LLM doesn’t find enough specifications, or “time to think”, the model may give a very realistic response to something that’s not true and this is called hallucinations. We can reduce hallucinations by asking it to find any relevant information/quotes from the input text and use them to find relevance.
Hallucinations : Find information for an imaginary product
Colgate is a real company that makes toothbrushes! However this toothbrush does not exist. Interestingly, ChatGPT comes up with a very believable summary.
prompt = f"""Tell me about Gum Health Optic Pro battery powered \
toothbrush by Colgate"""
response = get_completion(prompt)
print(response)
Wikipedia quotes 5 Facebook founders, and when asked to name seven founders, it does report two additional names.
# https://en.wikipedia.org/wiki/Facebook
prompt = f"""Name seven founders of Facebook"""
response = get_completion(prompt)
print(response)
Count the number of words in a text
LLMs like ChatGPT find counting hard. They essentially need specific steps to do that. But the good thing is, parts of these restrictions can be applied at the Application layer of the system using this prompt and LLM. Checkout the discussion on the OpenAI Community Forum on this topic.
text = f"""You should express what you want a model to do by \
providing instructions that are as clear and specific as you \
can possibly make them."""
prompt = f"""Count the number of words in this text.
```{text}```"""
response = get_completion(prompt)
print("Number of words in the text: ", len(text.split(" ")))
print("Model's response: " + response)
Summarize a text to the exact number of words
Similarly, it can find it hard to adhere to an exact word count limit. It does however manage to work with a range. We will see more examples of range text word limits in applications.
text = f"""You should express what you want a model to do by \
providing instructions that are as clear and specific as you \
can possibly make them."""
prompt = f"""
Summarize this text in exactly 20 words
```{text}```
"""
response = get_completion(prompt)
print("Response: " + response)
print("Number of words in the response: ", len(response.split(" ")))
Process and Applications
Prompt engineering is an iterative process. There is no one perfect prompt, or one prompt-fits-all solution to making the most of an LLM like ChatGPT. Thus developing a process is important to help diversify the application. Experiment and Iterate through different prompts by giving specifications, updating instructions and setting limits and definitions to get the LLM to focus on what’s important.
The following are examples of how, with efficient prompt engineering and different prompts, one model can be used for different use cases.
Summarization & Information Extraction
LLMs can be used to summarize large paragraphs of text for different purposes. Some ways to make the most of a summary are by
- Instructing the LLM to set word/character or sentence limits
- Directing it to focus on one or two key topics for the summary
- Extraction to capture the theme or intent behind the text
This can be applied to summarizing customer reviews for a Product for different departments like the Shipping & Packaging Department, Pricing Department and Design Department.
Inference, Text Classification, Sentiment Analysis
LLMs can be used to make inferences from text. With an effective prompt, one Model and one API can be leveraged to infer different ideas like emotion, topics from a news article or product’s positive or negative orientation from a review text. This helps turn around speed for new features and application development. Some inference applications include Sentiment extraction, Topic Analysis, Product Review orientation, Brand Analysis, Customer Feedback Analysis, etc.
Here’s an example for a sentiment analysis use-case. This prompt can be easily extended to different kinds of reviews.
tweet = f"""@AmericanAir just landed - 3 hours Late Flight - \
and now we need to wait TWENTY MORE MINUTES for a gate! \
I have patience but none for incompetence."""
prompt = f"""
Identify the following items from the tweet:
- Sentiment (positive or negative)
- Is the author of the tweet expressing anger? (true or false)
- Service/Product they are talking about
- Company associated with the service or product
The review is delimited with triple backticks. \
Format your response as a JSON object with \
"Sentiment", "Anger", "Service/Product" and "Brand" as the keys.
If the information isn't present, use "unknown" \
as the value.
Make your response as short as possible.
Format the Anger value as a boolean.
Review text: '''{tweet}'''"""
response = get_completion(prompt)
Transforming
One of the key abilities of LLMs is text transformations. These are applying features like sentiment, tone, introducing translations, formal or informal writing, spell check, proof-reading etc to user input.
Here’s an example of converting a Leader’s frustration with an employee into empathetic and constructive feedback.
text = f""" I am absolutely upset with Derek's performance this quarter. \
He was supposed to deliver a project by the 25th of Jan and now it is 3 months past. \
We're still in the design phase. This is either poor estimation or lack of determination I think. \
Derek needs to step up and either ask for help, accept help or communicate delays and blockers early on."""
prompt = f"""You are a member of product leadership. Your reportee has given you some employee feedback as \
delimited by triple back quotes. \
Your task is to respond to the reportee with empathetic and constructive feedback on what \
they can do better (if any)
and convert the reportee's feedback into constructive feedback for the employee \
to be given to the employee which the \
employee will receive. If there is positive feedback, keep as is. \
Format the response in JSON with \
keys writer_suggestion, employee_feedback
```{text}```"""
response = get_completion(prompt)
print(response)
Expanding
LLMs can be used to expand from a shorter piece of text. Using ChatGPT’s API parameter — temperature, randomness can be introduced to the response from the LLM. Where 0 temperature results in a deterministic response for a prompt and input, 1 results in high randomness. This feature can be used to treat the LLM like a brainstorm partner, an AI customer agent, to write email responses etc.
Code Generation
ChatGPT can produce code in different languages. There are a vast number of code-generation tasks you can perform with clever prompts.
Reasoning
LLMs can problem solve by breaking down the tasks into steps. The following example gets the LLM to solve the problem for itself before it applies reasoning to the question. [7]
prompt = """The odd numbers in this group add up to an even number: \
15, 32, 5, 13, 82, 7, 1.
Solve by breaking the problem into steps. First, identify the odd numbers, \
add them, and indicate whether the result is odd or even. Then verify the \
claim in the statement as true or false"""
response = get_completion(prompt)
print(response)
Conclusion
The course overall brings more clarity to this widely popular field of Prompt Engineering. There’s a takeaway for anyone even remotely curious about how ChatGPT can be extended to different applications. Here’s the Prompt Power Hour Colab notebook with several more examples to try out for yourself!
References
Some prompt examples and text were picked from these resources, for this summary. There are several other resources relevant to LLMs and ChatGPT and here’s a curated list of what we found interesting and valuable.
- Reddit thread on ChatGPT’s opinion about effective prompt engineering
- Deeplearning.ai’s Course — ChatGPT Prompt Engineering for Developers
- Github Prompt Examples : InstructGPT
- The famous paper on transformers Attention is All you Need — Vaswani et al.(2017)
- Movie Review — Book Club: The Next Chapter
- ChatGPT Data Science Prompts
- Dair.ai Prompt Engineering Guide
- Survey Paper on Prompting Methods called Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing — Liu et al(2021)
- Chip Huyen’s blog post on Building LLM Applications for Production
- Awesome Prompt Engineering on Github
- @omarsar0’s curated list of resources for prompt engineering. Tweet
- Learning with Hasan — Prompt Engineering Guide
