Relativity Guide to Prompt Engineering for AI

The key to getting AI systems like Harvey AI to generate effective drafts of legal briefs, and solve other problems is to devise a good prompt to get AI to give you the result that you need. In the AI for Legal Basics certification course, Harvey emphasizes that its system will not work well if given ambiguous instructions, and that projects should be divided into discrete tasks. A good prompt will indicate what should not be done, and the prompt should specify which authorities should guide it.

As a new addition to its extensive array of electronic discovery training materials, Relativity has prepared a guide, The Legal Professional’s Guide to Prompt Engineering, which can be downloaded here: https://resources.relativity.com/legal-professionals-guide-prompt-engineering-lp.html

The core idea is to alter the language often used in legal documents to phrasing that provides AI with better guidance about what to generate:

Here are some key takeaways:

1. Having the ability to write well is key. Prompt engineers often have degrees in English, rather than in fields related to technology.

2. Relativity references OpenAI's tips for engineering effective prompts which include:

   1. Use the latest LLM.

   2. Clearly distinguish between the instructions for what the AI system should do and the information it should be reviewing. OpenAI marks text to be analyzed with 3 quotation marks """:

   3. 

      Be specific about the outcome. Give examples of the results that the system should generate. You want to avoid 'zero-shot prompting', which provides instructions without demonstrating the desired result.

      
      

      

3. Much like the tried and true EDRM model, Relativity recommends thinking of prompt engineering as an iterative process. It's necessary to interact with the system to refine the result that it produces.

4. AI can be instructed to indicate its own reasoning. CoT - chain-of-thought prompting is when a prompt tells the system to explain how it is reaching a conclusion. A prompt can, for example ask that the system identify an issue, the relevant rules, and state how the rule applies to the facts, showing how a conclusion is reached.

5. Use role-based prompting: a prompt can specify that a system answer a question as someone working in a specific position would.

6. Contextual prompting is when a prompt includes the text cited to for the facts of a case or the relevant law. The content of a contract or a statute is added to the prompt.

7. AI systems are also guided by system prompts which users can't see that restrict the possible results. They may be prevented from giving answers with a political perspective.

8. Relativity's aiR for Review is limited to 15,000 characters. Compare this to the much higher token / page limits in Harvey discussed in the March 20, 2026 Tip of the Night.

9. Algorithms should be used to optimize prompts. They cite a research study conducted at VMware, Rick Battle and Teja Gollapudi, The Unreasonable Effectiveness of Eccentric Automatic Prompts, arXiv:2402.10949v2 (2024), which found that optimized prompts will give higher exact matches on average than prompts which merely encourage the system to arrive at a solution.

Relativity has its own prompt optimizer, or kickstarter.

It's possible to upload up to 10 documents (which can't have more than 300K characters), such as complaints, memoranda summarizing a case, or requests for production, to prime this function so that it can autogenerate criteria for a prompt.

10. The active voice should be used in prompts, and double negatives should always be avoided.

11. Boolean operators can be used in prompts, and even putting certain phrases in ALL CAPS or adding exclamation points can lead to a better result.

12. AI systems may get confused by some legal terms which are too vague such as 'reasonable' or 'substantial'.

13. There is currently some debate as to whether prompts should be regarded as work product, or if they ought to be disclosed in ESI protocols just as search terms are.

Harvey Context Window Token Limits

Generative AI models work with context windows which restrict the number of tokens which can be entered in a single query.

So when you're setting up a workflow in Harvey AI, which may use multiple prompts to generate a legal document, each 'block' is limited to 240 pages of text, which might be 100,000 to 200,000 tokens.

This post on harvey.ai , lists the following context range limits:

A review table in Harvey is kind of like a spreadsheet in which entries are provided in a cell for categories separated by columns. There can only be 60 pages of text in each cell.

So you can see here in this demonstration that it is identifying which agreements uploaded as PDFs have a particular type of provision, how this provision is defined, and the basis for the provision to become activated:

A Harvey thread, which is limited to 240 pages of text that consists of a prompt, a series of questions and answers, and uploaded documents. Threads are used to get information about a subset of documents quickly. It's possible to stay under the limit by structuring queries so that the system only analyzes relevant materials. Threads are used to get information about a subset of documents quickly.

So, in a thread the user can interact with Harvey to get results by entering commands in the pane on the left that modifies the work product on the right.

Harvey allows for groups of documents to be collected in big sets of up to 100,000. Queries or commands can then be entered to make Harvey generate content based on just the documents in a vault. So, a user could upload thousands of contracts a business is party to, and get Harvey to generate a table indicating how each agreement meets the requirements of a particular regulation.

A database of court decisions discussing AI Hallucinations

Damien Charlotin, a lecturer in legal data analysis at Sciences Po in Paris, has created an online database listing decisions by courts from around the world in which caselaw, legal norms, and even exhibits were identified as AI hallucinations. See: https://www.damiencharlotin.com/hallucinations/?q=&sort_by=-date&states=USA&period_idx=0

A high percentage of the parties identified as using AI were individuals representing themselves pro se. The database links to PDFs of the decisions, and sometimes provides short summaries of the holding faulting parties for the improper use of AI.

A recent decision by the Supreme Court of North Dakota, Volker v. Nygaard, No. 20250309, 2026 WL 533638 (N.D. Feb. 26, 2026), upheld a district court decision which dismissed a plaintiff's claims under Fed. R. Civ. P. 11 because he repeatedly used non-existent case citations which were created by an AI tool. "During the hearing on the motion, the court warned Volker that his filings contained fictitious legal citations. Despite this warning, Volker filed additional briefs containing fictitious citations . . . At the Order to Show Cause hearing, the district court found that Volker had willfully defied the court and dismissed the action with prejudice as a Rule 11 sanction." Id. at 1.

Charlotin has also developed a system, Pelaikan, which checks the accuracy of citations used in legal briefs. The database of court decisions related to AI hallucinated content includes Pelaikan reports on the briefs criticized by the decisions . These reports explain why the citations are incorrect:

In a decision this January, NYSCEF Doc. No. 45, Decision, Order and Judgment After Sanctions Hearing, Cassata v. Michael Macrina Architect, P.C., Index No. 617183/2025, 2026 WL 263521 (N.Y. Sup. Ct. Jan. 27, 2026), the New York Supreme Court for Suffolk County fined an attorney who, "did not conduct a reasonable, human-based verification of every cited authority before filing her opposition.", and failed to correct her filing when the mistakes were identified. This decision by Justice Linda Kevins includes an exhibit which lists other cases reviewing similar conduct by attorneys, which she considered before imposing sanctions against the defendants' attorneys.