Module 5 – Human in the Loop: Training Your Judgment Skills
AI doesn't eliminate the need for attorney judgment, it concentrates it. As AI handles more of the routine, what remains requires more discernment, more scrutiny, and more professional accountability.
This module builds that capacity. It covers where AI fails in ways that only domain expertise can catch, a review framework that replaces approval-mode reading with deliberate professional evaluation, the supervisory obligations that apply to AI-assisted work under existing professional responsibility rules, and the habits that will keep your practice current regardless of where the technology goes next.
PS-006
Welcome To Player Snips
Please watch the getting started video below before you begin using your new superpowers
Identify Your Best First Agent Use Case
Using the Legal Agent Use Case Map, identify the three recurring tasks in your practice that score highest on the evaluation framework — happens often, consistent inputs, catchable outputs. Then choose one to build in the next chapter.
Before you move on, write one sentence for your chosen task:
What the task is
What the agent will receive as input each time
What it will produce as output
PS-015
Evaluate a Set of AI Outputs Using the Review Framework
The following three AI-generated outputs were produced in response to legal task prompts. Each one reads reasonably well on first pass. Using the five-question review framework from this lesson, evaluate each one. For each output, identify: what passes, what fails, and what you would need to verify or correct before this output went anywhere.
Click on the dropdown arrows for the correct answers.
Legal accuracy — Fails. The three-part test described is an accurate general framework, but the characterisation that agreements "two years or less" and covering "a defined regional area" generally satisfy the requirements misstates the law. Enforceability varies enormously by state. California bans non-competes for employees almost entirely. Minnesota, North Dakota, and Oklahoma have similar prohibitions. Several other states have recently passed significant restrictions. Presenting a general multi-factor test as if it applies uniformly is legal flattening — the most dangerous hallucination that isn't technically a hallucination.
Completeness — Fails. The output doesn't know which state's law applies. That single omission means the entire analysis may be irrelevant to the actual matter.
Jurisdictional correctness — Fails. No jurisdiction specified. The output is written as if there is a nationally uniform standard. There isn't.
Reasoning — Partially passes. The internal logic of the general framework is sound. The reasoning fails because it's applied to the wrong question — it describes what courts generally consider without telling the attorney what this specific jurisdiction actually does.
Defend every sentence — No. "Courts across the country have held that agreements lasting two years or less... generally satisfy these requirements" is not defensible in California, Minnesota, or several other states.
Relevance — Fails. Without knowing the governing jurisdiction, this summary cannot answer the question the client is actually asking.
Legal accuracy — Passes. Nothing in this email is legally wrong.
Completeness — Fails significantly. The email contains no date, time, or location for the deposition. It doesn't tell the client what they need to bring or do to prepare. It doesn't reflect anything specific about this client, this case, or this deposition. It is a generic deposition notification email that would apply to any client in any matter.
Jurisdictional correctness — Passes for what it covers. Nothing jurisdiction-specific is required for this communication.
Reasoning — Passes for what it covers.
Defend every sentence — Borderline. The explanation of what a deposition is may be unnecessary for a client who has been in litigation for months and already understands the process — it could feel condescending. That's a client-context judgment call the attorney needs to make, not AI.
Relevance — Fails. The email doesn't actually tell the client the most important thing: when and where to show up. It also doesn't reflect anything about the relationship, the client's likely anxiety level, or what this deposition specifically concerns. An attorney who sent this without editing it would be sending a template, not a communication.
Legal accuracy — Fails. The argument is legally coherent but the statement that "courts have consistently dismissed implied covenant claims that are duplicative of express contract claims" is overconfident. This is a well-established principle in some jurisdictions — California, New York, and others apply it — but it is not uniformly applied in all states. Some states treat the implied covenant as a separate cause of action with independent vitality even where express terms exist. Presenting it as a universal rule without a jurisdiction-specific citation is the legal flattening failure mode.
Completeness — Fails. The section references "Section 4.2 of the Agreement" but doesn't quote the provision or describe what it says. An argument that this section "expressly addresses" service delivery is weaker without showing the court what the section actually says.
Jurisdictional correctness — Fails. No jurisdiction is cited anywhere in the argument. "Courts have consistently dismissed" without a single citation to the controlling authority in the relevant court is not brief-ready.
Reasoning — Partially passes. The internal logic — duplicative claims / express terms govern / implied covenant cannot add obligations that contradict express terms — is structurally sound. The weakness is in the evidentiary support for the conclusion, not the logical structure.
Defend every sentence — No. "Courts have consistently dismissed implied covenant claims that are duplicative of express contract claims" cannot be defended without a citation and would not survive a judge who pushes back.
Relevance — Passes. The output addresses the actual argument requested and is structured for the right purpose.
PS-041
Draft Your 90-Day AI Adoption Plan
You've covered six modules and built a foundation most attorneys practicing today don't have. Now decide what you're going to do with it in the next 90 days.
Reflect on the following three prompts in your Adoption Plan document (attached):
One skill to build: Which part of this track felt most underdeveloped in your current practice — prompting, research verification, document review, safe habits? Pick one and commit to deliberate practice on it over the next 90 days.
One habit to establish: Which activity from this track — the pre-AI checklist, the five-question review framework, the prompt library — would have the biggest impact on your practice if it became automatic? Decide now that it will.
One tool to evaluate: Is there an AI tool you've been curious about but haven't properly assessed? Use the evaluation rubric from Module 1 and give it a real look in the next 30 days.
PS-001
Configure Your First Agent
Using the Agent Configuration Template, configure your agent for the use case you identified in 5.1. Work through all five elements before moving to 5.3:
Assemble your content block: what is the communication goal, who is the recipient, what is the right register for this relationship, what prior context does AI need?
Goal: one sentence, specific enough that someone who doesn't know your practice could understand exactly what this agent does
Instructions: concrete, section-by-section, with format specified. If you find yourself writing something general like "be professional," go back and make it specific — what does professional mean for this output type?
Permissions: what does this agent need access to? What should it explicitly not touch?
Output format: write out the template your agent will follow on every output. If you can't write the template, the instructions aren't specific enough yet.
Escalation rule: what does the agent do when the input doesn't fit its scope?
PS-008
Legal accuracy — Fails. The three-part test described is an accurate general framework, but the characterisation that agreements "two years or less" and covering "a defined regional area" generally satisfy the requirements misstates the law. Enforceability varies enormously by state. California bans non-competes for employees almost entirely. Minnesota, North Dakota, and Oklahoma have similar prohibitions. Several other states have recently passed significant restrictions. Presenting a general multi-factor test as if it applies uniformly is legal flattening — the most dangerous hallucination that isn't technically a hallucination.
Completeness — Fails. The output doesn't know which state's law applies. That single omission means the entire analysis may be irrelevant to the actual matter.
Jurisdictional correctness — Fails. No jurisdiction specified. The output is written as if there is a nationally uniform standard. There isn't.
Reasoning — Partially passes. The internal logic of the general framework is sound. The reasoning fails because it's applied to the wrong question — it describes what courts generally consider without telling the attorney what this specific jurisdiction actually does.
Defend every sentence — No. "Courts across the country have held that agreements lasting two years or less... generally satisfy these requirements" is not defensible in California, Minnesota, or several other states.
Relevance — Fails. Without knowing the governing jurisdiction, this summary cannot answer the question the client is actually asking.
Legal accuracy — Passes. Nothing in this email is legally wrong.
Completeness — Fails significantly. The email contains no date, time, or location for the deposition. It doesn't tell the client what they need to bring or do to prepare. It doesn't reflect anything specific about this client, this case, or this deposition. It is a generic deposition notification email that would apply to any client in any matter.
Jurisdictional correctness — Passes for what it covers. Nothing jurisdiction-specific is required for this communication.
Reasoning — Passes for what it covers.
Defend every sentence — Borderline. The explanation of what a deposition is may be unnecessary for a client who has been in litigation for months and already understands the process — it could feel condescending. That's a client-context judgment call the attorney needs to make, not AI.
Relevance — Fails. The email doesn't actually tell the client the most important thing: when and where to show up. It also doesn't reflect anything about the relationship, the client's likely anxiety level, or what this deposition specifically concerns. An attorney who sent this without editing it would be sending a template, not a communication.
Legal accuracy — Fails. The argument is legally coherent but the statement that "courts have consistently dismissed implied covenant claims that are duplicative of express contract claims" is overconfident. This is a well-established principle in some jurisdictions — California, New York, and others apply it — but it is not uniformly applied in all states. Some states treat the implied covenant as a separate cause of action with independent vitality even where express terms exist. Presenting it as a universal rule without a jurisdiction-specific citation is the legal flattening failure mode.
Completeness — Fails. The section references "Section 4.2 of the Agreement" but doesn't quote the provision or describe what it says. An argument that this section "expressly addresses" service delivery is weaker without showing the court what the section actually says.
Jurisdictional correctness — Fails. No jurisdiction is cited anywhere in the argument. "Courts have consistently dismissed" without a single citation to the controlling authority in the relevant court is not brief-ready.
Reasoning — Partially passes. The internal logic — duplicative claims / express terms govern / implied covenant cannot add obligations that contradict express terms — is structurally sound. The weakness is in the evidentiary support for the conclusion, not the logical structure.
Defend every sentence — No. "Courts have consistently dismissed implied covenant claims that are duplicative of express contract claims" cannot be defended without a citation and would not survive a judge who pushes back.
Relevance — Passes. The output addresses the actual argument requested and is structured for the right purpose.
PS-040
-
1Set a Google Alert for your state bar
Go to google.com/alerts and create an alert for your state bar's name plus "artificial intelligence" or "generative AI" — for example, "State Bar of California" "artificial intelligence" or "Illinois ARDC" "AI". Set it to deliver weekly. This catches formal opinions, guidance documents, and press releases the moment they're indexed.
-
2Bookmark your state bar's ethics opinions page
Find the ethics opinions or formal guidance section of your state bar's website and bookmark it now, while you're thinking about it. Set a recurring calendar reminder — quarterly is enough — to check it directly. Google Alerts will catch most things, but not everything gets indexed immediately. -
3Add one practitioner-level source
Pick one source that tracks AI ethics developments across jurisdictions — the ABA's Legal Technology Resource Center, Bob Ambrogi's LawNext, or the Justia 50-State AI Ethics Survey are reliable options. You don't need all three. One source you'll actually read is worth more than five you'll ignore.
By the end of this activity you have a functioning monitoring system. It will surface what matters without requiring you to go looking for it.