In the Petition from the Cal Bar to the California Supreme Court filed April 29, the Cal Bar indicated that the psychometrician and his firm were asked

to draft 14 questions for the Cal Bar Experiment.  Based on how the Petition was written, the psychometrician’s firm was asked to draft 14 questions

“so that the November study would test a total of 49 questions.”  The inference is that Kaplan couldn’t generate the requisite number of questions on time.

From page 12, footnote 3:

Prior to this, in late September 2024, Admissions staff
requested that ACS draft two questions per subject area for
inclusion in the November bar examination study, so that the
November study would test a total of 49 questions. ACS drafted
14 questions for the November bar examination study using
OpenAI ChatGPT, and the remaining 35 questions were drafted
by Kaplan. Of the 14 ACS-drafted questions, 11 were carried over
and used on the February 2025 bar examination because they
were among the top performing questions from the November
study.

Observations:

1. Kaplan couldn’t generate 49 questions in time.
2. Why ask psychometricians — who don’t necessarily have legal training — to draft MBE questions?
3. Psychometrician’s firm used AI (Open AI and ChatGPT, specifically).  Kaplan’s contract indicates (Article I, section 1.1.5.5) that it “shall not use artificial intelligence in a manner that violates the provisions of Article 18.”  Evidently psychometricians can?  Article 18 states:  18. 1 Contractor warrants and represents that it (including its Representatives) shall not
   use artificial intelligence (“Al”) in a manner that causes or may cause a dilution of Intellectual
   Property Rights for, or in any way preclude the copyrightability or State Bar copyright ownership
   of, any Work Product, Test Materials, or individual test item, including any stimulus, stem, and
   response options. Without limiting the generality of the foregoing, Contractor warrants and
   represents that (a) it shall not use Al in a manner that does not conform to the US Copyright
   Office Guidance (https://copyright.gov/ai/ai_policy_guidance.pdf) (or any update, amendment,
   or new guidance) regarding the requirements for copyrightability and ownership; (b) the
   elements of authorship in any Work Product, Test Materials, and individual test item (the literary
   expression and any elements of selection, arrangement, etc.) shall be conceived, executed, and
   actually formed by humans, not the Al; (c) any use of Al tools shall be solely to enhance limited
   elements of existing human-created Work Product, and any Al contributions shall be the result
   of human original mental conception; (d) any Al-generated content shall be de minimis; and (e)
   any use of Al shall not require the State Bar to exclude or disclaim any content from any copyright
   registration application for any Work Product.
4. If the psychometricians used Open AI and ChatGPT to generate questions, where did Open AI and ChatGPT get those questions from?  Wouldn’t they have consulted NCBE-related content that was available online?  If not that, what alternative materials were available?

Just when you thought this couldn’t get any worse…

On Friday, April 18, the Cal Bar’s Committee of Examiners decided to reduced the raw passing score on the February 2025 Cal Bar Exam from 560 to 534.  Problem?  The Cal Bar can’t do that unilaterally.  It must file a petition to the California Supreme Court.  If the Court approves (ideally before April 28, so that the Cal Bar could implement those changes in results letters), then the Cal Bar felt that it could still publish results on time, on May 2.

Sooooooo…. the Cal Bar didn’t exactly file that petition on April 18.

Then, last week, the Cal Bar revealed that its psychometric vendor, ACS Ventures, used artificial intelligence to craft 23 of the 200 multiple choice questions that applicants saw on the February exam.  In addition, 48 questions were taken from the Cal Bar’s  bank of questions for the First-Year Law Students’ Exam (aka, the Baby Bar, which consists of 100 contracts, crim, and torts MBEs).  Kaplan, you know, the firm that received all that money to create the MBE questions, drafted only 100 of the 200 questions.  No word on who drafted the 29 questions deemed to be experimental and which were not graded.

Remember, the Cal Bar said it would file its petition with the Court on April 18.  As I said, it didn’t happen.  Then, the Cal Supreme Court found about the non-lawyer psychometrician drafting MBE questions with AI just like the rest of us, via press release (trust me, the justices were NOT amused, on multiple levels).  Then, last Thursday, on April 24, the Cal Supreme Court issued a statement:  “Because the court was not made aware of the use of AI to draft some of the multiple-choice questions for the February bar exam, the court has asked the State Bar, in its petition regarding the scoring of the exam, to explain to the court how and why AI was used to draft, revise, or otherwise develop certain multiple-choice questions, efforts taken to ensure the reliability of the AI-assisted multiple-choice questions before they were administered, the reliability of the AI-assisted multiple-choice questions, whether any multiple-choice questions were removed from scoring because they were determined to be unreliable, and the reliability of the remaining multiple-choice questions used for scoring.”  As of Friday, April 25, the Supreme Court confirmed that the Cal Bar hadn’t sent its petition to the Court regarding lowering the passing score for the February 2025 exam.  This is probably because the Court asked the Cal Bar to address how and why they used the 200 questions they used for the MBE on the February 2025 exam.

NOW it comes to pass that tonight, Monday night, April 28, the Cal Bar sent applicants an email:  the Cal Bar still hadn’t filed the petition that it was going to file on April 18.  The Bar said that “we anticipate doing so tomorrow, on April 29.  The timing of our petition submission will not give the Court much time to rule in a manner that allows us to apply the scoring recommendations adopted by the Court and then release February Bar Exam results on May 2.  There may therefore be a slight delay in releasing results.” (emphasis mine).

And, at the end of the email, the Cal Bar said that it is “committed to sharing exam results as soon as we are able.”

Unprecedented, this.  The Cal Bar results may not publish on time.  We’ll see when the Cal Bar files its petition.  We’ll see how quickly the Cal Supreme Court rules on the petition.  Remember, the Cal Bar cannot control the Cal Supreme Court in any way:  what it decides, or when it decides.  Do you think the Supreme Court will adjust the cut score given the Cal Bar’s fundamental alteration of the MBE, by allowing non-lawyers to use AI to generate MBE questions?  Not to mention the use of seemingly easier questions (48 questions for a 1L exam used on a Cal Bar exam), which may have been deliberately used to improve student performance to dilute the impact of Kaplan’s inability to deliver more than 50% of what it promised to do?

Either way, as the Cal Bar said, “there may therefore be a slight delay in releasing results.”  I hope applicants receive results on Friday!  We shall see.

UC Law SF recently informed me that I received a Teacher of the Year award for my 1L Legal Research and Writing Fall semester class!

I’m deeply honored to receive the award.  I have already accepted my offer to teach the class for Year 26 this fall.  Can’t wait to help a new group of students and meet them on the first day of law school!

Hello everyone,

Remember how you and I were exceedingly unhappy about the announcement of the February 2025 Cal Bar “Retake” on March 3-4?  Only to be surpassed by the announcement that one rogue actor published (evidently accurately) one of the questions around March 1, and so the “retake” was moved a few weeks later, to mid-March?  And how that seemed SO UNFAIR because the Bar announced that they were using the same questions?  And the psychometrician also said there was no other way to have a reliable exam than to have the same questions?

That seemed SO unfair.  And it seemed like some people theoretically could have gotten quite an unfair advantage by consulting the deep reddits or the darkest parts of the web and found out what the questions were about before they saw them?

Well, the Cal Bar caved.  Even thought they said there was NO WAY they could provide new questions in time for the March 17 test, well, what do you know, they found a way.

Today, April 16, I discovered that the Cal Bar published the questions that people saw from the original exam on February 25… AND the re-test questions from mid-March.  I expected to see one new question because that’s what the Cal Bar said was going to happen.  Replace the spoiled question with a new one and retest the other 4 essays and a PT.

Nope.  What did I see today?

February 25 (original exam… note the questions were inexplicably presented in a different order for people, but they took the same questions)

1. Crim Pro
2. Wills/Trusts
3. Property
4. Contract Remedies
5. PR

And on the re-test?

1. Evidence
2. Community Property
3. Con Law
4. PR (different fact pattern/calls of question)
5. Contract remedies (different fact pattern/calls of question)

Whither the PTs?

The PTs were different.

Totally different exams, people!  I’m guessing State Bar Counsel informed the Bar that they risked SERIOUS legal exposure if they used the same questions.  Regardless of the rationale, one thing is clear:  the questions were different on Feb. 25 than they were in mid-March.

Now what does that all mean for July predictions?  Who knows? Totally unprecedented circumstances.