Category: Uncategorized

Psychometrician drafted 14 of 49 questions in Cal Bar Experiment (OpenAI and Chat GPT used)

In the Petition from the Cal Bar to the California Supreme Court filed April 29, the Cal Bar indicated that the psychometrician and his firm were asked

to draft 14 questions for the Cal Bar Experiment.  Based on how the Petition was written, the psychometrician’s firm was asked to draft 14 questions

“so that the November study would test a total of 49 questions.”  The inference is that Kaplan couldn’t generate the requisite number of questions on time.

From page 12, footnote 3:

Prior to this, in late September 2024, Admissions staff
requested that ACS draft two questions per subject area for
inclusion in the November bar examination study, so that the
November study would test a total of 49 questions. ACS drafted
14 questions for the November bar examination study using
OpenAI ChatGPT, and the remaining 35 questions were drafted
by Kaplan. Of the 14 ACS-drafted questions, 11 were carried over
and used on the February 2025 bar examination because they
were among the top performing questions from the November
study.

 

Observations:

  1. Kaplan couldn’t generate 49 questions in time.
  2. Why ask psychometricians — who don’t necessarily have legal training — to draft MBE questions?
  3. Psychometrician’s firm used AI (Open AI and ChatGPT, specifically).  Kaplan’s contract indicates (Article I, section 1.1.5.5) that it “shall not use artificial intelligence in a manner that violates the provisions of Article 18.”  Evidently psychometricians can?  Article 18 states:  18. 1 Contractor warrants and represents that it (including its Representatives) shall not
    use artificial intelligence (“Al”) in a manner that causes or may cause a dilution of Intellectual
    Property Rights for, or in any way preclude the copyrightability or State Bar copyright ownership
    of, any Work Product, Test Materials, or individual test item, including any stimulus, stem, and
    response options. Without limiting the generality of the foregoing, Contractor warrants and
    represents that (a) it shall not use Al in a manner that does not conform to the US Copyright
    Office Guidance (https://copyright.gov/ai/ai_policy_guidance.pdf) (or any update, amendment,
    or new guidance) regarding the requirements for copyrightability and ownership; (b) the
    elements of authorship in any Work Product, Test Materials, and individual test item (the literary
    expression and any elements of selection, arrangement, etc.) shall be conceived, executed, and
    actually formed by humans, not the Al; (c) any use of Al tools shall be solely to enhance limited
    elements of existing human-created Work Product, and any Al contributions shall be the result
    of human original mental conception; (d) any Al-generated content shall be de minimis; and (e)
    any use of Al shall not require the State Bar to exclude or disclaim any content from any copyright
    registration application for any Work Product.
  4. If the psychometricians used Open AI and ChatGPT to generate questions, where did Open AI and ChatGPT get those questions from?  Wouldn’t they have consulted NCBE-related content that was available online?  If not that, what alternative materials were available?

We ARE getting results on Friday… right?!

Just when you thought this couldn’t get any worse…

On Friday, April 18, the Cal Bar’s Committee of Examiners decided to reduced the raw passing score on the February 2025 Cal Bar Exam from 560 to 534.  Problem?  The Cal Bar can’t do that unilaterally.  It must file a petition to the California Supreme Court.  If the Court approves (ideally before April 28, so that the Cal Bar could implement those changes in results letters), then the Cal Bar felt that it could still publish results on time, on May 2.

Sooooooo…. the Cal Bar didn’t exactly file that petition on April 18.

Then, last week, the Cal Bar revealed that its psychometric vendor, ACS Ventures, used artificial intelligence to craft 23 of the 200 multiple choice questions that applicants saw on the February exam.  In addition, 48 questions were taken from the Cal Bar’s  bank of questions for the First-Year Law Students’ Exam (aka, the Baby Bar, which consists of 100 contracts, crim, and torts MBEs).  Kaplan, you know, the firm that received all that money to create the MBE questions, drafted only 100 of the 200 questions.  No word on who drafted the 29 questions deemed to be experimental and which were not graded.

Remember, the Cal Bar said it would file its petition with the Court on April 18.  As I said, it didn’t happen.  Then, the Cal Supreme Court found about the non-lawyer psychometrician drafting MBE questions with AI just like the rest of us, via press release (trust me, the justices were NOT amused, on multiple levels).  Then, last Thursday, on April 24, the Cal Supreme Court issued a statement:  “Because the court was not made aware of the use of AI to draft some of the multiple-choice questions for the February bar exam, the court has asked the State Bar, in its petition regarding the scoring of the exam, to explain to the court how and why AI was used to draft, revise, or otherwise develop certain multiple-choice questions, efforts taken to ensure the reliability of the AI-assisted multiple-choice questions before they were administered, the reliability of the AI-assisted multiple-choice questions, whether any multiple-choice questions were removed from scoring because they were determined to be unreliable, and the reliability of the remaining multiple-choice questions used for scoring.”  As of Friday, April 25, the Supreme Court confirmed that the Cal Bar hadn’t sent its petition to the Court regarding lowering the passing score for the February 2025 exam.  This is probably because the Court asked the Cal Bar to address how and why they used the 200 questions they used for the MBE on the February 2025 exam.

NOW it comes to pass that tonight, Monday night, April 28, the Cal Bar sent applicants an email:  the Cal Bar still hadn’t filed the petition that it was going to file on April 18.  The Bar said that “we anticipate doing so tomorrow, on April 29.  The timing of our petition submission will not give the Court much time to rule in a manner that allows us to apply the scoring recommendations adopted by the Court and then release February Bar Exam results on May 2.  There may therefore be a slight delay in releasing results.(emphasis mine).

And, at the end of the email, the Cal Bar said that it is “committed to sharing exam results as soon as we are able.”

Unprecedented, this.  The Cal Bar results may not publish on time.  We’ll see when the Cal Bar files its petition.  We’ll see how quickly the Cal Supreme Court rules on the petition.  Remember, the Cal Bar cannot control the Cal Supreme Court in any way:  what it decides, or when it decides.  Do you think the Supreme Court will adjust the cut score given the Cal Bar’s fundamental alteration of the MBE, by allowing non-lawyers to use AI to generate MBE questions?  Not to mention the use of seemingly easier questions (48 questions for a 1L exam used on a Cal Bar exam), which may have been deliberately used to improve student performance to dilute the impact of Kaplan’s inability to deliver more than 50% of what it promised to do?

Either way, as the Cal Bar said, “there may therefore be a slight delay in releasing results.”  I hope applicants receive results on Friday!  We shall see.

Teacher of the Year Award!

UC Law SF recently informed me that I received a Teacher of the Year award for my 1L Legal Research and Writing Fall semester class!

I’m deeply honored to receive the award.  I have already accepted my offer to teach the class for Year 26 this fall.  Can’t wait to help a new group of students and meet them on the first day of law school!

 

UPDATE! Conspiracy theory UPDATE!

Hello everyone,

Remember how you and I were exceedingly unhappy about the announcement of the February 2025 Cal Bar “Retake” on March 3-4?  Only to be surpassed by the announcement that one rogue actor published (evidently accurately) one of the questions around March 1, and so the “retake” was moved a few weeks later, to mid-March?  And how that seemed SO UNFAIR because the Bar announced that they were using the same questions?  And the psychometrician also said there was no other way to have a reliable exam than to have the same questions?

That seemed SO unfair.  And it seemed like some people theoretically could have gotten quite an unfair advantage by consulting the deep reddits or the darkest parts of the web and found out what the questions were about before they saw them?

Well, the Cal Bar caved.  Even thought they said there was NO WAY they could provide new questions in time for the March 17 test, well, what do you know, they found a way.

Today, April 16, I discovered that the Cal Bar published the questions that people saw from the original exam on February 25… AND the re-test questions from mid-March.  I expected to see one new question because that’s what the Cal Bar said was going to happen.  Replace the spoiled question with a new one and retest the other 4 essays and a PT.

Nope.  What did I see today?

February 25 (original exam… note the questions were inexplicably presented in a different order for people, but they took the same questions)

  1. Crim Pro
  2. Wills/Trusts
  3. Property
  4. Contract Remedies
  5. PR

And on the re-test?

  1. Evidence
  2. Community Property
  3. Con Law
  4. PR (different fact pattern/calls of question)
  5. Contract remedies (different fact pattern/calls of question)

Whither the PTs?

The PTs were different.

 

Totally different exams, people!  I’m guessing State Bar Counsel informed the Bar that they risked SERIOUS legal exposure if they used the same questions.  Regardless of the rationale, one thing is clear:  the questions were different on Feb. 25 than they were in mid-March.

Now what does that all mean for July predictions?  Who knows? Totally unprecedented circumstances.

 

 

Taking a Test

June 29 update about masks, admittance ticket, and mock exam

Hello everyone!
Remember that item I posted about the July 2022 FAQ document being replaced with info about Clear Health Pass/Mask Mandate that was copied and pasted from a bar exam page shortly before the February 2022 exam?  As in, dates included late January and early February?

Well, that was short.  Less than a few hours later, it’s gone.  So the April 14 FAQ is now back where it belonged, and no updates have issued.

So as of June 29, no need to upload vaccination status anywhere.

As of June 29, masks aren’t required for the July 2022 Cal Bar Exam.  I strongly recommend you wear one for 3- to 3-1/2 hours a day just in case LA County announces a mask mandate that would begin July 15-22.

If the weekly Cal Bar Friday update indicates, final answer, no masks required (on July 1, 8, 15), then stop wearing a mask.

If the weekly Cal Bar Friday update on July 22 (the Friday before the exam) says masks are optional, then you can take off your mask.

Admittance tickets and mock exams are theoretically available as of Tuesday.  Evidently these are distributed in batches over the next week.  Call the Cal Bar if you don’t have your admittance ticket by July 5.

Remember, you MUST complete the mock exam by July 22, otherwise you won’t be admitted to the exam.  And who wants that?

 

 

 

 

Alameda County rescinds mask mandate. LA may institute one in mid-July

Hello everyone.

 

Well, that was fast!  Alameda County is rescinding its mask mandate, effective Saturday, June 25.

 

Per the LA Times:

 

Three weeks after becoming the first California county to reinstitute a mask mandate in most indoor public settings amid climbing coronavirus cases and hospitalizations, Alameda County has rescinded the order — citing improving conditions.

The move, effective at 12:01 a.m. Saturday, coincides as the San Francisco Bay Area’s second most populous county progresses from the high to medium COVID-19 community transmission level as defined by the U.S. Centers for Disease Control and Prevention.

 

So that means no chance of wearing a mask on exam day, right?  Not so fast.  LA County estimated that it would reinstitute a mask mandate in late June.  Then that estimate moved to early July due to lagging reporting during the Memorial Day weekend.  Now, the estimate is mid-July.  Per the LA Times:

Health officials in Los Angeles County have said they would reimpose a public indoor mask mandate should the region fall in the high COVID-19 community level for two consecutive weeks. That category, the worst on the CDC’s three-tier scale, indicates not only significant community transmission but also that hospital systems may grow strained by coronavirus-positive patients.

Based on current hospitalization trends, L.A. County would likely not reach that category until mid-July.

However, that projection is “based on that assumption of a continued rate of increase that doesn’t change, and that’s really impossible to predict,” said Dr. Paul Simon, chief science officer for the L.A. County Department of Public Health.

“We’re cautiously optimistic that we may level off and, in the best of all worlds, we will begin to see a decline in hospital admissions sooner rather than later,” he told reporters Thursday. “But it is hard to predict.”

Hmmm… “must be in the high COVID-19 community level for two consecutive weeks.”  “Not reach that category until mid-July”.

So what does the Cal Bar do?  Are we going to escape scot-free?  Or will there be an announcement of a mask mandate in LA County and prompt the Cal Bar to mandate masks days before the bar exam?  Who knows?

For the second consecutive week, the Cal Bar issued a Friday email and noted that masks are optional for the exam.

I still recommend that students wear masks for 3- to 3-1/2 hours a day.  If you see an email from the Cal Bar on Friday, July 22, and you see that masks are still optional, then laugh as you take it off (or if you are inclined to keep it on, you can).  But if LA County comes back with a mask mandate on July 20, or 22, or 24, and the Cal Bar is forced to email you that masks are mandatory, what will you do?  Would you prefer to have practiced wearing a mask for hours at a time for a month?  Or have no recent experience with it, and touch your mask 70 times over a three-hour period, and be super distracted?

I don’t know what the final answer is here.  Do you?  Would you prefer to have a plan B, just in case?

Tread carefully, friends.  Good luck to you as continue your preparation for the exam.

 

 

 

 

 

pencil

February 2022 Exam Percentile Table

Applicants will find this to be of interest:

The overall pass rate was 33.9%.

The Percentile Table shows us some relevant info:

45% scored higher than 1340.

40% scored over 1355.

35% scored higher than 1382.

30% scored higher than 1400.

The overall pass line is 1390, and you need more than 1350 to get a re-read.

So… roughly 32.5% passed on the first read.  About 10% of the applicant pool got a second read, and the second read appears to have increased the pass rate by roughly 1.4%.

 

 

Mean MBE score for February 2021 exam higher than February 2020

Per ncbex.org
National Mean of 134.0 for February 2021 MBE

The national Multistate Bar Examination (MBE) mean scaled score for February 2021 was 134.0, an increase of about 1.4 points from the February 2020 mean of 132.6. 16,759 examinees took the MBE in February 2021; of those, 1,665 took the MBE in the 18 jurisdictions that administered the exam in person, and 15,094 took the MBE in the 33 jurisdictions that administered the exam remotely. February 2021 had about 12% fewer examinees compared to the 19,122 who sat in February 2020.[1]

Reliability for the February 2021 exam was 0.93, slightly higher than the reliability for February 2020. (Reliability is an indicator of the consistency of a set of examination scores, with a maximum value of 1.0.)

 

Graph shows the February MBE national mean scaled scores from 2017 through 2021.