Psychometrician drafted 14 of 49 questions in Cal Bar Experiment (OpenAI and Chat GPT used)
In the Petition from the Cal Bar to the California Supreme Court filed April 29, the Cal Bar indicated that the psychometrician and his firm were asked
to draft 14 questions for the Cal Bar Experiment. Based on how the Petition was written, the psychometrician’s firm was asked to draft 14 questions
“so that the November study would test a total of 49 questions.” The inference is that Kaplan couldn’t generate the requisite number of questions on time.
From page 12, footnote 3:
Prior to this, in late September 2024, Admissions staff
requested that ACS draft two questions per subject area for
inclusion in the November bar examination study, so that the
November study would test a total of 49 questions. ACS drafted
14 questions for the November bar examination study using
OpenAI ChatGPT, and the remaining 35 questions were drafted
by Kaplan. Of the 14 ACS-drafted questions, 11 were carried over
and used on the February 2025 bar examination because they
were among the top performing questions from the November
study.
Observations:
- Kaplan couldn’t generate 49 questions in time.
- Why ask psychometricians — who don’t necessarily have legal training — to draft MBE questions?
- Psychometrician’s firm used AI (Open AI and ChatGPT, specifically). Kaplan’s contract indicates (Article I, section 1.1.5.5) that it “shall not use artificial intelligence in a manner that violates the provisions of Article 18.” Evidently psychometricians can? Article 18 states: 18. 1 Contractor warrants and represents that it (including its Representatives) shall not
use artificial intelligence (“Al”) in a manner that causes or may cause a dilution of Intellectual
Property Rights for, or in any way preclude the copyrightability or State Bar copyright ownership
of, any Work Product, Test Materials, or individual test item, including any stimulus, stem, and
response options. Without limiting the generality of the foregoing, Contractor warrants and
represents that (a) it shall not use Al in a manner that does not conform to the US Copyright
Office Guidance (https://copyright.gov/ai/ai_policy_guidance.pdf) (or any update, amendment,
or new guidance) regarding the requirements for copyrightability and ownership; (b) the
elements of authorship in any Work Product, Test Materials, and individual test item (the literary
expression and any elements of selection, arrangement, etc.) shall be conceived, executed, and
actually formed by humans, not the Al; (c) any use of Al tools shall be solely to enhance limited
elements of existing human-created Work Product, and any Al contributions shall be the result
of human original mental conception; (d) any Al-generated content shall be de minimis; and (e)
any use of Al shall not require the State Bar to exclude or disclaim any content from any copyright
registration application for any Work Product. - If the psychometricians used Open AI and ChatGPT to generate questions, where did Open AI and ChatGPT get those questions from? Wouldn’t they have consulted NCBE-related content that was available online? If not that, what alternative materials were available?