We used ChatGPT 3.5 for this experiment. According to the openAI website, ChatGPT 4.0 is much improved.

ChatGPT is a LLM which relies on determining patterns from written language and formalizing a cohesive and correct response to certain prompts. We decided to stress test ChatGPT by asking it to answer the example LSAT questions from the LSAC website. ChatGPT should be able to get 80% right to have a decent shot at being admitted to a good law school.

ChatGPT was able to answer 5/10 correct. ChatGPT would not be able to gain entrance into a good law school.

One thing to note is that ChatGPT presents its findings as completely true and valid and attempts to provide detailed explanations of its thinking. This goes to show that AI cannot be 100% relied upon to provide correct answers to difficult questions!

Incorrect

Laird: Pure research provides us with new technologies that contribute to saving lives. Even more worthwhile than this, however, is its role in expanding our knowledge and providing new, unexplored ideas.

Kim: Your priorities are mistaken. Saving lives is what counts most of all. Without pure research, medicine would not be as advanced as it is.

Laird and Kim disagree on whether pure research

a: derives its significance in part from its providing new technologies
b: expands the boundaries of our knowledge of medicine
c: should have the saving of human lives as an important goal
d: has its most valuable achievements in medical applications
e: has any value apart from its role in providing new technologies to save lives

ChatGPT:

Laird and Kim disagree on whether pure research:

E: has any value apart from its role in providing new technologies to save lives

LSAC:

has its most valuable achievements in medical applications

Correct

choose the correct response to the blow prompt:

"Executive: We recently ran a set of advertisements in the print version of a travel magazine and on that magazine’s website. We were unable to get any direct information about consumer response to the print ads. However, we found that consumer response to the ads on the website was much more limited than is typical for website ads. We concluded that consumer response to the print ads was probably below par as well."

The executive’s reasoning does which one of the following?

A: bases a prediction of the intensity of a phenomenon on information about the intensity of that phenomenon’s cause
B: uses information about the typical frequency of events of a general kind to draw a conclusion about the probability of a particular event of that kind
C: infers a statistical generalization from claims about a large number of specific instances
D: uses a case in which direct evidence is available to draw a conclusion about an analogous case in which direct evidence is unavailable
E: bases a prediction about future events on facts about recent comparable events

ChatGPT:
The correct response is:

D: uses a case in which direct evidence is available to draw a conclusion about an analogous case in which direct evidence is unavailable

LSAC:

uses a case in which direct evidence is available to draw a conclusion about an analogous case in which direct evidence is unavailable

correct!

During the construction of the Quebec Bridge in 1907, the bridge’s designer, Theodore Cooper, received word that the suspended span being built out from the bridge’s cantilever was deflecting downward by a fraction of an inch (2.54 centimeters). Before he could telegraph to freeze the project, the whole cantilever arm broke off and plunged, along with seven dozen workers, into the St. Lawrence River. It was the worst bridge construction disaster in history. As a direct result of the inquiry that followed, the engineering “rules of thumb” by which thousands of bridges had been built around the world went down with the Quebec Bridge. Twentieth-century bridge engineers would thereafter depend on far more rigorous applications of mathematical analysis.

Which one of the following statements can be properly inferred from the passage?

Bridges built before about 1907 were built without thorough mathematical analysis and, therefore, were unsafe for the public to use.
Cooper’s absence from the Quebec Bridge construction site resulted in the breaking off of the cantilever.
Nineteenth-century bridge engineers relied on their rules of thumb because analytical methods were inadequate to solve their design problems.
Only a more rigorous application of mathematical analysis to the design of the Quebec Bridge could have prevented its collapse.
Prior to 1907 the mathematical analysis incorporated in engineering rules of thumb was insufficient to completely assure the safety of bridges under construction.

ChatGPT:

The statement that can be properly inferred from the passage is:

Prior to 1907 the mathematical analysis incorporated in engineering rules of thumb was insufficient to completely assure the safety of bridges under construction.

This is evident from the passage's mention that the disaster led to a shift towards more rigorous applications of mathematical analysis in bridge engineering, implying that the previous methods were not comprehensive enough to ensure safety.

Incorrect

The supernova event of 1987 is interesting in that there is still no evidence of the neutron star that current theory says should have remained after a supernova of that size. This is in spite of the fact that many of the most sensitive instruments ever developed have searched for the tell-tale pulse of radiation that neutron stars emit. Thus, current theory is wrong in claiming that supernovas of a certain size always produce neutron stars.

Which one of the following, if true, most strengthens the argument?

Most supernova remnants that astronomers have detected have a neutron star nearby.
Sensitive astronomical instruments have detected neutron stars much farther away than the location of the 1987 supernova.
The supernova of 1987 was the first that scientists were able to observe in progress.
Several important features of the 1987 supernova are correctly predicted by the current theory.
Some neutron stars are known to have come into existence by a cause other than a supernova explosion.

CharGPT
The statement that, if true, most strengthens the argument is:

Some neutron stars are known to have come into existence by a cause other than a supernova explosion.

This strengthens the argument by providing an alternative explanation for the absence of a neutron star following the supernova event of 1987. If neutron stars can come into existence through other means aside from supernova explosions, then it becomes plausible that the absence of a neutron star in this particular supernova event does not necessarily contradict current theory.

LSAC:

Sensitive astronomical instruments have detected neutron stars much farther away than the location of the 1987 supernova.

correct

Political scientist: As a political system, democracy does not promote political freedom. There are historical examples of democracies that ultimately resulted in some of the most oppressive societies. Likewise, there have been enlightened despotisms and oligarchies that have provided a remarkable level of political freedom to their subjects.

The reasoning in the political scientist’s argument is flawed because it

confuses the conditions necessary for political freedom with the conditions sufficient to bring it about     
fails to consider that a substantial increase in the level of political freedom might cause a society to become more democratic
appeals to historical examples that are irrelevant to the causal claim being made
overlooks the possibility that democracy promotes political freedom without being necessary or sufficient by itself to produce it
bases its historical case on a personal point of view

Chat GPT:

The reasoning in the political scientist's argument is flawed because it:

overlooks the possibility that democracy promotes political freedom without being necessary or sufficient by itself to produce it.

The argument presented by the political scientist dismisses democracy's role in promoting political freedom solely based on historical examples, without considering the possibility that democracy may contribute to political freedom in some contexts, even if it's not the sole factor or guarantee of political freedom. Therefore, the argument overlooks this important possibility, making the reasoning flawed.

LSAC:

overlooks the possibility that democracy promotes political freedom without being necessary or sufficient by itself to produce it

correct

Journalist: To reconcile the need for profits sufficient to support new drug research with the moral imperative to provide medicines to those who most need them but cannot afford them, some pharmaceutical companies feel justified in selling a drug in rich nations at one price and in poor nations at a much lower price. But this practice is unjustified. A nation with a low average income may still have a substantial middle class better able to pay for new drugs than are many of the poorer citizens of an overall wealthier nation.

Which one of the following principles, if valid, most helps to justify the journalist’s reasoning?

People who are ill deserve more consideration than do healthy people, regardless of their relative socioeconomic positions.
Wealthy institutions have an obligation to expend at least some of their resources to assist those incapable of assisting themselves.
Whether one deserves special consideration depends on one’s needs rather than on characteristics of the society to which one belongs.
The people in wealthy nations should not have better access to health care than do the people in poorer nations.
Unequal access to health care is more unfair than an unequal distribution of wealth.

ChatGPT:

The principle that, if valid, most helps to justify the journalist's reasoning is:

Whether one deserves special consideration depends on one’s needs rather than on characteristics of the society to which one belongs.

This principle supports the journalist's argument by emphasizing the importance of addressing the needs of individuals based on their circumstances rather than on the wealth or characteristics of their society. It underscores the idea that access to essential medicines should be determined by individual need rather than by the economic disparities between nations.

LSAC:

Whether one deserves special consideration depends on one’s needs rather than on characteristics of the society to which one belongs.

incorrect

Several critics have claimed that any contemporary poet who writes formal poetry—poetry that is rhymed and metered—is performing a politically conservative act. This is plainly false. Consider Molly Peacock and Marilyn Hacker, two contemporary poets whose poetry is almost exclusively formal and yet who are themselves politically progressive feminists.

The conclusion drawn above follows logically if which one of the following is assumed?

No one who is a feminist is also politically conservative.
No poet who writes unrhymed or unmetered poetry is politically conservative.
No one who is politically progressive is capable of performing a politically conservative act.
Anyone who sometimes writes poetry that is not politically conservative never writes poetry that is politically conservative.
The content of a poet’s work, not the work’s form, is the most decisive factor in determining what political consequences, if any, the work will have.

ChatGPT:

The conclusion drawn above follows logically if the following assumption is assumed:

The content of a poet’s work, not the work’s form, is the most decisive factor in determining what political consequences, if any, the work will have.

This assumption supports the conclusion that writing formal poetry does not inherently equate to a politically conservative act. It suggests that the political implications of a poet's work are primarily determined by its content rather than its form. Therefore, the conclusion that Molly Peacock and Marilyn Hacker, despite writing formal poetry, are politically progressive feminists is supported by this assumption.

LSAC:

No one who is politically progressive is capable of performing a politically conservative act.

correct

About two million years ago, lava dammed up a river in western Asia and caused a small lake to form. The lake existed for about half a million years. Bones of an early human ancestor were recently found in the ancient lake-bottom sediments that lie on top of the layer of lava. Therefore, ancestors of modern humans lived in western Asia between two million and one-and-a-half million years ago.

Which one of the following is an assumption required by the argument?

There were no other lakes in the immediate area before the lava dammed up the river.
The lake contained fish that the human ancestors could have used for food.
The lava that lay under the lake-bottom sediments did not contain any human fossil remains.
The lake was deep enough that a person could drown in it.
The bones were already in the sediments by the time the lake dried up.

Chat GPT:

The assumption required by the argument is:

The bones were already in the sediments by the time the lake dried up.

This assumption is necessary to support the conclusion that ancestors of modern humans lived in western Asia between two million and one-and-a-half million years ago based on the discovery of the early human ancestor bones in the ancient lake-bottom sediments. If the bones were not already in the sediments before the lake dried up, their presence in the sediments would not necessarily imply the existence of human ancestors in the region during that time period.

LSAC:

The bones were already in the sediments by the time the lake dried up.

incorrect

In jurisdictions where use of headlights is optional when visibility is good, drivers who use headlights at all times are less likely to be involved in a collision than are drivers who use headlights only when visibility is poor. Yet Highway Safety Department records show that making use of headlights mandatory at all times does nothing to reduce the overall number of collisions.

Which one of the following, if true, most helps to resolve the apparent discrepancy in the information above?

In jurisdictions where use of headlights is optional when visibility is good, one driver in four uses headlights for daytime driving in good weather.
A law making use of headlights mandatory at all times is not especially difficult to enforce.
Only very careful drivers use headlights when their use is not legally required.
There are some jurisdictions in which it is illegal to use headlights when visibility is good.
The jurisdictions where use of headlights is mandatory at all times are those where daytime visibility is frequently poor.

ChatGPT:

The statement that, if true, most helps to resolve the apparent discrepancy in the information provided is:

The jurisdictions where use of headlights is mandatory at all times are those where daytime visibility is frequently poor.

This statement helps to resolve the discrepancy by suggesting that the effectiveness of mandatory headlights at reducing collisions depends on the prevailing visibility conditions in the jurisdiction. If mandatory headlight use is implemented in areas where daytime visibility is frequently poor, then the lack of reduction in collisions despite the mandate may be due to other contributing factors related to poor visibility conditions.

LSAC:

Only very careful drivers use headlights when their use is not legally required.

incorrect

The Venetian Renaissance painter Vittore Carpaccio used sumptuous reds in most of his paintings. Since the recently discovered Venetian Renaissance painting Erato Declaiming contains notable sumptuous reds, it is probably by Carpaccio.

Which one of the following contains a pattern of flawed reasoning most similar to that in the argument above?

Most Renaissance painters worked in a single medium, either tempera or oil. Since the Renaissance painting Calypso's Bower is in oil, its painter probably always used oil.
In Italian Renaissance painting, the single most common subject was the Virgin and Child, so the single most common subject in Western art probably is also the Virgin and Child.
Works of art in the Renaissance were mostly commissioned by patrons, so the Renaissance work The Dances of Terpsichore was probably commissioned by a patron.
The anonymous painting St. Sebastian is probably an early Florentine painting since it is in tempera, and most early Florentine paintings were in tempera.
Since late-Renaissance paintings were mostly in oil, the Venetian late-Renaissance painter Arnoldi, whose works are now lost, probably painted in oil.

Chat GPT:

The flawed reasoning in the argument is that it assumes a single characteristic (the use of sumptuous reds) is sufficient to attribute authorship to Carpaccio without considering other factors or possibilities. A similar pattern of flawed reasoning can be found in the following statement:

Since late-Renaissance paintings were mostly in oil, the Venetian late-Renaissance painter Arnoldi, whose works are now lost, probably painted in oil.

This statement, like the argument about Carpaccio, assumes that a single characteristic (late-Renaissance paintings being mostly in oil) is sufficient to attribute a specific characteristic (painting in oil) to a particular artist (Arnoldi) without considering other factors or possibilities.

LSAC:

The anonymous painting St. Sebastian is probably an early Florentine painting since it is in tempera, and most early Florentine paintings were in tempera.