Supreme Court Says 'No' to AI Hallucinations, Quashes Tribunal Orders Over Fake Citations

In a landmark judgment, the Supreme Court of India has set aside orders passed by the National Company Law Tribunal (NCLT) and the National Company Law Appellate Tribunal (NCLAT) after discovering that the judgments relied on AI-generated fictitious legal citations. 

The Court warned that artificial intelligence can assist legal research but cannot replace judicial reasoning or professional verification. It has also directed the Bar Council of India (BCI) to examine the use of AI in legal practice and recommend appropriate safeguards.

Key Highlights of NEWS

• The Supreme Court quashed NCLT and NCLAT orders in the Essel Infraprojects insolvency case.
• The Court found that the tribunals had relied on AI-generated fictitious legal citations.
• Judges warned that AI-generated "hallucinations" could seriously undermine the administration of justice.
• The Court stressed that judges and lawyers remain responsible for verifying every legal citation.
• The Bar Council of India has been asked to examine AI use in legal practice and suggest safeguards.
• The judgment is likely to become a key reference point for responsible AI adoption in India's legal ecosystem.

AI can speed up legal research, but understanding its limitations is just as important. Explore upGrad's Artificial Intelligence courses to build practical AI skills and learn responsible AI development.

Why Did the Supreme Court Set Aside the Tribunal Orders?

This wasn't a routine procedural error.

The Supreme Court found that the NCLT and NCLAT orders in the Essel Infraprojects insolvency case referred to judicial precedents that simply didn't exist. During its review, the Court observed that several cited cases were fictitious and appeared to be the result of AI-generated legal research rather than authentic judicial precedents.

That finding became central to the judgment.

Instead of merely correcting the citations, the Supreme Court chose to set aside both tribunal orders, making it one of the strongest judicial responses in India to the misuse of artificial intelligence in legal proceedings.

The Problem Wasn't AI Alone

The Court didn't blame artificial intelligence by itself.

It pointed to the failure to independently verify information before relying on it in judicial orders. AI tools can generate convincing text, but they can also produce fabricated cases, incorrect quotations, or non-existent legal references, a phenomenon commonly known as AI hallucination.

For the Court, the issue wasn't technological capability.

It was human responsibility.

Every Citation Must Be Verified

The judgment makes one point unmistakably clear.

Judges, lawyers, and legal professionals remain responsible for verifying every authority they rely upon, regardless of whether the information comes from books, databases, or AI-powered research tools. Technology may assist legal work, but it cannot replace professional diligence.

What Did the Supreme Court Say About AI Hallucinations?

The Court didn't reject artificial intelligence.

It rejected blind reliance on it.

While acknowledging that AI can improve efficiency and assist with research, the Bench warned that unchecked dependence on AI-generated content poses serious risks to the justice delivery system. 

The judgment notes that AI-generated hallucinations may appear authentic even when they contain entirely fabricated legal references.

That makes independent verification essential.

AI Can Assist. It Can't Replace Judicial Reasoning.

The Supreme Court recognized that AI tools are becoming increasingly common in legal research and document preparation.

However, it emphasized that these technologies should remain assistive tools, not decision-makers. Judicial reasoning requires careful analysis of facts, applicable law, and binding precedents, tasks that ultimately demand human judgment and accountability.

The Court's message was practical rather than anti-technology.

Use AI.

But verify everything.

An "Invisible, Insidious and Catastrophic" Risk

In one of the judgment's strongest observations, the Bench described unchecked AI hallucinations in judicial work as an "invisible, insidious and catastrophic" threat if left unaddressed. 

The warning wasn't directed at AI development itself, but at the possibility of fabricated information quietly entering legal proceedings without proper scrutiny.

That concern extends beyond one insolvency dispute.

It raises broader questions about how AI should be used in professions where accuracy is non-negotiable.

The Supreme Court Didn't Stop at a Warning

The judgment goes beyond criticism.

It also proposes a path forward.

Recognizing that AI tools are already finding their way into legal practice, the Supreme Court directed the Bar Council of India (BCI) to examine the issue and consider developing appropriate safeguards for the profession.

The Bar Council of India Has Been Asked to Step In

The Court has asked the BCI to constitute an expert committee to study the use of artificial intelligence in legal practice and adjudication. The committee has been asked to examine both the opportunities and the risks associated with AI before recommending suitable guidelines.

That's a significant development.

Instead of discouraging AI adoption altogether, the Court is encouraging structured governance that allows technology to be used responsibly.

The Focus Is Responsible Adoption

The judgment doesn't suggest that AI should be banned from legal work.

Rather, it reinforces that technology must operate within established professional standards. Any AI-generated output used in court proceedings must be independently verified before it forms part of legal arguments or judicial reasoning.

Using AI effectively starts with understanding how it works, not just what it generates. Build hands-on expertise in Machine Learning, Generative AI, MLOps, and Agentic AI with upGrad's Executive Diploma in Machine Learning & AI with MLOps, Gen AI & Agentic AI.

Why This Judgment Matters Beyond One Insolvency Case

At first glance, this may appear to be a dispute involving a single insolvency proceeding.

It isn't.

The Supreme Court's observations extend far beyond the Essel Infraprojects case. As AI tools become increasingly common across legal research, drafting, compliance, and litigation, the judgment establishes an important principle. Technology can improve efficiency, but responsibility for accuracy always rests with the human using it.

That message isn't limited to the legal profession.

Human Verification Is Now the Central Message

Artificial intelligence can summarize documents, identify precedents, draft contracts, and answer legal questions within seconds.

That's useful.

It also comes with a well-known limitation. Generative AI models can sometimes produce information that appears accurate but has no factual basis. In legal proceedings, even a single fabricated citation can affect arguments, judicial reasoning, and ultimately the administration of justice.

The Supreme Court's judgment reinforces that AI-generated content should never be treated as authoritative without independent verification.

What Does This Mean for Lawyers and Law Firms?

The legal profession is already experimenting with AI-powered research tools.

That trend is unlikely to slow down.

Instead, this judgment may encourage firms to adopt stronger internal review processes before AI-generated material is used in client advice, court filings, or legal opinions.

AI Can Speed Up Research, Not Replace Due Diligence

For legal professionals, AI remains a valuable productivity tool.

It can summarize lengthy judgments, identify relevant statutes, organize case law, and assist in drafting. Those benefits remain intact.

What changes is the expectation around verification.

Lawyers may increasingly be required to document how AI-assisted research was reviewed before relying on it in professional practice.

Governance Could Become a Competitive Advantage

As AI adoption grows, firms that establish clear governance policies may inspire greater confidence among clients and regulators.

That includes defining where AI can be used, who reviews AI-generated outputs, and how factual accuracy is verified before documents are finalized.

Expert Insight | This Isn't an AI Problem. It's a Professional Responsibility Problem.

The Supreme Court's judgment sends a broader message.

AI doesn't remove accountability.

It shifts greater responsibility onto the people using it.

Over the past two years, conversations around artificial intelligence have largely focused on how capable these systems have become. This ruling redirects attention to a more important question. 

How should professionals use AI responsibly in environments where mistakes can have legal, financial, or societal consequences?

That's a conversation every industry will eventually have.

Whether it's law, healthcare, banking, education, or public administration, AI-generated information cannot replace human judgment when decisions carry real-world consequences.

The Bigger Story Isn't AI Hallucinations

It's the future of responsible AI adoption.

India's Supreme Court hasn't rejected artificial intelligence.

It has drawn a clear boundary.

AI may assist professionals with research, drafting, and analysis, but it cannot replace human verification, legal reasoning, or professional accountability. As AI becomes part of everyday work, organizations will need governance frameworks that define where AI can help, where human review is mandatory, and how risks are managed before decisions are made.

That's likely to become one of the defining themes of enterprise AI adoption over the next few years.

Conclusion

The Supreme Court's decision is more than a ruling on incorrect legal citations. It is one of India's clearest judicial statements on the responsible use of artificial intelligence. 

By setting aside tribunal orders based on fictitious AI-generated references and directing the Bar Council of India to examine AI use in legal practice, the Court has emphasized a simple but important principle.

AI can assist legal professionals, but accuracy, accountability, and judicial reasoning must always remain human responsibilities.