Legal filings may contain concealed prompts that manipulate AI tools used by courts, raising risks of ‘prompt injection’ attacks that distort judicial research and decision-making.
Emerging Threat of Prompt Injection in Judicial Proceedings
As Indian courts increasingly adopt AI tools for legal research and document analysis, the risk of ‘prompt injection’—where malicious instructions are embedded within pleadings to manipulate AI outputs—has emerged as a critical concern. In a recent analysis, Delhi-based IP litigator Rohit Pradhan highlights how seemingly innocuous text in petitions could trigger unintended AI behaviour, potentially skewing summarization or citation recommendations.
Unlike AI ‘hallucinations,’ which stem from model limitations, prompt injection is a deliberate exploit. For instance, hidden textual patterns could instruct an AI system to ignore certain authorities or emphasize specific arguments. While e-filing systems currently lack safeguards against such inputs, Pradhan recommends technical checks at the point of upload and judicial awareness of AI-assisted research limitations.
To mitigate risk, courts may need to adopt input sanitization protocols and require disclosures of AI-assisted drafting. For practitioners, this underscores the ethical duty to ensure filings do not contain manipulative constructs, even if generated inadvertently. As AI integration deepens, procedural integrity demands proactive policy development.
