Predictive pattern analysis in administrative and tax litigation through generative AI and regular expressions

Abstract

Text classification is the task of assigning a piece of text to an appropriate category, with the set of possible categories varying by domain. In this study, we evaluate ChatGPT’s accuracy in classifying judicial rulings issued by an Argentine court, using prompts based on regular expression matches. We compare its performance to two alternative approaches: (1) classification based on natural language descriptions of the categories, and (2) a traditional regex-based algorithm. The regex patterns were developed by domain experts employed at the partner organization, and all experiments were conducted in Spanish. Our results show that ChatGPT achieves high accuracy when prompted with both natural language descriptions and regex-based category identifications. However, generating outputs with ChatGPT is significantly more time-consuming. Despite this limitation, the model’s accessible interface and minimal technical requirements make it a promising tool for automating document classification. Furthermore, the use of regex-based prompts enables human oversight and active involvement in the classification process—an advantage in domains where interpretability and accountability are critical. All code, prompts, and evaluation materials are publicly available at <https://github.com/FAIR-IALAB-UBA/ai-regex>.