Japanese tort-case dataset for rationale-supported legal judgment prediction

Legal information processing aims to provide computational aid in legal procedures, including predicting the outcome of a case through legal judgment prediction (LJP). LJP is beneficial to both legal professionals and the general public. It allows them to predict litigation outcomes in advance, and they can take better actions based on expectations. For example, they can get faster conciliation and smoother negotiations, resulting in more efficient legal services. LJP also reduces the cost of legal services and improves their access. Easier access to justice is important for those with limited or no access to traditional legal services.

LJP has been a longstanding research topic in artificial intelligence and, like other domains, has adopted machine learning (ML) techniques. The ML techniques require large datasets for training and evaluating ML models. Xiao et al. (2018) proposed a dataset of 2.6M Chinese criminal cases annotated with applicable laws, charges, and prison terms. Chalkidis et al. (2019) presented a dataset of 11.5K cases from the European Court of Human Rights, which is designed for violated article detection and case importance prediction. Katz et al. (2017) constructed a dataset of 28K cases from the Supreme Court of the United States. Semo et al. (2022) released the LJP dataset focused on class action cases in the United States. Chalkidis et al. (2022) proposed a collection of datasets to evaluate model performance across different legal tasks, including LJP tasks in English. In contrast, there is no LJP dataset employing real judgment documents in the Japanese jurisdiction. The LJP tasks and their datasets should be designed to reflect differences in jurisdictions. Against this backdrop, we construct the first dataset of LJP, Japanese Tort-case Dataset (JTD), for the Japanese LJP research.

In JTD, we deal with judgment on civil cases about torts (Civil Code, Art. 709).¹ Tort is an important and popular topic in civil cases. Japanese law affirms a tort as a negligent or intentional infringement of rights or legal interests that cause a plaintiff to suffer loss or harm. In modern society, torts play an important role in disputes on the internet, for example, cases of defamation and privacy infringement on social media. In such cases, tort law is often used to determine liability since there is usually no explicit contract between the parties.

Figure 1 shows an overview of our two tasks: Tort Prediction (TP) and Rationale Extraction (RE). A tort case involves two parties: plaintiffs and defendants. Plaintiffs are claimants of the case, arguing that a defendant’s action is a tort, while defendants contest plaintiffs’ arguments. TP predicts whether a tort is affirmed (T, a Boolean value), given undisputed facts (U) and arguments from both parties (P from plaintiffs and D from defendants). Undisputed facts are not disputed by any parties or agreed upon by both parties. They provide the LJP model with context to validate the parties’ arguments. The final decision on a tort (T) should be based on the arguments that are accepted by the judge. Thus the accepted arguments can be considered rationales for the final decision (T). RE identifies the accepted arguments (\(R^P\) for plaintiffs and \(R^D\) for defendants, both are sequences of Boolean values, denoting accepted arguments as True) in the parties’ arguments (P and D). To summarise, our tasks take (U, P, D) as input and output \((T, R^{P}, R^{D})\).

Figure 2 shows an example of an instance. There is an undisputed fact (U), four claims from the plaintiff (P), and one claim from the defendant (D). A gold standard for the tort prediction task is false (\(T_{gold}\)), meaning the subject of this instance is not considered a tort. Gold standard labels for the rationale extraction task are \(\{True, True, False, False\}\) for the plaintiff (\(R^{P}_{g}\)) and \(\{False\}\) for the defendant (\(R^{D}_{g}\)).

Our main contributions are the following. We propose new tasks for the Japanese LJP, which consist of judicial decision prediction and identification of their rationales. We conducted a large-scale annotation with 41 legal experts. In the annotation, the annotators captured direct causal relations between the court decisions and arguments from the parties, allowing multiple court decisions on multiple subject matters in a case. From the 3477 annotated documents, we built JTD consisting of 7978 tort-related instances. To establish baseline performance with the dataset, we conduct experiments employing hierarchical Transformer architecture and multi-task learning approaches. Moreover, we perform a detailed error analysis by legal experts to identify sources of errors and suggest future research directions. Our dataset will be available for researchers on a website.

ML-based approaches have been popular in LJP research. ML-based systems automatically learn how judges make decisions from a large number of cases (e.g., judgment documents). These models take fact descriptions as input and predict outcomes or relevant laws. European Court of Human Rights cases are popular sources of datasets for LJP (Aletras et al. 2016; Medvedeva et al. 2018; Chalkidis et al. 2019; Valvoda et al. 2023). Galli et al. (2022) constructed a dataset for an outcome prediction task for Italian Value Added Tax decisions. Katz et al. (2017) built a dataset of cases from the Supreme Court of the United States to train their models. Semo et al. (2022) constructed another LJP dataset of class action cases in the United States. LJP on Chinese Criminal cases is another big venue of ML-based LJP models (Luo et al. 2017; Zhong et al. 2018; Hu et al. 2018; Long et al. 2019; Xu et al. 2020). A Japanese dataset for legal tasks is available for Competition on Legal Information Extraction/Entailment (COLIEE) (Rabelo et al. 2020). However, COLIEE is designed for legal entailment and information retrieval on the Japanese bar exam, and its data size is limited.

Due to the lack of a large reliable dataset, the Japanese LJP research has hardly employed the ML-based approach. Instead, the symbolic approach has been popular for Japanese LJP. The symbolic systems predict outcomes of legal reasoning by rules and logic Nitta et al. (1993); Nitta et al. (1993). Although the symbolic systems require human experts’ intervention in development, we can easily interpret their behaviour. PROLEG Satoh et al. (2010) demonstrated that a logic programming system could work for the Japanese legal system. PROLEG is a legal reasoning system based on Prolog, implementing a decision-making theory used in civil litigation in Japan. However, there still needs to be a solution to extracting logical clauses from natural language text Navas-Loro et al. (2018).

The success of deep learning methods in many areas raises concerns about the lack of explainability (Jacovi and Goldberg 2020) behind their output. LJP outputs are expected to align with Justice, and LJP systems should be trusted by society. Therefore, explainability is more important and serious in the legal domain than in other domains. Even if LJP systems are used as assistant tools for legal consulting, they can affect people’s behaviours and indirectly influence their social status and assets. Thus, the LJP system has to explain the reason for predictions. To accommodate the needs, recent LJP studies introduced explanation tasks, including court view generation Ye et al. (2018), rationale paragraph extraction Chalkidis et al. (2021), and case features extraction as rationales Ferro et al. (2019); Branting et al. (2021). Following the prior work, we design our explanation task as rationale extraction similar to Chalkidis et al. (2021), but our extraction task is at span level instead of paragraph level. In addition, our target of rationale extraction is argumentative claims from the parties (e.g., plaintiff’s factual allegations) instead of submitted fact descriptions.

To establish an LJP baseline using JTD, we employ a hierarchical Transformer architecture, which achieves competitive performance in various legal NLP tasks Chalkidis et al. (2022). Our hierarchical Transformer-based models (Fig. 4) are designed to capture both word-level context and span-level context by implementing a span-level Transformer encoder on top of word-level Transformer encoders. We call this model Inter-Span Transformer (IST). IST takes a sequence of spans as input, where each span corresponds to a claim from the plaintiff or defendant side. Its outputs are a single Boolean flag for tort judgment prediction and a sequence of Boolean flags for rationale extraction. These final outputs are obtained through a linear layer just after the output from the span-level encoder. IST accepts party-type embeddings to distinguish between the plaintiff’s claims and the defendant’s claims by assigning different embeddings according to who submitted the claim. IST also considers positions of claims in inputs via position embeddings. As an auxiliary input, IST takes undisputed facts. To inject features from undisputed facts into every encoded claim, IST independently encodes all the concatenated undisputed facts into a single vector, namely fact embeddings. An input representation to the span-level encoder, which corresponds to n-th claim by a plaintiff, is the sum of a claim vector from the word-level encoder, fact embeddings (\(E^{f}\)), party-type embeddings of the plaintiff (\(E^{p}\)), and position embeddings (\(E^{ps}_n\)).

As word-level encoders, we utilise two different types of pretrained BERT (Devlin et al. 2019) models, BERT-base-Japanese (BERTja)⁷ and Japanese-LegalBERT (JLBERT) (Miyazaki et al. 2022). Both BERTja and JLBERT use the same model architecture as the original BERT-base model with 12 layers, 768 dimensions of hidden states, and 12 attention heads. We feed a vector corresponding to the [CLS] token from BERT output into a linear layer and use its output as a claim vector. While BERTja is pretrained only with the Japanese Wikipedia corpus, JLBERT is adapted to the Japanese legal domain. JLBERT is firstly pretrained with the Japanese Wikipedia corpus and then further pretrained with Japanese judgment documents in Civil code cases.

Our JTD also uses Japanese judgment documents as the source of the dataset, and we manually checked the number of overlapping documents between the JLBERT pre-training data and the JTD test set. We found 97.4 % of documents from the JTD test in the pre-training data as well. Nevertheless, we decided to employ JLBERT in our experiments because of the following three reasons: (1) JLBERT is the only available pretrained language model in the Japanese legal domain, (2) JLBERT was pretrained on the proprietary judgment document corpus so that we cannot pre-train our version of JLBERT model pretrained without documents in the JTD test set. (3) As our dataset and tasks are produced with manually annotated judgment documents, masked language modelling cannot steal a look into our two LJP tasks. Our JTD test set is still unseen data for BERTja-based models. We can perform a sanity check by conducting the same experiments with BERTja. By comparing BERTja-based and JLBERT-based models, we avoid overestimating the performance of JLBERT-based models and provide an informative oversight of how the domain-adopted models behave in the Japanese legal judgment tasks.

We conducted a detailed error analysis employing human legal experts on the tort prediction outputs to identify the source of prediction errors.

We clarify limitations and ethical considerations here to prevent misuse of our dataset and misunderstanding of our findings.

The task design for LJP reflects essential elements of the Japanese jurisdiction. However, there are differences from real-world conditions. The inputs in our tasks are only Undisputed Facts, Plaintiff’s Claims, and Defendant’s Claims, which are obtained from judgment documents. In real court cases, there are other documents, including third-party expert opinion letters, detailed evidence documents, and other undisclosed documents (e.g., private information and confidential patent information). They are intermediate outputs or auxiliary inputs in court cases but are still important. This difference limits the model’s capability, as shown by manual error analysis results 6.2. The difference often stems from the limited scope of publicly available information. We suspect such limitation also applies to other jurisdictions that do not disclose the whole court document sets. As many LJP datasets are constructed from only judgment documents, we must recognise that the current LJP task design can be limited and reflect only a part of legal decision-making.

The annotation scheme for our dataset was validated through the preliminary experiment with 25 documents annotated by five legal experts. We note that the number of documents used in the preliminary experiment is lower than that of our final dataset, i.e. 3477 documents, and we did not perform dual-coder annotation for the final dataset. Although we took alternative measures (extended tutorials, and annotator selection via dry-run) to ensure the quality of annotation, we acknowledge that the final dataset may contain inevitable human errors.

We also acknowledge that our dataset is limited in its scope. We must point out that the dataset contains tort-related cases, which is a part of the whole legal judgement in the Japanese Civil jurisdiction. Thus, our findings from the experimental results cannot directly apply to the general Japanese LJP tasks. However, we believe our baseline results provide informative clues to other civil legal judgement tasks. Another limitation is the quality of the annotations. While we have made an effort to keep the annotation quality as high as possible, there is still a chance of errors and misunderstandings by the annotators. Although the agreement study shows reasonable performance, we plan to update and expand our JTD on a long-term schedule continuously.

Given the limitations, we emphasise that one should not rely solely on a model trained on JTD in one’s legal decision-making. Our recommended use case of the JTD-trained models is legal assistance services, but they should not be fully automated. We recommend using the models with legal professionals such as lawyers¹⁰. This human–machine hybrid approach will improve the lead time of case handling and correct potential errors from models’ false predictions.

Moreover, we emphasise that JTD is not intended to develop a judgment system. In other words, we did not intend to replace judges or courts with JTD-trained models. Important intellectual activities of judges include the conceptualisation of new rights and the update of law interpretations in response to age. However, the current LJP tasks in JTD do not cover such aspects. Therefore, we argue only human judges should take responsibility for such authority.

We proposed a novel dataset for the Japanese legal judgment prediction featuring tort cases under the Japanese Civil Code. We specifically targeted two tasks: tort prediction and rational extraction. This is the first dataset consisting of real Japanese court cases with human expert annotation. Although we still have to continue to improve the annotation quality, our annotation procedure will be a good starting point to produce reliable datasets in the Japanese legal domain.

We conducted a feasibility study of the two tasks with the baseline models. We also compared the performance between single-task and multi-task approaches. The baseline experiments confirmed the feasibility of our tasks, and we found that the multi-task approach performed better. Moreover, we manually analysed the outputs from the IST models with experienced legal specialists to diagnose the errors. The results suggest that there are difficult cases even for human experts.

Our dataset and extended expert analysis only made these novel findings possible. We believe our dataset provides a useful resource in legal NLP for not just Japanese LJP research but also comparative experiments and analysis across different languages and jurisdictions.

In future work, we plan to extend the size of our dataset and increase the types of cases in addition to tort cases. Additionally, adapting our annotation scheme and tasks to other jurisdictions is an interesting direction for further research.

Our JTD dataset is available for non-commercial academic research purposes. We also share the original document set used in JTD construction. All the files are available as “Japanese Tort-case Dataset” at https://​www.​gsk.​or.​jp/​catalog/​gsk2024-a.