Automate the case review on legal case documents and find the most critical cases using network analysis

Last update: Dec 28, 2022

Overview

Automation on Legal Court Cases Review

This project is to automate the case review on legal case documents and find the most critical cases using network analysis.

Short write-up

Affiliation: Institute for Social and Economic Research and Policy, Columbia University

Project Information:

Keywords: Automation, PDF parse, String Extraction, Network Analysis

Software:

Python : pdfminer, LexNLP, nltk sklearn
R: igraph

Scope:

Parse court documents, extract citations from raw text.
Build citation network, identify important cases in the network.
Extract judge's opinion text and meta information including opinion author, court, decision.
Model training to predict court decision based on opinion text.

Polit Study on 159 Legal Court Documents (in `pilot_159` folder)

1. Process PDF documents using `Python`

Ipython Notebook	Description
`1.Extraction by LexNLP.ipynb`	Extract meta inforation use `LexNLP` package.
`2.Layer Analysis on Sigle File. ipynb`	Use `pdfminer` to extract the raw text and the paragraph segamentation in the PDF document.
`3.Patent Position by Layer.ipynb`	Identify the position of patent number in extracted layers from PDF.
`4.Opinion and Author by Layer.ipynb`	Extract opinion text, author, decisions from the layers list.
`5.Wrap up to Meta Data.ipynb`	Store extracted meta data to `.json` or `.csv`
`6.Visualize citation frequency.ipynb`	Bar plot of the citation frequencies

2. Data: Parse PDF documents via `Python`

These datasets are NOT included in this public repository for intellectual property and privacy concern

File
`pdf2text159.json`	A dictionary of 3 list: `file_name`, `raw_text`, `layers`.
`cite_edge159.csv`	Edge list of citation network
`cite_node159.csv`	Meta information of each case: `case_number`, `court`, `dates`
`reference_extract.csv`	cited cases in a list for every case, untidy format for analysis
`citation159.csv`	file citation pair, tidy format for calculation
`regulation159.csv`	file regulation pair, tidy format for calculation

3. Analyze and Visualize using `R`

File
`Calculate Citation Frequency.Rmd`	Analyze `reference_extract.csv`
`Citation Network.Rmd`	Analyze `cite_edge159`

4. Visulization Chart Sample

Citation Frequency

Citation Network

Network Visulization and Predictive Modeling on 854 Legal Court Cases (in `Extraction_Modelling` folder)

1. Extract opinion and meta information from raw text data

`.ipynb` notebook	Description
`Full Dataset Merge.ipynb`	Merge the 854 cases dataset
`Edge and Node List.ipynb`	Create edge and node list
`Full Extractions.ipynb`	Extract author, judge panel, opinion text
`Clean Opinion Text.ipynb`	Remove references and special characters in opinion text

2. Datasets

These datasets are NOT included in this public repository for intellectual property and privacy concern

Dataset	Description
`amy_cases.json`	large dictionary {file name: raw text} for 854 cases, from Lilian's PDF parsing
`full_name_text.json`	convert `amy_cases.json` key value pair to two list: `file_name`, `raw_text`
`cite_edge.csv`	edge list of citation
`cite_node.csv`	node list contains `case_code`, `case_name`, `court_from`, `court_type`
`extraction854.csv`	full extractions include `case_code`, `case_name`, `court_from`, `court_type`, `result`, `author`, `judge_panel`
`decision_text.json`	json file include `author`, `decision`(result of the case), `opinion` (opinion text), `cleaned_text` (cleaned opinion text)
`cleaned_text.csv`	csv file contains allt the cleaned text
`predict_data.csv`	cleaned dataset for NLP modeling predict court decision

3. Visulization using R

R markdown file
`Full Network Graph.Rmd`	draw the full citation network
`Citation Betwwen Nodes.Rmd`	draw citation between all the available cases
`Clean Data For Predictive Modelling.rmd`	clean text data for predictive modeling

Interactive Graph

Play with Interactive Graph

Full Citation Network (all cases and cited cases)

Citation Between Available Cases

4. Predictive Modeling using Python

`ipynb` notebook
`NLP Predictive Modeling.ipynb`	Try different preprocessing, and build a logistic regression to predict court decision.

Automate the case review on legal case documents and find the most critical cases using network analysis

Related tags

Overview

Automation on Legal Court Cases Review

Project Information:

Polit Study on 159 Legal Court Documents (in pilot_159 folder)

1. Process PDF documents using Python

2. Data: Parse PDF documents via Python

3. Analyze and Visualize using R

4. Visulization Chart Sample

Citation Frequency

Citation Network

Network Visulization and Predictive Modeling on 854 Legal Court Cases (in Extraction_Modelling folder)

1. Extract opinion and meta information from raw text data

2. Datasets

3. Visulization using R

Interactive Graph

Full Citation Network (all cases and cited cases)

Citation Between Available Cases

4. Predictive Modeling using Python

Visulization of the Bi-gram (words) with the strongest coefficient

Owner

Yi Yin

CLAHE Contrast Limited Adaptive Histogram Equalization

Moscow DEG 2021 elections plots

DrawBot lets you draw images taken from the internet on Skribbl.io, Gartic Phone and Paint

Some examples with MatPlotLib library in Python

A GUI for Pandas DataFrames

a plottling library for python, based on D3

Easily convert matplotlib plots from Python into interactive Leaflet web maps.

Python Package for CanvasXpress JS Visualization Tools

demir.ai Dataset Operations

A gui application to visualize various sorting algorithms using pure python.

Visualizations for machine learning datasets

Small U-Net for vehicle detection

Gaphas is the diagramming widget library for Python.

Here are my graphs for hw_02

Smoking Simulation is an app to simulate the spreading of smokers and non-smokers, their interactions and population during certain amount of time.

Quickly and accurately render even the largest data.

Generate the report for OCULTest.

nvitop, an interactive NVIDIA-GPU process viewer, the one-stop solution for GPU process management

Editor and Presenter for Manim Generated Content.

The implementation of the paper "HIST: A Graph-based Framework for Stock Trend Forecasting via Mining Concept-Oriented Shared Information".

Polit Study on 159 Legal Court Documents (in `pilot_159` folder)

1. Process PDF documents using `Python`

2. Data: Parse PDF documents via `Python`

3. Analyze and Visualize using `R`

Network Visulization and Predictive Modeling on 854 Legal Court Cases (in `Extraction_Modelling` folder)