houfu/pdpc-decisions repository overview

⁠pdpc-decisions

This package contains utilities which allow you to create a corpus of decisions from the Personal Data Protection Commission of Singapore's Data Protection Enforcement Cases⁠.

The primary use of such a corpus is for studying, possibly using data science tools such as natural language processing.

It currently has the following features:

• Visit the Personal Data Protection Commission of Singapore's Data Protection Enforcement Cases⁠ and compile a table of decisions with information from the summaries provided by the PDPC for each case.
• Save this table of decisions as CSV
• Download all the PDF files of the decisions from the PDPC's website. If the decision is not a PDF, collects the information provided on the decision web page and saves it as a text file.
• Convert the PDF files into text files

⁠Features provided by scraper

• Published date
• Respondent
• Title
• Summary
• URL of PDF of decision

The features are discovered by passing --extras to the command.

• [Extras] Citation
• [Extras] Basic enforcement information (Financial penalty, warning, directions)
• [Extras] References (referred by, referring to)

⁠What pdpc-decisions uses

• Python 3
• PDF Miner
• Selenium
• Chrome
• spaCy

⁠Installation

⁠Docker Image

I dockerised the application for my personal ease of use. It is probably the easiest and most straight-forward way to use the application and I recommend it too. The dockerised application also contains all pre-requisites so there is no need for any manual installs.

You need to have docker installed. Pull the image from docker hub⁠.

docker pull houfu/pdpc-decisions

After that you can run the image and pass commands and arguments to it. For example, if you would like the application to do all actions.

docker run houfu/pdpc-decisions all

This isn't clever because downloads will be stored in the docker image and not easily accessed. Bind a volume in your filesystem and use the --root option to direct the application to save the files there. For example:

docker run \ 
  --mount type=bind,source="$(pwd)"/target,target=/code/download \ # Target directory must exist!
  houfu/pdpc-decisions \
  all \
  --root /code/download/

⁠Local install

• Install via PIP

pip install pdpc-decisions

• Once the package is installed, used the command line tool pdpc-decisions to use the script.

• If necessary, install Chrome⁠ and ChromeDriver⁠ for Selenium to work.

The main entry point for the script is pdpcdecision.py

⁠Usage

The script accepts the following actions and options:

Accepts the following actions.

"all" Does all the actions (scraping the website, saving a csv, downloading all files and creating a corpus).

"corpus" Converts PDF format of decisions into plain text files.

"csv" Save the items gathered by the scraper as a csv file.

"files" Downloads all the decisions from the PDPC website into a folder.

Options:

--csv FILE Filename for saving the items gathered by scraper as a csv file. [default: scrape_results.csv]

--download DIRECTORY Destination folder for downloads of all PDF/web pages of PDPC decisions [default: download/]

--corpus DIRECTORY Destination folder for PDPC decisions converted to text files [default: corpus/] -r, --root DIRECTORY Root directory for downloads and files [default: Your current working directory]

--extras/--no-extras Add extra features to the data collected. This increases processing time. This feature is ignored if action is files or downloads. (Experimental and requires reading of actual decisions) [default: False, '--no-extras']

--extra-corpus/--no-extra-corpus Enable experimental features for corpus. This increases processing time.

--verbose Verbose output

--help Show this message and exit.

⁠Contact

Feel free to let me have your suggestions, comments or issues using the issue tracker or by emailing me⁠.

It would also be nice to hear how you have used this corpus by using the above contacts.