Sign inSign up

code4sa/gazettescrape-base-priv

By code4sa

Updated almost 9 years ago

recreate

Image
0

2.2K

code4sa/gazettescrape-base-priv repository overview

gazettescrape - South African Government Gazette scraper

Running tests

pip install nose
nosetests

Non-python dependencies

  • qpdf
  • pdfinfo

Minimal Configuration

Depends on the following DB URI being accessible or overridden to another

GAZETTE_DB_URI = 'postgres://gazettes@localhost/gazettes'

Scraping

Government Printing Works (GPW) spider

Optional Arguments:

start_url=http://www.gpwonline.co.za/Gazettes/Pages/Published-Liquor-Licenses.aspx

Override default start URL set with a single specific URL

e.g. locally

scrapy crawl -a start_url='http://www.gpwonline.co.za/Gazettes/Pages/Published-Liquor-Licenses.aspx' gpw

or

scrapy crawl gpw
Western Cape Province
scrapy crawl western_cape
Google Drive
python gazettes/archive.py --scrape-gdrive

DB Migrations

Locally
PYTHONPATH=. alembic revision --autogenerate -m "I changed field something something"
PYTHONPATH=. alembic upgrade +1
Production/other environments

Copy alembic.ini and refer to that.

PYTHONPATH=. alembic -c alembic_prod.ini upgrade +1

Deployment

To scrapinghub

Deploy the spiders to scrapinghub

shub deploy 79283

Extended configuration

To store files in S3, set/override the following settings:

FILES_STORE = "s3://code4sa-gazettes/scrape/"

AWS_ACCESS_KEY_ID = "..."
AWS_SECRET_ACCESS_KEY = "..."

To store the item feed in S3, set/override the following settings:

FEED_URI = s3://code4sa-gazettes/scrape-feed/spider-%(name)s/start-%(time)s.json
FEED_FORMAT = jsonlines

Archival

Archival figures out the unique ID of the gazettes and archives them under human-readable paths in the configured storage location.

Locally/dev
source env-example
PYTHONPATH=. python gazettes/archive.py --help

PYTHONPATH=. python gazettes/archive.py --archive-gazettes

Find what's missing

The current strategy for finding what's missing is to take each series of issue numbers we have, find the gaps in the range we expect to have. We look for gaps between the first issue number we expect to have, and the largest issue number we currently have in that series. This doesn't identify gaps at the end of a series which is primarily a limitation for regularly-cycled series like Gauteng and Free State Provincial gazettes.

To find what's missing, run PYTHONPATH=. python gazettes/missing.py which will write a file missing.csv to the current working directory indicating missing contiguous groups of issue numbers per issue number series.

Production
dokku apps:create gazettescrape
dokku docker-options:add gazettescrape run,deploy "-v /tmp/gazettes-tmp:/tmp/gazettes-tmp"
dokku config:set WEB_SCRAPE_STORE_URI="s3://aws-access-key-id:aws-access-key@code4sa-gazettes/scrape/" \
                 ARCHIVE_STORE_URI="s3://aws-access-key-id:[email protected]/" \
                 DB_URI="postgres://gazettes:password@hostname/gazettes" \
                 LOG_LEVEL="INFO" \
                 GOOGLE_EMAIL=...service-account-email... \
                 GOOGLE_PRIVATE_KEY="...pkey as given in the string in the json credentials replacing each \n with _ because \ is complicated when many things are clever and escape and do weird shit...."
# ... push repository to dokku host ...

And add the followin crontab entry to run after the scrapinghub jobs are probably finished, or run manually

0 2 * * * /usr/local/bin/dokku --rm run gazettescrape python gazettes/archive.py --scrape-gdrive --archive-gazettes --write-index --write-atom --rebuild-opengazettes 2>&1 >/var/log/gazettescrape/archive.log

Gazette types

  • National
    • Regulation Gazette
      • "This gazette contains information that government wishes to communicate specifically pertaining to regulations only"
      • e.g.
        • Mineral Resources, Department of
          • Mine Health and Safety Act: Notice for nominations of members of Mine Health and Safety Council
        • South African Revenue Service
          • Customs and Excise Act, 1964: Amendment of Schedule No. 1 (No. 1/1/1543)
        • Trade and Industry, Department of
          • Private Security Industry Regulation Act: Amendment to the Regulations
          • Private Security Industry Regulation Act: Notice calling for public comment
          • Legal Metrology Act: Amendments to the compulsory specification for Lampholders
    • Government Gazette
      • Government Notice
        • e.g.
          • ALTERATION OF SURNAMES IN TERMS OF SECTION 26 OF THE BIRTHS AND DEATHS REGISTRATION ACT, 1992 (ACT NO. 51 OF 1992)
            • Full old name
            • SA ID Number
            • Address
            • New surname
          • ALTERATION OF FORENAMES IN TERMS OF SECTION 24 OF THE BIRTHS AND DEATHS REGISTRATION ACT, 1992 (ACT NO. 51 OF 1992)
            • Full old name
            • SA ID Number
            • Address
            • New forenames
          • Revised Draft Policy and Bill on Preservation and Development of Agricultural Land :Invitation to consultation workshops and public comments
          • National Heritage Resources Act (25/1999) :Declaration of the Liliesleaf Farm as National Heritage Sites
      • General Notice
        • e.g.
          • Agricultural Products Standards Act (119/1990) :Standards and requirements regarding control of the export of peaches and nectarines: Amendment
          • National Environmental Management: Biodiversity Act, (10/2004) :Norms and Standards for the Management of Damage-Causing Animals in South Africa
      • Board Notice
        • e.g.
          • South African Council for the Architectural Profession Act (44/2000) :Finding and sanction imposed by council: Alleged improper conduct: Michael Fakude
          • Auditing Profession Act (26/2005) :Call for Nomination of Persons to Serve on the Independent Regulatory Board for Auditors (IRBA)
      • Proclamations
        • e.g.
          • Consitution of the Republic of South Africa, 1996: Establishment of a Commission of Enquiry into the tragic incident at or near the area commonly known as the Marikana Mine in Rusternburg, North West Province, South Africa
          • Tax Administration Act (28/2011): Determination of the dates on which certain provisions of the Act must come into operation
    • Extraordinary Gazette
  • Provincial
    • Provincial Gazette
      • Western Cape
        • Provincial Notices
          • Assented Acts
          • Regulations and request for comment
          • Invitation for nominations to boards
          • Removal of restrictions
        • Local Authorities
          • Rezoning notice
          • Planning/use applications
          • Tarrif changes
      • Free State
        • Provincial Notices
          • School resource allocation table for following year
          • Allocation of financial assistance to municipalities
          • Human settlements accreditation to municipalities
          • Invitation for Nomination of Members to serve on the Free State Rental HousingTribunal
          • Changes to land use restrictions, zoning
          • Conversion of certain rights into leasehold/ownership
          • Notice of application for public driving permits
          • Notice of application for liquor license
          • Notice of application for gambling machine operator license
          • Notice of proposed amendments to regulations e.g. under Free State Gamling and Liquor Act, 2010
          • Notice of establishment of municipality
          • Widthdrawal of declaration of wildlife sanctuary
          • Designation of a Fire Station
        • Notices
    • Provincial Liquour License Gazette
    • Provincial Separate Gazette
    • Provincial Extraordinary Gazettes
      • e.g. demarcation changes
  • Liquor License Application Notices
    • Some dedicated ones per province:
      • Northern Cape
      • Gauteng
    • Some combined in one national
      • Limpopo
      • KwaZulu-Natal
      • Mpumalanga
      • North West
    • Some among their provincial Gazettes
      • Eastern Cape
      • Free State
      • Western Cape
    • Roughly they contain
      • Name, street and postal address of applicant
      • Kind of license applied for
      • Kind of liquor to be sold
      • Name under which business is to be conducted and full address of premises
      • Determination, consent, approval or authority applied for
  • Legal Gazette A
    • Business Notices
      • province sections
    • Company Notices
      • province sections
    • Orders of the Court
      • province sections
    • Supressions and Discharge of Petitions
    • General
      • province sections
    • Administration of Estates Acts Notices
      • Form J 297: Election of executors, curators and tutors
        • province sections
      • Form J 295: Curators and tutors: Masters' notices
        • province sections
      • Form J 193: Notice to creditors in deceased estates
        • province sections
      • Form J 187: Liquidation and Distribution accounts in deceased estates lying or inspection
        • province sections
    • Insolvency Act and Companies Acts Notices
      • Form J 28: Estates or companies sequestrated or wound up provisionally
      • Form J 29:
        • First meetings of creditors, contributories, members of debenture-holders of sequestrated estates, companies being wound up or placed under provisional judicial management
        • CLOSE CORPORATIONS: First meetings of creditors and members of close corporations being wound up
      • Form 1: Appointment of trustees and liquidators and proof of claims in sequestrated estates or companies being wound up
      • Form 2: Meeting of creditors in sequestrated estates or companies being wound up
      • Form 4: Liquidation accounts and plans of distribution or contribution in sequestrated estates or companies being wound up
      • Form 5: Payment of dividends and collection of contributions in sequestrated estates or companies being wound up
      • Form 6: Applications for rehabilitation
      • Form 7: Notice of trustees
      • Form 8: Dates fixed for creditors to prove claims
      • Form 9: Notices of surrender of a debtor’s estate
  • Legal Gazette B
  • Legal Gazette C
Types as listed in GPW price guide
  • National Gazette
  • Regulation Gazette
  • Petrol Price Gazette
  • Road Carrier Permits
  • Unclaimed Monies (justice, labour or lawyers)
  • Parliament (acts, white paper, green paper)
  • Legal Gazettes A, B anc C
  • Tender Bulletin
  • National Liquor License Gazette
  • Provincial Gazette
  • Provincial Liquor License Gazette
    • GPW publishes these:
      • Gauteng
      • Northern Cape
      • Mpumalanga

Gazette Properties

  • Issue number
  • Title
  • Publication Date
  • Jurisdiction Level ("national" or "provincial")
  • Jurisdiction (e.g. "RSA" or "KZN")
  • Issue type? (e.g. "ordinary" "extraordinary")

Publications

  • http://www.gpwonline.co.za/Gazettes/Pages/Published-Separate-Gazettes.aspx
    • Page linked from
      • Name - (gazette number) (pub day-month) (govt dept or something)
        • e.g. "40132 11-07 National Treasury", "40133 11-07 Icasa"
      • Full Publication date
      • File name - same as link label with _ instead of spaces
    • Gazette contents
      • Cover page
        • Title: "Government Gazette"
        • Subtitle: none
        • Volume Number
        • Publication Date
        • Gazette Number
      • Table of Contents
        • Section Headings
          • e.g. "Board Notices" when filename is "BoardFSB"
          • e.g. "General Notices" when filename is "TradeIndus" or "Icasa"
          • e.g. "Government Notices" when filename is "NatTreas"
        • Notice Number
        • Notice Title
        • Gazette Number
        • Page number in gazette
      • Page header: "Government Gazette"
  • http://www.gpwonline.co.za/Gazettes/Pages/Published-National-Regulation-Gazettes.aspx
    • Page linked from
      • Name
        • e.g. "... NationalRegulation" or "... Labour" or "... NatRegulation"
    • Gazette contents
      • Cover page
        • Title: "Government Gazette"
        • Subtitle: "Regulation Gazette"
        • Volume Number e.g. "Vol. 609"
        • Gazette Number e.g. "No. 39817"
        • Regulation Gazette Number e.g. "No. 10577"
      • Page header: "Government Gazette"
      • Table of Contents
        • Section Headings
          • e.g. "Government Notices"
            • Subsection Headings
              • e.g. "Environmental Affairs, Deptartment of" when name is "NationalRegulation"
              • e.g. "Labour, Department of" when name is "NationalRegulation" or "Labour"
  • http://www.gpwonline.co.za/Gazettes/Pages/Published-National-Government-Gazettes.aspx
  • Page linked from
    • Name
      • e.g. "... NationalGovernment" or "... NationalGazette" Gazette contents
      • cover page
        • Title: "Government Gazette"
        • Subtitle: none
      • Weekly Index
        • description: "For purposes of reference, all Proclamations, Government Notices, General Notices and Board Notices published are included in the following table of contents which thus forms a weekly index. Let yourself be guided by the gazette numbers in the righthand column"
        • Section Headings
          • e.g. "Proclamation"
          • e.g. "Government Notice"
          • e.g. "General Notice"
          • e.g. "Board Notice"
      • Table of Contents

Gazette number Series

  • Each province's gazettes
    • ordinary
    • extraordinary
    • liquour
  • National
    • general
    • separate
    • regulation
    • national liquor
    • legal notices A, B and C
    • Road carrier permits
  • Tender Bulletin

National Gazettes and Tender Bulletins seem to fall under the same volume.

Western Cape Provincial Gazettes don't seem to have volumes - just one big series.

Free state doesn't have volumes and the series wraps on 1 April. Or at least it might have volumes and parts within an issue.

Other notes

  • Sometimes the cover page says "Part 1 of 2". It seems like the second part is included in the same PDF.
  • An example of an official reference to a gazette
    • Government Notice 2432, Government Gazette, Vol. 400, No. 19377 of 19 Octover 1998
    • Republic of South Africa. 2006. Government Tender Bulletin Vol. 488 No. 2424. Pretoria: Government Printing Works

Structured Gazette Archive

Gazettes are to be stored according to the following stucture:

/south-africa/1998/government-gazette-vol-400-no-19377-dated-1998-10-19.pdf
/western-cape/2007/western-cape-provincial-gazette-no-6454-dated-2007-07-27.pdf
/south-africa/2016/tender-bulletin-vol-611-no-2916-dated-2016-05-13.pdf
/free-state/2015/free-state-provincial-gazette-no-79-dated-2015-08-25.pdf
/eastern-cape/2016/eastern-cape-provincial-gazette-vol-23-no-3703-dated-2016-07-12.pdf

That is (jurisdiction)/(publication year)/(unique-id).pdf

Unique IDs for gazettes take the following form:

government-gazette-vol-400-no-19377-dated-1998-10-19
western-cape-provincial-gazette-no-6454-dated-2007-07-27
tender-bulletin-vol-611-no-2916-dated-2016-05-13
free-state-provincial-gazette-no-79-dated-2015-08-25
eastern-cape-provincial-gazette-vol-23-no-3703-dated-2016-07-12

That is:

  • Government Gazettes (published by GWP)
    • government-gazette
    • -(volume and issue number)
    • -dated-(publication date in YYYY-MM-DD form)
  • Provincial Gazettes (published by GPW)
    • (province short name)
    • -provincial-gazette
    • -(volume and issue number)
    • -dated-(publication date in YYYY-MM-DD form)
  • Western Cape (published by Western Cape Provincial Govt)
    • western-cape-provincial-gazette
    • -(issue number)
    • -dated-(publication date in YYYY-MM-DD form)
  • Free State (published by office of the premier of FS)
    • TBC

Publishers

  • North West Province Provincial
    • self - up to 31 January 2006?
    • transferred to GPW as from 1 February 2006
  • Western Cape - self
  • KwaZulu-Natal
    • self - up to 26 April 2007
    • GPW as from 27 April 2007
  • Free State - self

Tag summary

Content type

Image

Digest

Size

189.7 MB

Last updated

almost 9 years ago

docker pull code4sa/gazettescrape-base-priv