niccokunzmann/first_timer_scraper repository overview

⁠first_timer_scraper

Visit the daily build⁠.

This is inspired by first-timers-only issues: How can we make it possible for new-comers to contribute to a project.

This web service tries to solve this by looking at the data:

• What did first-timers contributed to?
• Did they continue to contribute?
• Which issues did they solve?
• Which projects are best at welcoming new contributors?

⁠First Timer Contributions

To find first timer pull-requests, we look at organizations and their repositories.

• What are first timer repositories?

  • They MUST have at least two people contributing
  • They MUST have pull requests

• What is a first-timer pull-request?

  • They MUST have at least one commit
  • In the repository history:
    • The issue commit MUST be preceded by commits of the author
    • At least one "other author" MUST have contributed before
    • There MUST NOT be a commit of the author before the "other author"
  • They may be on any branch.

• What is a first-timer issue? There is no automatic linking between the pull-request and the issue. Thus, we must assume that one of the following takes place:

  • There is a public web page linked in the pull-request. This is the case with external issue trackers like with Django. We may assume certain issue trackers.
  • The issue is referenced as GitHub issue in the pull-request.
  • The issue is referenced by commits by the author preceding the merge commit.

⁠Implementation

For each organization submitted:

1. submit all repositories

For each submitted repository

1. clone
2. check if it can be a first-timer repository
3. extract for all commits
   • author email
   • author name
   • commit hash
   • time
4. Get all pull-requests
   • Get the first commit
   • Get the author, email
   • Get the GitHub user issuing this pull-request
5. find first-timer pull-requests

⁠API

ENDING is either .html or .json.

• General
  • GET /
    Show a description of the project.
    • Link to this repository#readme.
    • Link to download the code /source.
    • How to contribute.
    • What this is about.
  • GET /source
    Get the source code as a zip file.
• GitHub Authentication:
  • GET /auth Show a form to register a new authentication.
  • POST /auth Add username and password to those usable to scrape GitHub. They will be tried and removed if invalid.
• Organization
  • GET /organizations<ENDING>
    schema⁠ example responses⁠
    List all organizations with links to their statuses.
    Users may be listed among them. Arguments: offset=0&limit=10
    • offset is the number of organizations to skip
    • limit is the number of organizations to display
  • GET /organization/<organization><ENDING>
    schema⁠ example responses⁠
    Get an organization and its status. Same as the element of /organizations.json Ending:
    • json
    • html
    • svg
  • POST /organization<ENDING>
    Submit an organization for scraping. This shows an html page with a link to the status of the organization.
• Repository
  • GET /repositories<ENDING>
    schema⁠ example responses⁠
    List all repositories with links to their statuses. See below for the content of the list.
  • GET /repository/<organization>/<repository><ENDING>
    schema⁠ example responses⁠
    Get the repository and its status.
    Ending:
    • json
    • html
    • svg
  • POST /repository
    Submit a repository for scraping
• User
  • GET /users<ENDING>
    Same as GET /organizations<ENDING>.
  • GET /user/<user><ENDING>
    Same as GET /organization/<user><ENDING>.
  • POST /user/<user>
    Trace back the user's repositories to their origins. Submit all found organizations.
• Issues
  • TODO: do we need an issue endpoint?

⁠Minimal definitions

When objects are defined, they contain minimal definitions. They can be used to infer the most important data and find the full data. E.g. repository["urls"]["json"] always points to the repository endpoint.

repository, user, organization and issue have this in common:

{
  "name": "<name>",
  "urls": {
    "html": "<html_url>",
    "json": "<json_url>",
    "github_html": "<github_html_url>",
    "github_api": "<github_json_url>",
  },
  "last_update": "<start_time>",
  }
}

• last_update_time times are given like this: 2011-01-26T19:01:12Z in UTC. You can parse it with time.strptime("2011-01-26T19:01:12Z", "%Y-%m-%dT%H:%M:%SZ")

• Additionally repository, issue always define

  {
    "full_name": "<full_name>"
  }
  

  Copy

⁠Command Line

python3 -m first_timer_scraper <CACHE_FOLDER> <SECRETS_FOLDER> <MODEL_FOLDER>

• CACHE_FOLDER is the folder where the scraped data is stored. This is the cache. It is totally ok to remove all this data.
• SECRETS_FOLDER is the folder where the secrets are stored. These are the secrets to access the GitHub API.
• MODEL_FOLDER is the storage place of the model/the data base this is built. If you delete this, you can start scraping anew.

⁠Installation

You need to install Python 3⁠ and pip. Under Ubuntu, you can do this:

sudo apt-get -y install python3 python3-pip

To install all required packages, execute

pip3 install --user -r requirements.txt

⁠Windows

py -3 -m pip install --user -r requirements.txt
py -3 -m first_timer_scraper.app data secret model

⁠Docker

This runs the docker container:

docker run --rm                                \
           -p 8080:8080                        \
           -v "secret:/app/secret"             \
           -v "model:/app/model"               \
           niccokunzmann/first_timer_scraper

The parameters have the following meaning:

• The API of the first-timer-scraper is available as http://localhost:8080⁠.
• The secret data including passwords are stored in the folder ./secret.
• The model data is stored in the folder ./model
• --rm is for development purposes. It removes the container and all the cache when the container stops.

When you ran the command, you can visit http://localhost:8080⁠, submit credentials and scrape repositories.

You can build the Docker image like this:

docker build . -t niccokunzmann/first_timer_scraper

⁠Deployment

Deployment takes up to 24 hours from a merge into master:

1. A commit is added to master⁠.
2. An automated build⁠ is started.
3. The application is deployed⁠ around 4:00 GMT

⁠Data Model

The data model describes what is saved when scraped.

{
  "loklak": {
    "repos": {
      "loklak_server": {
        "first_timer_prs": {
          "112": "contributor1"
        },
        "last_update_requested": "2011-01-26T19:01:12Z"
      }
    },
    "last_update_requested": "2011-01-26T19:01:12Z",
    "first_timer_prs":{}
  },
  "contributor1": {
    "first_timer_prs": {
      "loklak/loklak_server": {
        "created_at" : "2011-01-26T19:01:12Z"
        "number": 112 // lowest number wins
        "last_update_requested": "2011-01-26T19:01:12Z"
      }
    },
    "last_update_requested": "2011-01-26T19:01:12Z",
    "repos": {}
  }
}

⁠Further Reading

• GitHub API:
  • Rate Limiting: https://developer.github.com/v3/#rate-limiting⁠
• PyGithub: https://github.com/PyGithub/PyGithub/⁠