cerebriai/ptc repository overview

⁠Flight Scraper Tool

This Python script, scraper.py, is designed to retrieve flight data from various airline websites using web scraping techniques. It populates flight details and prices into an existing database.

⁠Usage

⁠Prerequisites

• Python 3.x
• Required Python packages:
  • pandas
  • requests
  • beautifulsoup4
  • selenium
  • psycopg2 (for PostgreSQL)

⁠Setup

1. Ensure you have PostgreSQL installed and running.
2. Modify the SQLConn dictionary in the code to match your PostgreSQL connection details.
3. Install required Python packages using pip install -r requirements.txt.

⁠Running the Script

1. Initialization: Initialize the scraper class by instantiating it. Ensure the database connection string is properly set up.
2. Data Retrieval:
   • getAuditData(): Fetches flight data from the database. This data will be used to construct URLs for scraping.
   • update_flight_amount(row): Scrapes airline websites for flight details and updates the data in the table.
3. Execution:
   • Run the main() function to execute the scraping process.
4. Completion:
   • Once the process finishes, the flight table in the database will be updated with Kayak data.

⁠Functionality

⁠scraper Class

• Initialization: Connects to the PostgreSQL database using the provided connection details.
• Data Retrieval:
  • getAuditData(): Fetches flight data from the database.
• Web Scraping Methods:
  • get_html_from_url_bs(url): Retrieves HTML content using requests library.
  • get_html_from_url_selenium(url): Utilizes selenium to access dynamic content and retrieve HTML.
• Parsing Methods:
  • Airline-specific parsing methods for United, American Airlines (AA), and Kayak.
• URL Construction Methods:
  • Constructs URLs for United, AA, and Kayak based on provided flight data.

⁠Dependencies

• requests: For making HTTP requests.
• beautifulsoup4: For parsing HTML content.
• pandas: For handling data structures.
• selenium: For automating web browsers.
• psycopg2: For PostgreSQL database interaction.

⁠Imported Libraries

• requests: For making HTTP requests.
• BeautifulSoup (from bs4): For HTML parsing.
• quote (from urllib.parse): For URL encoding.
• pandas: For handling data structures.
• create_engine (from sqlalchemy): For database engine creation.
• datetime: For handling date and time.
• math: For mathematical operations.
• webdriver, Service, By, Options (from selenium): For browser automation.
• time: For handling time-related operations.

⁠Notes

• Ensure proper configuration of web drivers and required settings for Selenium to function correctly.
• Adjust timeouts in Selenium-based methods according to page load times.