nionmaron/playwright-api repository overview

⁠Playwright API

⁠📄 Description

The Playwright API is a RESTful API developed with FastAPI and Playwright, designed for web content scraping. It allows you to submit a list of URLs and receive back the text extracted from the <h1>, <h2>, <h3>, and <p> tags of each page, formatted in Markdown. This project is containerized with Docker, simplifying deployment and usage.

⁠🚀 Features

• Selective Extraction: Collects only the titles (<h1>, <h2>, <h3>) and paragraphs (<p>) from web pages.
• Markdown Formatting: Converts the extracted content into Markdown, preserving the hierarchy of titles and the formatting of paragraphs.
• RESTful API: Simple and intuitive interface for integration with other applications.
• Docker Containerization: Facilitates deployment and scalability of the application.

⁠🛠️ Technologies Used

• FastAPI: Python framework for building fast and efficient APIs.
• Playwright: Browser automation library, ideal for modern scraping.
• Docker: Platform for developing, shipping, and running applications in containers.
• BeautifulSoup4: Python library for parsing HTML and XML.
• Markdownify: Library for converting HTML to Markdown.

⁠📦 Docker Image

⁠Image Name

nionmaron/playwright-api:v0.0.1

⁠How to Run

To start the API using Docker, execute the following command:

docker run -d --name playwright-api -p 8000:8000 nionmaron/playwright-api:v0.0.1

• Parameters:
  • -d: Runs the container in detached mode (background).
  • --name playwright-api: Names the container for easy reference.
  • -p 8000:8000: Maps port 8000 of the host to port 8000 of the container.

⁠Accessing the API

After starting the container, the API will be accessible at:

http://localhost:8000

⁠Interactive Documentation

Access the interactive documentation generated by FastAPI at:

http://localhost:8000/docs

⁠📚 How to Use

⁠Endpoint /scrape

⁠Method: POST

⁠Description:

Receives a list of URLs and returns the content extracted from the <h1>, <h2>, <h3>, and <p> tags in Markdown format.

⁠Payload (JSON):

{
    "urls": [
        "https://www.example.com",
        "https://www.anotherexample.com"
    ]
}

⁠Response (JSON):

{
    "https://www.example.com": "# Page Title\n\nParagraph content...\n## Subtitle\n\nMore content...\n### Smaller Subtitle\n\nEven more content...\n",
    "https://www.anotherexample.com": "# Another Title\n\nParagraph content...\n"
}

⁠cURL Request Example:

curl -X POST "http://localhost:8000/scrape" \
     -H "Content-Type: application/json" \
     -d '{"urls": ["https://www.infomoney.com.br/"]}'

⁠Response Example:

{
    "https://www.infomoney.com.br/": "# Page Title\n\nParagraph content...\n## Subtitle\n\nMore content...\n### Smaller Subtitle\n\nEven more content...\n"
}

⁠Example Usage with R

If you use R to interact with the API, here's an example script:

# Load necessary packages
library(httr)
library(jsonlite)

# Define the API URL and the URL you want to scrape
api_url <- "http://localhost:8000/scrape"
target_urls <- c("https://www.infomoney.com.br/")  # Add more URLs as needed

# Build the JSON payload
payload <- list(urls = as.list(target_urls))
payload_json <- toJSON(payload, auto_unbox = TRUE)

# Send the POST request to the API
response <- POST(
  url = api_url,
  body = payload_json,
  encode = "json",
  content_type_json()
)

# Check if the request was successful
if (status_code(response) == 200) {
  # Parse the response content
  response_content <- content(response, as = "text", encoding = "UTF-8")
  response_json <- fromJSON(response_content)
  
  for (url in target_urls) {
    markdown_text <- response_json[[url]]
    
    if (grepl("^Error", markdown_text)) {
      cat(sprintf("Error processing %s: %s\n", url, markdown_text))
    } else {
      # Display the Markdown text in the console
      cat(sprintf("Markdown Text for %s:\n\n", url))
      cat(markdown_text)
      
      # Optional: Save the text to a Markdown file
      safe_filename <- gsub("https?://", "", url)
      safe_filename <- gsub("[^a-zA-Z0-9_-]", "_", safe_filename)
      filename <- paste0(safe_filename, ".md")
      writeLines(markdown_text, con = filename)
      cat(sprintf("\n\nContent saved to '%s'.\n\n", filename))
    }
  }
  
} else {
  # If the request failed, display an error message
  cat("Error accessing the API. Status:", status_code(response), "\n")
  cat("Error message:", content(response, as = "text", encoding = "UTF-8"), "\n")
}

⁠🧰 Requirements

• Docker: Ensure Docker is installed on your machine. Install Docker⁠
• Docker Hub Account: To pull the image from Docker Hub, you may need to be logged in. Create a Docker Hub Account⁠

⁠🔧 Configuration and Customization

If you wish to customize the API or adjust parameters, you can clone the repository and build the Docker image locally.

⁠1. Clone the Repository

git clone https://github.com/nionmaron/playwright-api.git
cd playwright-api

⁠2. Build the Docker Image

docker build -t playwright-api:v0.0.1 .

⁠3. Run the Docker Container

docker run -d --name playwright-api -p 8000:8000 playwright-api:v0.0.1

⁠📈 Monitoring and Logs

To view the API logs in real-time, use the following command:

docker logs -f playwright-api

This allows you to monitor requests, errors, and other relevant information directly in the terminal.

⁠📝 Final Considerations

• Performance: Utilizing Playwright provides efficient and modern scraping capabilities but can be resource-intensive depending on the volume of requests.
• Security: Always validate and sanitize incoming URLs to prevent scraping malicious sites.
• Legality: Ensure compliance with the Terms of Service of the websites you are scraping and adhere to ethical scraping practices.

⁠📄 License

This project is licensed under the MIT License⁠.

⁠📫 Contact

Developed by Nion Maron⁠.

For any questions or suggestions, feel free to reach out.