kukker/lightnovel-crawler repository overview

⁠Lightnovel Crawler

Downloads lightnovels from various online sources and generates ebooks in many formats.

⁠Table of contents

• Installation⁠
  • ⏬ Standalone Bundle (Windows, Linux)⁠
  • 📦 PIP (Windows, Mac, and Linux)⁠
  • 📱 Termux (Android)⁠
  • 📱 Pydroid (Android)⁠
  • Chatbots⁠
    • Telegram⁠
    • Discord⁠
  • Run from source⁠
  • Heroku Deployment⁠
• General Usage⁠
  • Available options⁠
  • Running the bot⁠
• Development⁠
  • Adding new source⁠
  • Adding new Bot⁠
  • Supported sources⁠
  • Supported output formats⁠
  • Supported bots⁠
• The project structure⁠

⁠(A) Installation

This application uses Calibre to convert ebooks.
Install it from https://calibre-ebook.com/download⁠
Without it, you will only get output in epub, text, and web formats.

⁠A1. Standalone Bundle (Windows, Linux)

⏬ Windows: lightnovel-crawler v2.16.0 ~ 21MB⁠

⏬ Linux: lightnovel-crawler v2.16.0 ~ 27MB⁠

⁠A2. PIP (Windows, Mac, and Linux)

📦 A python package named lightnovel-crawler is available at pypi⁠.

To install this app or to update installed one via pip, just run:

$ pip install --user -U lightnovel-crawler

Remember, in some cases you have to use python3 -m pip or pip3 or python -m pip. And you do not need --user option, if you are running from root.

Next, open your terminal and enter:

$ lightnovel-crawler

# Or, a shortcut:
$ lncrawl

⁠A3. Termux (Android)

📱 Using Termux, you can run this app in your android phones too. Follow this instructions:

• Install Termux⁠ from playstore.
• Open the app and run these commands one by one:
  • apt update && apt upgrade
  • termux-setup-storage
  • pkg install ndk-sysroot make python zlib clang
  • pkg install libxml2 libxslt libiconv libcrypt libffi zlib libjpeg-turbo
  • pkg install nodejs-lts
  • pip install -U lightnovel-crawler to install the latest version of this app.
• Now exit the console and relaunch it.
• Type cd ~/storage/downloads to store novels there.
• Type lncrawl to start.
• You navigate up using Volume UP + W and down using Volume UP + S.
• Run pip install -U lightnovel-crawler again to install the latest updates.

⁠A4. Pydroid 3 (Android)

📱 Alternative to easy Termux is Pydroid3. It is a bit faster but harder to use and no run-in-backgroud support.

• Install Pydriod 3⁠ from playstore.
• Open the app and navigate to Pip from the drawer menu.
• Type lightnovel-crawler in place of Library name and press the Install button.
• To reinstall or remove the installed package:
  • Go to Libraries tab inside Pip page (left of Install tab).
  • Find lightnovel-crawler and press the Uninstall button.
  • Then go over to Install tab again to install the latest version.
• To use the app, select Terminal from the drawer menu. A console will appear.
• Type lncrawl to start.
• You navigate up using Volume UP + W and down using Volume UP + S.

⁠A5. Chatbots

⁠A5.1 Telegram

Visit this link to get started with the telegram bot: https://t.me/epub_smelter_bot⁠

⁠A5.2 Discord

Join our server: https://discord.gg/7A5Hktx⁠

Or, visit this link to install discord bot to your own server: https://discordapp.com/oauth2/authorize?client_id=537526751170002946&permissions=51264&scope=bot⁠

Send !help to open the bot help message.

⁠A6. Run from source

• First clone the repository:

$ git clone https://github.com/dipu-bd/lightnovel-crawler

• Open command prompt inside of the project folder and install requirements:

$ pip install --user -r requirements.txt

• Run the program (use python v3.5 or higher):

$ python __main__.py

# Or, in short,
$ python .

⁠A7. Heroku Deployment

Simply fill out the environment variables and you get a running instance.

⁠(B) General Usage

⁠B1. Available options

To view list of available options:

$ lncrawl -h
================================================================================
                           📒 Lightnovel Crawler 🍀2.16.0
                  https://github.com/dipu-bd/lightnovel-crawler
--------------------------------------------------------------------------------
usage: lncrawl [options...]
       lightnovel-crawler [options...]

optional arguments:
  -h, --help            show this help message and exit

  -v, --version         show program's version number and exit
  -l                    Set log levels. (-l = warn, -ll = info, -lll = debug).
  --list-sources        Display a list of available sources.
  -s URL, --source URL  Profile page url of the novel.
  -f, --force           Force replace any existing folder.
  -i, --ignore          Ignore any existing folder (do not replace).
  --login USER PASSWD.  User name/email address and password for login.
  --all                 Download all chapters.
  --first [COUNT]       Download first few chapters (default: 10).
  --last [COUNT]        Download last few chapters (default: 10).
  --page START STOP.    The start and final chapter urls.
  --range FROM TO.      The start and final chapter indexes.
  --volumes [N [N ...]]
                        The list of volume numbers to download.
  --chapters [URL [URL ...]]
                        A list of specific chapter urls.
  --bot {console,telegram,discord,test}
                        Select a bot. Default: console.
  --suppress            Suppress all input prompts and use defaults.
  ENV                   [chatbots only] Pass query string at the end of all
                        options. It will be use instead of .env file. Sample:
                        "BOT=discord&DISCORD_TOKEN=***&LOG_LEVEL=DEBUG"

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⁠B2. Running the bot

There are two chatbots available at this moment: Telegram and Discord. To run your own bot server, follow these instructions:

# Clone this repository
$ git clone https://github.com/dipu-bd/lightnovel-crawler
# Install requirements
$ pip3 install --user -r requirements.txt
$ pip3 install --user -r bot_requirements.txt
# Edit the environment variables
# You should give your API keys and log info here
# Also specify which bot server you want to run
$ cp .env.example .env
$ vim .env
# Run the server using:
$ python3 .

There is a server.sh script to run a bot in ubuntu servers. It will basically execute the python __main__.py and send the task to run in background. I use it to run my discord bot in the server.

⁠(C) Development

You are very welcome to contribute in this project. You can:

• create new issues pointing out the bugs.
• solve existing issues.
• add your own sources.
• add new output formats.
• create new bots.

⁠C1. Adding new source

• Create new crawler using the spiders/_sample.py⁠ as blueprint. You can check out existing bots for idea.
• Import your crawler to spiders/__init__.py⁠ file.

⁠C2. Adding new Bot

• Create a new bot file using bots/_sample.py⁠ as a standard. You can check out existing bots for idea.
• Import your bot to bots/__init__.py⁠ file.

⁠C3. Supported sources

The list of currently available sources and the future plans are given below:

Rejected:

⁠C4. Supported output formats

When download is done, the following files can be generated:

• JSON (default)
• TEXT
• WEB
• EPUB
• DOCX
• MOBI
• PDF
• RTF
• TXT
• AZW3
• FB2
• LIT
• LRF
• OEB
• PDB
• PML
• RB
• SNB
• TCR
• HTML

⁠C5. Supported bots

• Console Bot
• Telegram Bot
• Discord Bot

⁠(D) The project structure

Click here⁠ to view details.