finaldie/auto-news
Auto-news official repo for Kubernetes
3.5K
finaldie/auto-news repository overview
Auto-News: Automatic News Aggregator with LLM
Auto-news loves Kubernetes (k8s), to deploy auto-news backend on Kubernetes, we can do it via either Helm or ArgoCD. This page, we are focusing on Helm path.
Before Start
We assume that we already:
- Have the K8s environment and the client machine that can connect to and operate it (also
kubectlinstalled) - Have Created a Notion page and enabled
page connectionfor auto-news backend (We need to notion page id and permission for the rest of the setup)
Install Helm
Follow the Helm installation guide, and pick one of the installation methods for the environment.
After installation, from top directory of the auto-news project, run below command to update helm repos:
make helm-repo-update
Installation Steps
1. Env file creation
make prepare-env
Now, we have the default env file at build/.env.k8s, fill the necessary environment vars the same as Installation via docker-compose
Then, we create the env file only for k8s, here we won't save the confidential tokens/passwords into Docker image, instead, we will store them as Kubernetes Secret
make k8s-env-create
2. Namespace Creation
Run the below command to create a namespace auto-news
make k8s-namespace-create
Verification:
❯ kubectl get namespace | grep auto-news
auto-news Active 28h
3. Secret Creation
make k8s-secret-create
Verification:
❯ kubectl get secret -n auto-news | grep airflow-secrets
airflow-secrets Opaque 13 19h
4. Build Docker image (Optional)
Skip this if we don't want to build a customized image, the default image can be found on Dockerhub.
If we still want to build a customized image and use it for the deployment, follow the below steps:
# Build docker image, replace <repo_and_imagename> and <x.y.z> accordingly
make k8s-docker-build repo=<repo_and_imagename> tag=<x.y.z>
# Push docker image to the registry (e.g. dockerhub)
docker login
# Replace <repo_and_imagename> and <x.y.z> accordingly
make k8s-docker-push repo=<repo_and_imagename> tag=<x.y.z>
After uploading the new image to the Registry, modify repository and tag in helm/values.yaml:
airflow:
images:
airflow:
repository: finaldie/auto-news
tag: 0.9.2
5. Helm Install Auto-News Chart
make k8s-helm-install
Verification:
❯ k get pods -n auto-news
NAME READY STATUS RESTARTS AGE
auto-news-statsd-6f64d89f4f-qpq5h 1/1 Running 0 19h
auto-news-milvus-attu-5b8c94484d-gjgwt 1/1 Running 0 19h
auto-news-minio-3 1/1 Running 0 19h
auto-news-minio-1 1/1 Running 0 19h
auto-news-minio-0 1/1 Running 0 19h
auto-news-minio-2 1/1 Running 0 19h
auto-news-redis-0 1/1 Running 0 19h
auto-news-postgresql-0 1/1 Running 0 19h
auto-news-milvus-standalone-8475ccfd5-5v4jd 1/1 Running 0 19h
auto-news-mysql-0 2/2 Running 0 19h
auto-news-redis-master-0 1/1 Running 0 4h43m
auto-news-redis-replicas-2 1/1 Running 0 4h43m
auto-news-etcd-2 1/1 Running 0 4h43m
auto-news-redis-replicas-1 1/1 Running 0 4h42m
auto-news-redis-replicas-0 1/1 Running 0 4h41m
auto-news-etcd-1 1/1 Running 0 4h41m
auto-news-etcd-0 1/1 Running 0 4h40m
auto-news-worker-0 2/2 Running 0 4h39m
auto-news-scheduler-5f8d65589b-p642q 2/2 Running 0 4h43m
auto-news-webserver-556676647-4mnlk 1/1 Running 0 4h43m
auto-news-scheduler-5f8d65589b-25mqs 2/2 Running 0 4h35m
auto-news-triggerer-0 2/2 Running 0 3h25m
auto-news-triggerer-1 2/2 Running 0 3h25m
auto-news-adminer-5c754b9554-9cg89 1/1 Running 0 144m
airflow-init-user-b6k9f 0/1 Completed 0 111m
6. Enable Airflow DAGs
make k8s-airflow-dags-enable
Verification all DAGs are enabled (except upgrade one):
- Enable port-forwarding:
kubectl port-forward service/auto-news-webserver 8080:8080 --namespace auto-news - Then open URL http://localhost:8080 from Browser
7. Verify Notion Inbox Page
If everything goes right, we will see the necessary sub-pages created, continue to set up the RSS list/Reddit list, and enjoy the peace of reading experience in the journey.
Any issues, feel free to drop one on Open Issue
Operations
Kubernetes Port-Forwarding
| Service | Command | Browser URL |
|---|---|---|
| Airflow | kubectl port-forward service/auto-news-webserver 8080:8080 --namespace auto-news --address=0.0.0.0 | http://localhost:8080 |
| Milvus | kubectl port-forward service/auto-news-milvus-attu -n auto-news 9100:3001 --address=0.0.0.0 | http://localhost:9100 |
| Adminer | kubectl port-forward service/auto-news-adminer -n auto-news 8070:8080 --address=0.0.0.0 | http://localhost:8070 |
Upgrading
To upgrade to the latest version from the auto-news release, run:
# At the root directory of auto-news
> git pull
> make k8s-helm-install
Tag summary
Content type
Image
Digest
sha256:28cdf4918…
Size
3.8 GB
Last updated
almost 2 years ago
docker pull finaldie/auto-news:0.9.15