amdih/plexussatellite

•Updated about 3 years ago

The Plexus Satellite manages isolated networked clusters on the AMD Accelerator Cloud.

Image

481

amdih/plexussatellite repository overview

⁠What is Plexus Satellite?

The Plexus Satellite container provides a rich set of tools for setting up and managing an isolated networked cluster on the AMD Plexus™ software stack. Clusters can have either Slurm, Edge SSH or Kubernetes resource managers. The Plexus Satellite container provides the following functions:

Verifies that the target cluster is compliant with the Plexus prerequisites.
Onboards the cluster to a specified Plexus instance.
Launches the broker satellite daemon to communicate between isolated clusters and Plexus. Designed for Data Scientists, by Data Scientists, the AMD Plexus™ software stack allows you to run your own private in-house AI cloud deployment and take advantage of your existing infrastructure and MLOps investment. With Plexus, your data scientists have a single pane of glass for working across on-premises and public clouds.

Plexus is a foundational component of AMD Accelerator Cloud (AAC), a data science platform built from the ground up for AI and HPC. Combining leading edge hardware with the rich feature set of the Plexus software stack, AMD Accelerator Cloud offers access to a co-located AI Platform-as-a-Service (AI PaaS). Customers that host their data lakes in co-location facilities can also take advantage of this on-premise installation, bringing the AI cloud experience to their data.

⁠Getting started

⁠Prerequisites

To access and run the Plexus Satellite container on Plexus, you will need the following:

An account on a Plexus platform (for example this one: https://aac.amd.com/signin⁠), or on a Plexus instance installed on your premises. NOTE: If you do not yet have an account on Plexus, please contact [email protected]⁠.
Either Provider or Admin permissions on your Plexus instance to permit you to add a new cluster. (Check with your Plexus admin)
A Docker hub account: https://hub.docker.com/⁠
Permissions to allow you to access this Plexus Satellite container: amdih/plexussatellite:2.7.
An up-to-date image installed -- Use docker pull to ensure an up-to-date image is installed. This example refers to the 2.7 version.

⁠Configure clusters

⁠Kubernetes clusters

First, the cluster must be configured with Plexus requirements.
The Kubeconfig file must be correctly formatted in yaml.
Create an account in the Plexus Control Panel. Only users with the Provider role can connect to Plexus Control panel when using Plexus Satellite (NOTE: If you have a regular User account on Plexus ask your Plexus Admin for Provider permissions).
You have more details about how to configure a Kubernetes cluster in this help article .

⁠Kubernetes Pod network isolation

Plexus network isolation works by default for the following IP ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 or 100.69.0.0/16. The network isolation will not work in your cluster if you have a different pod CIDR, so if you want to enable network isolation in your cluster, please contact Plexus support. You have more details about how to configure a Kubernetes cluster in this help article https://confluence.amd.com/display/PLX/Kubernetes+configuration+to+meet+AAC+requirements⁠

⁠Slurm clusters

There is a full guide to configure Slurm clusters in this help article https://confluence.amd.com/display/PLX/Slurm+configuration+to+meet+Plexus+requirements⁠

⁠Edge SSH clusters

An Edge SSH cluster can be composed by one or more nodes. Nodes have the following requirements:

Provide access through SSH connection.
Have any container manager either singularity or docker or both.
All the nodes have the same configuration, with the only difference of the hostname, latitude or longitude.
Remember that also user, password, port and home directory must be the same in all the nodes.

⁠Run the container

The container is executed by using the following pattern:

docker run -p 8080:80 --rm -it amdih/plexussatellite:2.7

You should run the container in interactive mode (-it). Otherwise the container goes into a loop.
-p 8080:80. Enable satellite GUI in port 8080. The container provides a GUI that can be accessed using your browser.

⁠Kubernetes optional parameters

docker run -p 8080:80 --rm -it -v /home/jorge/k8s-test.kubeconfig:/plexus/kubeconfig:ro amdih/plexussatellite:2.7

In case of Kubernetes clusters, mount the kubeconfig file in the container. The default used is
/plexus/kubeconfig. This file is required both to check and to onboard clusters.

⁠Slurm and SSH authentication parameters

docker run -p 8080:80 --rm -it -v /home/jorge/private_key.rsa:/plexus/private_key:ro amdih/plexussatellite:2.7

In case of both Slurm and Edge, authentication can be done by either password or private ssh key.
Mount the file of a private ssh key to connect to the cluster.

-v /home/jorge/private_key.rsa:/plexus/private_key:ro

⁠Edge SSH optional parameters

docker run -p 8080:80 --rm -it -v /home/jorge/nodes.txt:/plexus/nodes:ro amdih/plexussatellite:2.7

In case of Edge clusters, mount the nodes file in the container. The default used is
/plexus/nodes. This file is required to onboard edge multiple nodes clusters.
The format of the nodes file is one line per node configuration, each line has the format <hostname>;<latitude>;<longitude>.

⁠Local network settings

Required if you are using a Plexus platform which is in a local network or in a VPN.

Mount the host network with: --network=host . This is a required setting when you use any local network resource (cluster or Plexus server).
Mount the host dns configuration: -v /etc/resolv.gitconf:/etc/resolv.conf:ro. This is required when you are connecting to the cluster using a VPN connection.

NOTE: If you are using macOS, the host can be reached by using:

    host.docker.internal

Access the container menu. When you run the container it will request the name of the cluster resource manager, it can be either SSH, Slurm or Kubernetes.

    -- ---------------------------------------------- --
    -- -- Welcome to the cluster satellite script --- --
    -- ---------------------------------------------- --
    -- Select resource manager [SSH, Slurm or Kubernetes]:  --
    Enter resource manager [Kubernetes]:

When you select Kubernetes resource manager, the following menu will be displayed:

    -- Please enter your choice:  --
    1) Pre-flight cluster compatibility test
    2) Onboard your cluster in Plexus control panel
    3) Start the Plexus Satellite
    4) Quit    
    #?

Enter 1 to execute the cluster verification process.
Enter 2 to execute the cluster onboarding script.
Enter 3 to execute the broker satellite daemon.
Enter 4 to exit from the container.

⁠Verify that the Kubernetes cluster is compliant with the Plexus requisites

Menu Option 1 will check that the Kubernetes cluster configuration is compliant with the following Plexus requirements:

User role permissions
Storage class availability
Resources can be properly setup in the cluster

Once you enter 1 in the menu, it will ask you for:

The kubeconfig path in the container
The storage class that you will use to store persistent data in the cluster
The Namespace that you will use in the cluster (by default the Namespace specified in the current version of the kubeconfig file will be used in this check script)
The pod cidr is used to check if it is in de range of CIDR covered by the plexus network policies.

      #? 1  
     -- Option Pre-flight cluster compatibility test --   
     Enter kubeconfig path [/plexus/kubeconfig]:   
     Enter storage class: storage-class   
     Enter namespace [default]:
     Enter pod cidr[]: 10.0.0.0/8

⁠Onboard the Kubernetes cluster in the Plexus server

Menu Option 2 will set up the new cluster in the Plexus server.
Once you enter 2 in the menu, the container script will start to onboard your cluster by using the kubeconfig file you have provided, together with the following parameters that the system will ask you for:

The kubeconfig path in the container
The storage class name that you will use to store persistent data in the cluster
Storage size for users
The name of the cluster to be set up in Plexus.
Plexus server url
Your email of the Plexus server account
Your account password

    #? 2  
    -- Option: Onboard your cluster in Plexus control panel --
    Enter kubeconfig path [/plexus/kubeconfig]:  
    Enter storage class: storage-class  
    Enter default storage size for users (Gigabytes) [1]:    
    Enter cluster name: satellite-cluster  
    Is it a satellite cluster? [true]:  
    Enter Plexus server [https://aac-api.amd.com]:  
    Insert your Plexus email: [email protected]  
    Insert your Plexus password:  

    -- Onboarding cluster --  
    -- Creating user token in https://aac-api.amd.com --  
    Token generated with value: dc2dd0xxxxxxxxxxxxxx7a12fd5ba71241c777  
    -- Onboarding cluster --  
    -- Creating cluster "satellite-cluster" in https://aac-api.amd.com --  
    Cluster "satellite-cluster" successfully created in: https://aac-api.amd.com/clusters/777  
    Cluster "satellite-cluster" has uuid: 81bffe12-f7c3-4d3b-b4c7-84f4348ddacf

Once the process is completed the script will return the api server url from the new cluster and the cluster uuid. The cluster uuid will be required for launching the broker satellite; keep it for future broker executions:

API server URL: https://aac-api.amd.com/clusters/777⁠
Cluster "satellite-cluster" has uuid: 81bffe12-f7c3-4d3b-b4c7-84f4348ddacf

And you can find the new cluster in the user interface:

User Interface URL: https://aac.amd.com/clusters/777⁠

When you select Slurm resource manager, the following menu will be displayed:

    -- Please enter your choice:  --
    1) Pre-flight cluster compatibility test
    2) Onboard your cluster in Plexus control panel
    3) Start the Plexus Satellite
    4) Quit    
    #?

Enter 1 to execute the cluster verification process.
Enter 2 to execute the cluster onboarding script.
Enter 3 to execute the broker satellite daemon.
Enter 4 to exit from the container.

⁠Verify that the SLURM cluster is compliant with the Plexus requisites

Menu Option 1 will check that the SLURM cluster configuration is compliant with the following Plexus requirements:

Cluster has queues available
Queues have a proper configuration
Singularity is properly installed in the cluster. It must be installed in every node (including the head node)
Head node can download singularity images
Workload can be executed in the cluster on a specify partition
Workload ouput files are shared with every node

Once you enter 1 in the menu, it will ask you for:

The cluster host name
The cluster ssh port
The cluster username
The partition name for testing
The cluster home path. It is a folder shared among all the nodes.
Select authentication type, either password or private_key.
The cluster password

    #? 1  
     -- Option: Pre-flight Cluster compatibility test --
    Enter cluster host []: serve.com
    Enter cluster port []: 22
    Enter cluster username []: ubuntu
    Enter partition for testing [debug]: MI300
    Enter cluster shared home path []: /home/ubuntu/nfs    
    Select authentication type [password or private_key]:
    Enter authentication type [password]: private_key
    Enter cluster private key path [/plexus/private_key]: 
    Following password will be used to attempt to unlock the key.
    Enter cluster password:

⁠Onboard the Slurm cluster in the Plexus server

Menu Option 2 will set up the new Slurm cluster in the Plexus server.
Once you enter 2 in the menu, the container script will start to onboard your cluster by using the following parameters that the system will ask you for:

Note: SLURM cluster onboarding process is limited to password authentication. In case of using SSH-key authentication, the cluster must be onboarded from the UI.

First of all the credentials to access the cluster by ssh will be requested

The cluster host name
The cluster ssh port
The cluster shared home path
The cluster username
The cluster password

Later it requests the cluster name and credentials for Plexus.

The name of the cluster to be set up in Plexus.
Plexus server url
Your email of the Plexus server account
Your account password

    #? 2  
    -- Option: Onboard your cluster in Plexus control panel --
    Enter cluster host []: test.cluster.com
    Enter cluster port []: 22
    Enter cluster shared home path []: /home/ubuntu/nfs
    Enter cluster username []: user  
    Enter cluster password: 
    Enter cluster name []: satellite-slurm-cluster
    Is it a satellite cluster? [true]: 
    Enter Plexus server [https://aac-api.amd.com]:
    Insert your Plexus email: [email protected]  
    Insert your Plexus password:  

    -- Onboarding cluster --  
    -- Creating user token in https://aac-api.amd.com --  
    Token generated with value: dc2dd0xxxxxxxxxxxxxx7a12fd5ba71241c777  
    -- Onboarding cluster --  
    -- Creating cluster "satellite-slurm-cluster" in https://aac-api.amd.com --  
    Cluster "satellite-slurm-cluster" successfully created in: https://aac-api.amd.com/clusters/777  
    Cluster "satellite-slurm-cluster" has uuid: 81bffe12-f7c3-4d3b-b4c7-84f4348ddacf

API server URL: https://aac-api.amd.com/clusters/777⁠
Cluster "satellite-slurm-cluster" has uuid: 81bffe12-f7c3-4d3b-b4c7-84f4348ddacf

And you can find the new cluster in the user interface:

User Interface URL: https://aac.amd.com/clusters/777⁠

When you select SSH resource manager, the following menu will be displayed:

    -- Please enter your choice:  --
    1) Pre-flight cluster compatibility test
    2) Onboard your cluster in Plexus control panel
    3) Start the Plexus Satellite
    4) Quit    
    #?

Enter 1 to execute the cluster verification process.
Enter 2 to execute the cluster onboarding script.
Enter 3 to execute the broker satellite daemon.
Enter 4 to exit from the container.

⁠Verify that the Edge SSH cluster is compliant with the Plexus requisites

Menu Option 1 will check that the Edge SSH cluster configuration is compliant with the following Plexus requirements in one of the nodes. Remember every node must have the same configuration.

Node has enough resources
Singularity is properly installed in the cluster.
Docker is properly installed in the cluster.
Node can download singularity images
Singularity workload can be executed in the cluster
Docker workload can be executed in the cluster

Once you enter 1 in the menu, it will ask you for:

The cluster host name
The cluster ssh port
The cluster username
The cluster home path. It is a folder shared among all the nodes.
Select authentication type, either password or private_key.
The cluster password

     -- Option: Pre-flight Cluster compatibility test --
    Enter cluster host []: serve.com
    Enter cluster port []: 22
    Enter cluster username []: ubuntu
    Enter cluster shared home path []: /home/ubuntu/nfs    
    Select authentication type [password or private_key]:
    Enter authentication type [password]: private_key
    Enter cluster private key path [/plexus/private_key]: 
    Following password will be used to attempt to unlock the key.
    Enter cluster password:

⁠Onboard the Edge SSH cluster in the Plexus server

Menu Option 2 will set up the new Edge SSH cluster in the Plexus server.
Once you enter 2 in the menu, the container script will start to onboard your cluster by using the following parameters that the system will ask you for:

Note: Edge SSH cluster onboarding process is limited to password authentication. In case of using SSH-key authentication, the cluster must be onboarded from the UI.

First, the credentials to access the cluster by ssh will be requested

The nodes file path
The cluster ssh port
The cluster shared home path
The cluster username
The cluster password

Later it requests the cluster name and credentials for Plexus.

The name of the cluster to be set up in Plexus.
Plexus server url
Your email of the Plexus server account
Your account password

    #? 2  
    -- Option: Onboard your cluster in Plexus control panel --
    Enter nodes config path [/plexus/nodes]:
    Enter cluster port []: 22
    Enter cluster shared home path []: /home/ubuntu
    Enter cluster username []: user  
    Enter cluster password: 
    Enter cluster name []: satellite-edge-cluster
    Is it a satellite cluster? [true]: 
    Enter Plexus server [https://aac-api.amd.com]:
    Insert your Plexus email: [email protected] 
    Insert your Plexus password:  

    -- Onboarding cluster --  
    -- Creating user token in https://aac-api.amd.com --  
    Token generated with value: dc2dd0xxxxxxxxxxxxxx7a12fd5ba71241c777  
    -- Onboarding cluster --  
    -- Creating cluster "satellite-edge-cluster" in https://aac-api.amd.com --  
    Cluster "satellite-edge-cluster" successfully created in: https://aac-api.amd.com/clusters/777  
    Cluster "satellite-edge-cluster" has uuid: 81bffe12-f7c3-4d3b-b4c7-84f4348ddacf

API server URL: https://aac-api.amd.com/clusters/777⁠
Cluster "satellite-edge-cluster" has uuid: 81bffe12-f7c3-4d3b-b4c7-84f4348ddacf

And you can find the new cluster in the user interface:

User Interface URL: https://aac.amd.com/clusters/777⁠

⁠Launch broker satellite daemon on the cluster

Both clusters share the same Start the Plexus Satellite option. It will execute the broker daemon, which will be used for communicating between isolated clusters and the Plexus platform.
Once you have entered this option in the menu, it will launch the broker by using the server, user email and password, in addition to the cluster uuid.

We recommend that you set up 15 seconds in the pull interval - if it is set too low, it could provoke the Plexus API to reject requests from your satellite.

    #? 3    
    -- Option Start the Plexus Satellite --    
    Enter plexus server [https://aac-api.amd.com]:   
    Enter your Plexus email: [email protected] 
    Enter your Plexus password:   
    Enter cluster uuid: 81bffe12-f7c3-4d3b-b4c7-84f4348ddacf  
    Enter pull interval [15]:   
    -- Launching broker --  
    2020-10-09 14:44:02.780978. Getting auth token
    Satellite GUI available
    2020-10-09 14:44:02.817024. Pulling requests from API

⁠Trouble shooting

⁠Kubernetes clusters

⁠Workloads keep pending for ever

The node does not have the proper labels. Consider that CPU workloads are executed in cpu-only nodes and GPU workloads are executed in gpu nodes.
Solution: You need to label the cpu-only nodes with either node-role.kubernetes.io/plexus-worker-type=plexus-cpu-worker or the gpu nodes with node-role.kubernetes.io/plexus-worker-type=plexus-gpu-worker. or the hybrid cpu/gpu nodes with plexus-hybrid-cpu-gpu-worker
There are not enough resources in the nodes for the workload requirements. Solution: Decrease the number of gpus or cpus required by your workload.
The number of cpus assigned to the node in the cluster queue configuration is higher than that provided by the physical cluster, or they are used by the resource manager. Solution: Decrease the number of maximum cpus per workload in the queue configuration.

⁠Namespaces are not configured in the new cluster and it appears as disabled

There are several possible reasons for this problem:

The kubeconfig is not properly configured.
Solution: Fix the kubeconfig file. We recommend to launch the check script by using Option 1, before creating or launching the broker.
Cluster does not have the right configuration.
Solution: Your cluster needs to match the proper Cluster configuration. Read the Plexus help articles. We recommend that you launch the check script by using Option 1.
The time between cluster creation and launching the broker is higher than 60 seconds. Solution: You can discover the Queues after cluster creation by clicking on the Update Details in the cluster view.

⁠License

An End User License Agreement is included with this product. By pulling and using this container, you accept the terms and conditions of this license.

⁠Technical Support

Help articles: https://aac.amd.com/help⁠

Tag summary

Recent tags

Content type

Image

Digest

sha256:e93606c15…

Size

167 MB

Last updated

about 3 years ago

docker pull amdih/plexussatellite:2.7.1