microsoft/datax-dataxlocal repository overview

⁠Featured Tags

• datax/dataxlocal:v1 (Current)
  • docker pull mcr.microsoft.com/datax/dataxlocal:v1

⁠About This Image

Data Accelerator⁠ for Apache Spark democratizes streaming big data using Spark by offering several key features such as a no-code experience to set up a data pipeline as well as fast dev-test loop for creating complex logic. Our team has been using the project for two years within Microsoft for processing streamed data across many internal deployments handling data volumes at Microsoft scale. It offers an easy to use platform to learn and evaluate streaming needs and requirements. This image contains the local experience for Data Accelerator⁠.

⁠How to Use the Image

To run Data Accelerator locally, deploy locally⁠ and then check out the local mode tutorials⁠.

⁠Featured Repos

• Data Accelerator Repo on Github⁠

⁠Deployment

• Run the below commands in Powershell on Windows (and approve subsequent elevation request) or in Terminal on Mac

  docker run --rm --name dataxlocal -d -p 127.0.0.1:49080:2020 -p 127.0.0.1:4040:4040 mcr.microsoft.com/datax/dataxlocal:v1
  

  Copy

• If you want to get the latest docker image, delete the one you have downloaded previously and then run the above command. To delete already downloaded image, follow these steps:

  • Run these commands (in case you haven't already done so):

    docker stop dataxlocal            
    docker images -a
    

    Copy
  • This will list all the images on your box. Note the ImageId for all images listed where the repository equals msint.azurecr.io/datax/dataxlocal and then run the following command for each of the ImageId to remove them from the machine:

    docker image rm <ImageId>  
    

    Copy

⁠Deployment With Security using Basic Auth

• Run the below commands in Powershell on Windows (and approve subsequent elevation request) or in Terminal on Mac

• Refer to Nginx readme⁠

  docker run --rm --name dataxlocal -d -p 127.0.0.1:4040:4040 mcr.microsoft.com/datax/dataxlocal:v1
  

  Copy

  Try accessing and logging in with username foo and password bar.

  docker run --rm -d -e HTPASSWD='foo:$apr1$odHl5EJN$KbxMfo86Qdve2FH4owePn.' -e FORWARD_PORT=2020 --link dataxlocal:web -p 127.0.0.1:49080:80 --name auth beevelop/nginx-basic-auth
  

  Copy

  If you wish to use a different username and password, visit here⁠ to generate the HTPASSWD value

• If you want to get the latest docker image, delete the one you have downloaded previously and then run the above command. To delete already downloaded image, follow these steps:

  • Run these commands (in case you haven't already done so):

    docker stop dataxlocal
    docker stop auth
    docker images -a
    

    Copy
  • This will list all the images on your box. Note the ImageId for all images listed where the repository equals msint.azurecr.io/datax/dataxlocal and then run the following command for each of the ImageId to remove them from the machine. Also remove the images listed for repository beevelop/nginx-basic-auth and remove them as well:

    docker image rm <ImageId>  
    

    Copy

• Open the portal at: http://localhost:49080/home⁠ to start Data Accelerator and create your first Flow and / or checkout the samples

• Check out step by step tutorials⁠ for local mode

⁠Feedback

• Request new features on GitHub⁠
• Open a new issue on GitHub⁠
• Ask a question on Stack Overflow⁠
• Check out the contributing page⁠ to see the best places to log issues and start discussions.

Please also see our Code of Conduct⁠.

⁠FAQ and troubleshooting:

• Please refer to the FAQ⁠.

⁠License

• Legal Notice: Container License Information⁠

Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT License