[{"_1":2,"_161":-5,"_162":-5},"loaderData",{"_3":4,"_17":18,"_25":26,"_140":141},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","routes/_layout",{"_19":-5,"_20":21,"_22":21,"_23":11,"_24":21},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_88":-5,"_89":90,"_129":-5,"_130":11,"_131":21,"_132":11,"_133":-5,"_134":11,"_135":-5,"_136":137,"_138":91,"_139":-7},"canonicalUrl","https://hub.docker.com/r/fullstackml/oneboxml-spark-sql/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","fullstackml","reponame","oneboxml-spark-sql","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":55,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":74,"_79":21,"_80":81,"_85":33,"_86":87},"categories",[],"content_types",[44],"image","date_registered","2016-08-14T00:05:43.516776Z","description","Apache Spark SQL command.","is_private","last_updated","2017-09-08T00:22:12.20007Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[56],"application/vnd.docker.container.image.v1+json","name","pull_count",1139,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","# OneBoxML introduction\nOneBoxML is all in one data scientists tool box. OneBoxML extends your data science strengths by the power of cloud computing. Simply speaking, with OneBoxML you can run your data science scripts (Python, R, Apache Spark) in your local machine and EC2 instances of any types (how about an instance with 2TB of RAM or an instance with 16GB NVIDIA GRID GPUs?) and simply synchronize data from your machine to the cloud instances and back.\n\n# How it works?\n\n## Dockerizing data science tools\n\nOneBoxML uses [Docker](http://docker.com) under the hood and provides set of dockerized data scientists tools:\n\n1. [OneBoxML Python](https://hub.docker.com/r/fullstackml/oneboxml-python/) - dockerized Python with pre-installed machine learning libraries such as NumPy, SciPy, TensorFlow.\n3. [OneBoxML Rscript] (https://hub.docker.com/r/fullstackml/oneboxml-rscript/) - R with pre-installed libraries. \n4. [OneBoxML Spark Shell](https://hub.docker.com/r/fullstackml/oneboxml-spark-shell/) - dockerized standalone [Apache Spark](spark.apache.org). It is usefull for out-of-memory data wrangling and out-of-memory machine learning.\n\nYou don't need any docker skills or knowladge. OneBoxML Shell commands hide Docker from users. Examples:\n```bash\noneboxml-python test/pytest.py # Run your script by Python container in local machine.\noneboxml-run-instance --instance-type g2.8xlarge # Launch a GPU instance in EC2.\noneboxml-cloud-python test/pytest.py # Run python script in EC2 instance.\noneboxml-terminate-instances all # Stop the instance.\n```\n\nThe only requirement - you should have [docker installed](https://docs.docker.com/engine/installation/) in your machine and set up maximum amount of memory that docker can use. We recomend to use 80% of your RAM memory.\n\n## OneBoxML directory\n\nAn important concept in OneBoxML is **OneBoxML directory** which is a directory in your local machine. This directory is the only visiable directory from docker containers in your local machine as well as from remote EC2 instances. All your code and data has to be in this directory.\n\nUse ONEBOXML_DIR variable to set up the directory: \n```bash\nexport ONEBOXML_DIR=~/src/oneboxml\n```\n\n# Tutorial\n\n## Setup\n\nYou should have Bash command line and [installed Docker](https://docs.docker.com/engine/installation/).\n\nLet's clone the project source code in your ~/src directory:\n```bash\ncd ~/src\ngit clone https://github.com/dmpetrov/oneboxml.git\n```\n\nExport OneBoxML commands:\n```bash\ncd oneboxml\nsource oneboxml.rc\n```\n\nExport OneBoxML directory. All your code and data has to be in the box directory. This directory will be used to sync you data to cloud machines and back - from cloud servers:\n```bash\nexport ONEBOXML_DIR=~/src/oneboxml\n```\n\n## Run in local machine\n\nLet's run a python script with numpy and scipy in your local machine. You don't need numpy or scipy libraries. The dockerized OneBoxML Python will be launched with pre-installed libraries. \n```bash\noneboxml-python test/pytest.R test/pyinput.csv test/pyoutput.csv\n```\nThe script reads test/pyinput.csv file and creates output file **test/pyoutput.csv**.\n\n## Set up AWS credentials\n\nThe power of OneBoxML is running script in the EC2 cloud. To do that you have to set up AWS credentials. AWS access key and security access kay have to be set up in OneBoxML config file: ~/src/oneboxml/oneboxml.conf\n\nAccessKeyID = AKIAJ123456789012346\nSecretAccessKey = ASDFGHJKLasdfghjklASDFGHJKLasdfghjklASDF\n\nYou can find an instruction how to get the AWS keys here: http://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html\n\n## Run code in EC2 instance\n\nFirst, let's launch a regular EC2 instance with 7.5Gb of memory for our experiments. Type m3.large. See [EC2 instances types](https://aws.amazon.com/ec2/pricing/on-demand/).\n\n```bash\noneboxml-run-instance --instance-type m3.large\n```\nOutput:\n```\nNew m3.large instance i-53c8d962 was selected as active\nWaiting for a running status.\n...............\n```\n\nNow one instance is running and active (See the first column in the next command output). \n```bash\noneboxml-describe-instances\n```\nOutput:\n```\nActive Id Type State Storage IP public IP private \n *** i-53c8d962 m3.large running 54.166.78.104 172.31.59.211\n```\nMany instances might be created but only one of them can be active. Multi-instance environment will be described later. Now we are working with one single instance. Show a list of runing instances:\n\nSynchronize your local OneBoxML directory to the active instance:\n```bash\noneboxml-sync-to-cloud\n```\n\nRun the script in cloud:\n```\noneboxml-cloud-python test/pytest.py\n```\n\nSync data from your remote directory to OneBoxML directory:\n```bash\noneboxml-sync-to-local\n```\n\nNow the output file is synced from the EC2 instance to the local directory **test/pyoutput.csv**.\n\nThe EC2 instance could be stoped:\n```bash\noneboxml-terminate-instances all\n```\n\n\n# Experiments\n\n## Git semantic\n\n\nOPEN QUESTION: \n1. can we avoid syncing data directly? Let's use only git and S3!\n\nShow list of environments which is empty by default.\n

$ oneboxml env\n

\n\n

$ oneboxml sync-status\nNo remote server set up\nNothing to sync\n

\n\nRun new instance:\n

$ oneboxml run-instance --instance-type g2.8xlarge --name gpu-server\n$ oneboxml env  # Show the environment\n* gpu-server\n

\n\nRun a script remoutly in the new instance:\n

$ oneboxml sync-to gpu-server   # Push all data from local host to the instance\n$ oneboxml run --remote python test/pytest.py test/pyinput.csv output/pyoutput.csv\n$ oneboxml sync-status\nUnsynced remoute files:\n    output/pyoutput.csv\n$ oneboxml sync-from gpu-server # Change env to the local\n$ cat output/pyoutput.csv         # Output the result.\n$ oneboxml sync-status          # Everything is synced\n

\n\nSimple version of the same scenario:\n

$ oneboxml run --remote --full-sync python test/pytest.py test/pyinput.csv output/pyoutput.csv\n$ cat output/pyoutput.csv\n

\n\n\nVerbose version of the simple script:\n

$ oneboxml run -v --remote --full-sync python test/pytest.py test/pyinput.csv output/pyoutput.csv\nSync to server:\n    test/pytest.py\n    test/rtest.R\n    test/pyinput.csv\n    test/Rinput.csv\nRun command [gpu-server]: python test/pytest.py test/pyinput.csv output/pyoutput.csv\nSync from server:\n    output/pyoutput.csv\n$ cat test/pyoutput.csv\n

\n\n## Persist reproducible results\n\n

$ git checkout --track -b origin/py_experiment # create tracking branch (local and upstream)\nSwitched to a new branch 'py_experiment'\nBranch origin/py_experiment set up to track local branch master.\nSwitched to a new branch 'origin/py_experiment'\n$ vim test/pytest.py # Edit the source code\n$ git commit -am 'New script version'\n$ git push # push change to the \n$ oneboxml run --remote --repro --output output/pyoutput.csv --input test/pyinput.csv python test/pytest.py test/pyinput.csv output/pyoutput.csv\n$ oneboxml sync-status\nUnsynced remoute files:\n    output/pyoutput.csv\n    repro/622278b4ea910275ec572268c6492449c41fd5e4-output____pyoutput.csv\n$ oneboxml sync-from gpu-server\n$ cat output/pyoutput.csv    # Output the result.\n$ oneboxml sync-status       # Everything is synced\n

\n\nThe running command could be simplified.\nHowever, a script developer has to maintain this command semantics.\n

$ oneboxml run --remote --repro python test/pytest.py -i test/pyinput.csv -o output/pyoutput.csv\n

\n\n1\n\n2\n\n3\n","has_starred","hub_user","immutable_tags_settings",{"_75":11,"_76":77},"enabled","rules",[78],".*","is_automated","permissions",{"_82":21,"_83":11,"_84":11},"read","write","admin","user","storage_size",20222704630,"orgData","recentTags",[91],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":100,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},"content_type","creator",1025880,"full_size",648341497,"id",4311578,"images",[101],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},"architecture","amd64","digest","sha256:9b4fcaf23c2385f16371758128c0cfc64cc8f8acbc9dfae6d8cd52873d7d61e0","features","","last_pulled","2026-10-06T12:29:13.224068605Z","last_pushed","2017-09-08T00:22:09Z","os","linux","os_features","os_version","size","variant","2017-09-08T00:22:08.035154Z","last_updater","last_updater_username","media_type","latest","repository",855177,"tag_last_pulled","tag_last_pushed","tag_status","v2","sbxKitMetadata","isPotentialSpam","isHubSbxKitsEnabled","showSbxLogoOverride","sbxTrademarkText","displaySecurityMetadata","tagAttestations","tagsHaveAttestations",[],"selectedTag","attestationsUrl","routes/_layout.r.$namespace.$reponame.tags",{"_38":142,"_149":150,"_136":159,"_160":11},{"_40":143,"_42":144,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":145,"_57":35,"_32":33,"_58":59,"_60":44,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":146,"_79":21,"_80":148,"_85":33,"_86":87},[],[44],[56],{"_75":11,"_76":147},[78],{"_82":21,"_83":11,"_84":11},"tags",{"_151":64,"_152":-5,"_153":-5,"_154":155},"count","next","previous","results",[156],{"_92":44,"_93":94,"_95":96,"_97":98,"_99":157,"_50":118,"_119":94,"_120":33,"_121":56,"_57":122,"_123":124,"_125":109,"_126":118,"_127":66,"_128":21},[158],{"_102":103,"_104":105,"_106":107,"_108":109,"_110":111,"_112":113,"_114":107,"_115":-5,"_116":96,"_63":66,"_117":-5},[],"canDeleteTags","actionData","errors"]

fullstackml/oneboxml-spark-sql

Tags for fullstackml/oneboxml-spark-sql