datalinked/ides-shell
Datalinked ides-shell Official Images
Image
1
270
datalinked/ides-shell repository overview
Ides-shell Of Datalinked
【💥 】github repository: https://github.com/bebee4java/ides.git
Used to start an ides-shell repl. You can use it to do:
- Load and save various data sources for data exploration (such as ETL)
- Use standard SQL for data processing
- Execute native Scala/Spark code
- And so on ......
Usage example
Start the ides shell interactive console in local mode:
docker run -it -p 4044:4040 datalinked/ides-shell --master 'local[2]' --name 'ides-test'
explain:
- -it : run the container in interactive mode, usually used with -t
- -p : publish a container's port(s) to the host. e.g. publish spark-ui's port 4040 to the host port 4044
- you can specify the startup parameters after the image name, such as:
--master 'local[2]' --name 'ides-test'
show container status
docker ps |grep ides-shell
get as follows:
➜ ~ docker ps |grep ides-shell
04b8259fa4ec datalinked/ides-shell "/bin/bash bin/ides-…" 39 seconds ago Up 38 seconds 0.0.0.0:4044->4040/tcp hardcore_galileo
➜ ~
use ides-shell container
After startup, get as follows:
2021/02/02 08:37:20 INFO ConfigUtils: ----------------------
2021/02/02 08:37:20 INFO ConfigUtils: |ides.shell.mode|true|
2021/02/02 08:37:20 INFO ConfigUtils: ----------------------
2021/02/02 08:37:20 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/opt/ides-1.0.0-bin-spark2.4/spark-warehouse').
2021/02/02 08:37:20 INFO SharedState: Warehouse path is 'file:/opt/ides-1.0.0-bin-spark2.4/spark-warehouse'.
2021/02/02 08:37:21 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
2021/02/02 08:37:23 INFO DataSourceFactory: look for the DataSource plugin from packages: tech.ides.external.datasource, tech.ides.datasource.impl
2021/02/02 08:37:23 INFO Reflections: Reflections took 97 ms to scan 2 urls, producing 8 keys and 48 values
2021/02/02 08:37:23 INFO DataSourceFactory: A total of 4 dataSource plugins scanned: [excel, csv, text, hive].
Spark context Web UI available at http://04b8259fa4ec:4040
Spark context available as 'sc' (master = local[2], app id = local-1612255039381).
Spark session available as 'spark'.
Setting default log level to 'WARN'.
To adjust logging level use sc.setLogLevel(newLevel).
Ides shell support multi-line code input, you can use offMultiLineInput command to close, use onMultiLineInput will re-enable!
Ides ScriptQueryExecListener available as 'listener'.
Welcome to
___ ________ _______ ________
|\ \|\ ___ \|\ ___ \ |\ ____\
\ \ \ \ \_|\ \ \ __/|\ \ \___|_
\ \ \ \ \ \\ \ \ \_|/_\ \_____ \
\ \ \ \ \_\\ \ \ \_|\ \|____|\ \
\ \__\ \_______\ \_______\____\_\ \
\|__|\|_______|\|_______|\_________\ version 1.0.0
\|_________|
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_181)
Type in expressions to have them evaluated.
Type :help for more information.
ides> select 1 as a as tb;
| tb.show;
|
tb: org.apache.spark.sql.DataFrame = [a: int]
+---+
| a|
+---+
| 1|
+---+
ides>
Tag summary
Content type
Image
Digest
Size
1.2 GB
Last updated
about 4 years ago
docker pull datalinked/ides-shell