shyftnetwork/shyftgeth repository overview

⁠Go Empyrean

go-empyrean is based on a fork of go-ethereum. Much of the functionality and process for starting go-empyrean is the same as that for a regular ethereum node - as reflected in the notes detailed under the heading Go Ethereum below. Documentation for changes and enhancements added by Shyft is detailed under the section Shyft Notes below.

⁠SHYFT NOTES

https://shyftnetwork.github.io/go-empyrean/#setup⁠

⁠Tag Release Command

gren release --tags=v0.8.2 --data-source=prs --override

⁠Dependencies

• go 1.10
• postgres 10

To install go please review the installation docs here⁠, but ensure you download version 1.10. If you would like to install go with a script please check out this repo here⁠.

To install postgres please review the installation docs here⁠.

⁠Govendor and Packages/Dependencies

go get -u github.com/kardianos/govendor

export GOPATH=$HOME/go
export PATH=$PATH:$HOME/go/bin

$GOPATH/src/github.com/ShyftNetwork/

Geth uses govendor to manage packages/dependencies: Go Vendor⁠

This has some more information: Ethereum Wiki⁠

To add a new dependency, run govendor fetch , and commit the changes to git. Then the deps will be accessible on other machines that pull from git.

⁠Running Locally

To begin running locally, please ensure you have correctly installed go 1.10 and postgres (make sure postgres is running). Once cloned, in a terminal window run the following command:

Before running any CLI options ensure you run make geth in the root directory.

./shyft-config/shyft-geth.sh --setup This sets up postgres and the shyft chain db

./shyft-config/shyft-geth.sh --start This starts GETH

At this point you should see GETH running in the terminal and if you opened your postgres instance you should see data being populated into the tables. It might look something similiar to the image below.

To stop Geth, crtl+C in the terminal window, if you proceed with the start script mentioned above the Shyft chain will begin from the last block height, if you wish to start the chain fresh from genesis follow the below steps:

./shyft-config/shyft-geth.sh --reset This drops postgres and chaindb data

./shyft-config/shyft-geth.sh --start Starts GETH

To see transactions being submitted on the network see the sendTransactions command in the CLI section of this readme.

#####SHH/Whisper The shyft go_empyrean node, unlike go ethereum starts the SHH whisper client by default. This is to facilitate broadcast messaging from the shyft js bridge to each of the mining nodes.

To disable the whisper client a startup flag --disablewhisper is provided, which must be passed into the command line when starting up geth.

geth --disablewhisper

To overwrite the default whisper variables, the following flags are also provided:

--shh.maxmessagesize - sets the maximum message size fir the whisper client (integer) -(default: 1048576)  --shh.maxmessagesize=128
--shh.pow - the minimum POW accepted for processing whisper messages (float64 - default: 0.2) --shh.pow=0.3
--shh.restrict-light - restrictions connections between two whisper light clients (boolean - default: true) --shh.restrict-light

To authenticate whisper messages a call is made to a smart contract that has a predetermined address on the blockchain. Upon starting up a geth node if a user wishes to use this functionality they should ensure that the WhisperSignersContract variable in config.toml contains the contract address for authentication of Whisper Signers.

The authentication of WhisperSigner's broadcast messages relies on automatically generated go contract bindings using the the abigen cmd line utility. Should the contract be changed or modified these bindings will need to be regenerated. Steps for regenerating are as follows:

1. Generate the abi for the subject contract and save it at ./generated_bindings/contract_abis/whispersigner_abi.json.

2. Run the following command to regenerate the contract bindings:

./build/bin/abigen  --sol ./shyft-config/shyft-cli/web3/validSignersDeploy/ValidSigners.sol  --pkg shyft_contracts --out generated_bindings/whisper_signer_binding.go

It should be noted that the authentication currently relies on a smart contract boolean returning function [isValidSigner(bool)], that for a given signature address returns true if the contract or contract owner has a public key matching the signature.

⁠Docker Images

Two sets of Docker Images are available for ShyftGeth, the Postgresql Database, and the Shyft Blockchain Explorer, which can be used for local development and testnet connection. The development settings are included in docker-compose.yml, the testnet settings are included in docker-compose.production.yml. To launch these containers you will need to have docker-compose installed on your computer. Installation instructions for docker-compose are available here⁠.

To build the images for the first time please run the following command:

./shyft-geth --setup # clears persisted directories prior to docker build

docker-compose up --build

If you would like to reinitialize/rebuild the docker images you can run the above mentioned command as well.

To launch ShyftGeth, PG, the ShyftBlock Explorer Api and UI anytime after initial build - issue the following commands from the root of the project directory:

./shyft-geth --setup # clears persisted directories prior to docker build

docker-compose up

To stop/pause mining - enter:

docker-compose stop

And then just issue docker-compose up to continue mining.

⁠Docker Postgresql - DB Connection

From your local machine you can view the database by connecting to the database in the container at 127.0.0.1:8001

Use the following credentials:

⁠Docker Block Explorer Api

To access the shyftBlockExplorer open a browser and visit

http://localhost:3000

To rebuild any one of the services- issue the following commands:

Services:

• ShyftGeth
• Postgres Instance
• Shyft Explorer API
• Shyft Example Explorer UI

docker-compose up -d --no-deps --build <docker compose file service name>

ie. for shyftBlockExplorerApi:

docker-compose up -d --no-deps --build shyft_block_api

The Postgresql Database Container will persist the database data to the directory ./pg-data _. So if you do want to reinitialize the database you should delete this directory as well as the blockchain data directories (./shyftData ./privatenet) prior to launching the docker containers. There is a shell script available to delete these folders to run it execute the following command:

./shyft-config/shyft-cli/resetShyftGeth

Blockchain data is persisted to ./ethash/.ethash and ./shyftData__. If you would like to reset the test blockchain you will need to delete the __./ethash ./shyftData & ./privatenet__ directories.

The docker container for the ShyftBlockExplorerApi utilizes govendor to minimize its image size. If you would like the docker image for this container to reflect any uncommitted changes which may have occurred in the go-empyrean repository, ie. changes with respect to go-empyrean core (ie. cryptographic functions and database). Prior to launching the docker containers you should rebuild the vendor directory for the shyftBlockExplorerApi - by executing the following steps:

Remove existing shyftBlockExplorerApi vendor.json and vendored components:

rm -rf shyftBlockExplorerApi/vendor

reinitialize vendor.json

cd shyftBlockExplorerApi && govendor init

rebuild vendor.json using latest uncommitted changes

govendor add +external

Due to a bug in govendor and it not being able to pull in some dependencies that are c-header files you should execute the following commands - see these issues - which whilst closed appears to have not been fixed: https://github.com/kardianos/govendor/issues/124⁠ && https://github.com/kardianos/govendor/issues/61⁠

govendor remove github.com/ShyftNetwork/go-empyrean/crypto/secp256k1/^

govendor fetch github.com/ShyftNetwork/go-empyrean/crypto/secp256k1/^

NB: The Shyft Geth docker image size is 1+ GB so make sure you have adequate space on your disk drive/

⁠Shyft BlockExplorer API

In order to store the block explorer database, a custom folder was created ./shyft_schema that contains all the necessary functions to read and write to the explorer database.

The main functions exist in ./core/shyft_database_util.go and ./core/shyft_get_utils.go

To run the block explorer rest api that queries the postgres instance and returns a json body, open a new terminal window, navigate to the root directory of the project and run the following command:

go run blockExplorerApi/*.go

This will start a go server on port 8080 and allow you to either run the pre-existing block explorer or query the api endpoints. Its important to note, that if you have nothing in your postgres database the API will return nothing.

⁠Shyft Block Explorer UI

To demonstrate the ability to create your own block explorer, a custom folder was created ./shyftBlockExplorerUI that contains an example block explorer using react!

To run the Block Explorer UI, ensure that you have the API running as mentioned above. Then run the following command in a terminal:

cd shyftBlockExplorerUI

npm install

npm run start

This will start a development server on port 3000 and spin up an example block explorer that uses the API to query the postgres database.

TODO

• Find better dependency management solution that pulls in c header files without manual intervention
• Reduce size of the ShytfGeth docker container which is responsible for mining and running the blockchain
• Adjust docker scripts and ports to facilitate sending of test transactions
• Modify Docker scripts to facilitate hot reloading during development

⁠CLI

Run ./shyft-geth with one of the following flags:

• --setup - Setups postgres and the shyft chain db.
• --start - Starts geth.
• --reset - Drops postgress and chain db, and reinstantiates both.
• --js [web3 filename] - Executes web3 calls with a passed file name. If the file name is sendTransactions.js, ./shyft-geth --js sendTransactions.

⁠Chain Rollbacks

For development and testing purposes only, until a formal messaging system has been incorporated within go-empyrean, an endpoint is available and freely accessible to trigger a chain and postgresql database rollback.

To trigger a chain/pg database rollback the following command should be executed:

curl <node ip address>:8081/rollback_blocks/<block hashheader to rollback to>

ie. curl localhost:8081/rollback_blocks/0x6c7db5b09bda0277b480aece97d2efac70838cad4fe6ae45f68410c8cd7cd640

⁠Go Ethereum

Official golang implementation of the Ethereum protocol.

Automated builds are available for stable releases and the unstable master branch. Binary archives are published at https://geth.ethereum.org/downloads/⁠.

⁠Building the source

For prerequisites and detailed build instructions please read the Installation Instructions⁠ on the wiki.

Building geth requires both a Go (version 1.9 or later) and a C compiler. You can install them using your favourite package manager. Once the dependencies are installed, run

make geth

or, to build the full suite of utilities:

make all

⁠Executables

The go-ethereum project comes with several wrappers/executables found in the cmd directory.

⁠Running geth

Going through all the possible command line flags is out of scope here (please consult our CLI Wiki page⁠), but we've enumerated a few common parameter combos to get you up to speed quickly on how you can run your own Geth instance.

⁠Full node on the main Ethereum network

By far the most common scenario is people wanting to simply interact with the Ethereum network: create accounts; transfer funds; deploy and interact with contracts. For this particular use-case the user doesn't care about years-old historical data, so we can fast-sync quickly to the current state of the network. To do so:

$ geth console

This command will:

• Start geth in fast sync mode (default, can be changed with the --syncmode flag), causing it to download more data in exchange for avoiding processing the entire history of the Ethereum network, which is very CPU intensive.
• Start up Geth's built-in interactive JavaScript console⁠, (via the trailing console subcommand) through which you can invoke all official web3 methods⁠ as well as Geth's own management APIs⁠. This tool is optional and if you leave it out you can always attach to an already running Geth instance with geth attach.

⁠Full node on the Ethereum test network

Transitioning towards developers, if you'd like to play around with creating Ethereum contracts, you almost certainly would like to do that without any real money involved until you get the hang of the entire system. In other words, instead of attaching to the main network, you want to join the test network with your node, which is fully equivalent to the main network, but with play-Ether only.

$ geth --testnet console

The console subcommand have the exact same meaning as above and they are equally useful on the testnet too. Please see above for their explanations if you've skipped to here.

Specifying the --testnet flag however will reconfigure your Geth instance a bit:

• Instead of using the default data directory (~/.ethereum on Linux for example), Geth will nest itself one level deeper into a testnet subfolder (~/.ethereum/testnet on Linux). Note, on OSX and Linux this also means that attaching to a running testnet node requires the use of a custom endpoint since geth attach will try to attach to a production node endpoint by default. E.g. geth attach <datadir>/testnet/geth.ipc. Windows users are not affected by this.
• Instead of connecting the main Ethereum network, the client will connect to the test network, which uses different P2P bootnodes, different network IDs and genesis states.

Note: Although there are some internal protective measures to prevent transactions from crossing over between the main network and test network, you should make sure to always use separate accounts for play-money and real-money. Unless you manually move accounts, Geth will by default correctly separate the two networks and will not make any accounts available between them.

⁠Full node on the Rinkeby test network

The above test network is a cross client one based on the ethash proof-of-work consensus algorithm. As such, it has certain extra overhead and is more susceptible to reorganization attacks due to the network's low difficulty / security. Go Ethereum also supports connecting to a proof-of-authority based test network called Rinkeby⁠ (operated by members of the community). This network is lighter, more secure, but is only supported by go-ethereum.

$ geth --rinkeby console

⁠Configuration

As an alternative to passing the numerous flags to the geth binary, you can also pass a configuration file via:

$ geth --config /path/to/your_config.toml

To get an idea how the file should look like you can use the dumpconfig subcommand to export your existing configuration:

$ geth --your-favourite-flags dumpconfig

Note: This works only with geth v1.6.0 and above.

⁠Docker quick start

One of the quickest ways to get Ethereum up and running on your machine is by using Docker:

docker run -d --name ethereum-node -v /Users/alice/ethereum:/root \
           -p 8545:8545 -p 30303:30303 \
           ethereum/client-go

This will start geth in fast-sync mode with a DB memory allowance of 1GB just as the above command does. It will also create a persistent volume in your home directory for saving your blockchain as well as map the default ports. There is also an alpine tag available for a slim version of the image.

Do not forget --rpcaddr 0.0.0.0, if you want to access RPC from other containers and/or hosts. By default, geth binds to the local interface and RPC endpoints is not accessible from the outside.

⁠Programatically interfacing Geth nodes

As a developer, sooner rather than later you'll want to start interacting with Geth and the Ethereum network via your own programs and not manually through the console. To aid this, Geth has built-in support for a JSON-RPC based APIs (standard APIs⁠ and Geth specific APIs⁠). These can be exposed via HTTP, WebSockets and IPC (unix sockets on unix based platforms, and named pipes on Windows).

The IPC interface is enabled by default and exposes all the APIs supported by Geth, whereas the HTTP and WS interfaces need to manually be enabled and only expose a subset of APIs due to security reasons. These can be turned on/off and configured as you'd expect.

HTTP based JSON-RPC API options:

• --rpc Enable the HTTP-RPC server
• --rpcaddr HTTP-RPC server listening interface (default: "localhost")
• --rpcport HTTP-RPC server listening port (default: 8545)
• --rpcapi API's offered over the HTTP-RPC interface (default: "eth,net,web3")
• --rpccorsdomain Comma separated list of domains from which to accept cross origin requests (browser enforced)
• --ws Enable the WS-RPC server
• --wsaddr WS-RPC server listening interface (default: "localhost")
• --wsport WS-RPC server listening port (default: 8546)
• --wsapi API's offered over the WS-RPC interface (default: "eth,net,web3")
• --wsorigins Origins from which to accept websockets requests
• --ipcdisable Disable the IPC-RPC server
• --ipcapi API's offered over the IPC-RPC interface (default: "admin,debug,eth,miner,net,personal,shh,txpool,web3")
• --ipcpath Filename for IPC socket/pipe within the datadir (explicit paths escape it)

You'll need to use your own programming environments' capabilities (libraries, tools, etc) to connect via HTTP, WS or IPC to a Geth node configured with the above flags and you'll need to speak JSON-RPC⁠ on all transports. You can reuse the same connection for multiple requests!

Note: Please understand the security implications of opening up an HTTP/WS based transport before doing so! Hackers on the internet are actively trying to subvert Ethereum nodes with exposed APIs! Further, all browser tabs can access locally running webservers, so malicious webpages could try to subvert locally available APIs!

⁠Operating a private network

Maintaining your own private network is more involved as a lot of configurations taken for granted in the official networks need to be manually set up.

⁠Defining the private genesis state

First, you'll need to create the genesis state of your networks, which all nodes need to be aware of and agree upon. This consists of a small JSON file (e.g. call it genesis.json):

{
  "config": {
    "chainId": 0,
    "homesteadBlock": 0,
    "eip155Block": 0,
    "eip158Block": 0
  },
  "alloc": {},
  "coinbase": "0x0000000000000000000000000000000000000000",
  "difficulty": "0x20000",
  "extraData": "",
  "gasLimit": "0x2fefd8",
  "nonce": "0x0000000000000042",
  "mixhash":
    "0x0000000000000000000000000000000000000000000000000000000000000000",
  "parentHash":
    "0x0000000000000000000000000000000000000000000000000000000000000000",
  "timestamp": "0x00"
}

The above fields should be fine for most purposes, although we'd recommend changing the nonce to some random value so you prevent unknown remote nodes from being able to connect to you. If you'd like to pre-fund some accounts for easier testing, you can populate the alloc field with account configs:

"alloc": {
  "0x0000000000000000000000000000000000000001": {"balance": "111111111"},
  "0x0000000000000000000000000000000000000002": {"balance": "222222222"}
}

With the genesis state defined in the above JSON file, you'll need