This is a lightweight framework to perform User Acceptance Testing (UAT) using a Behavior Driven Development (BDD) model.
Why another framework?
In addition to unit testing and system testing we need a simple framework to perform arbitrary automation across products. The target is "happy path" integration testing based on use cases, not comprehensive deep-dive testing.
- enable automation accross multiple applications. Automation must be easy to write against application APIs.
- support remote host commands via SSH
- follow Behavior Driven Development (BDD) best practice to define workflows in a semi-formal language outside of implmentation. See BDD
- test results should be displayed as pass/fail based on user stories
- plug into continuous integration (CI)
- automate application integration for stage and demonstration environments. This means the result of the test should optionally be a configured system based on executed use cases.
- assume only user actions are automated with this framework. Use puppet, cloud-init, heat templates and kickstarts to take care of provisioning and initial installation configuration.
Automation needs to be really simple to write and maintain. If you can write a user story, a bash script and make sense of API documentation to make
POST calls you should be able to use this framework.
- Consistent, simple interface for application APIs.
- Simple method for running commands on remote hosts.
- Support dynamic host discovery for CI workflows.
- Familiarize yourself with Behave documentation
- Understand how the examples work in this repo.
- Make a copy of the configuration file and customize:
cp config/uat.cfg.sample config/uat.cfg
- Create an
ansible_inventoryfile for any hosts remote commands are run on:
cp config/ansible_inventory.sample config/ansible_inventory
Install python dependencies using pip. You may need to install python-devel and gcc first.
[sudo] yum install -y pip python-devel gcc [sudo] pip install -r requirements.txt
Execute tests (assumes current working directory is base of this repo)
- Run them all (very unusual):
- Run a specific feature file (common):
- Run specific scenario(s) by keyword (great for debugging):
behave -n <scenario_keyword>
--dry-runto see output but don't execute
- Run them all (very unusual):
Writing your first test
- Always start with a feature. This is a great time to freely think like a user and the end goal. Consider pairing up with a colleague and write the feature in 30 minutes.
- Run the feature you just wrote:
- Copy the output into a steps file. Some of these are already covered in existing steps. Keep them organized by target application.
- Delete redundant steps and re-word your feature lines as necessary. If a step is not implemented, fail it with
assert Falseuntil it's implemented.
- Build up the steps files to support the feature. Get it green.
- Check in the feature and get into a CI job right away. Watch it fail, fix, rinse, repeat.
By default debug print statements are captured with all stdout and stderr also. This makes debugging difficult. Pass argument
--no-capture when running Behave to view debug statements. For stderr pass in
Enable debug mode by setting environment variable:
Remote commands are executed via Ansible and SSH. For new commands try using ansible CLI then add them to a steps method.
$ ansible <host_group_from_inventory> -i config/ansible_inventory -m command -a <some shell command>
Dynamic inventory script. Example parses 'resources.json' for a specific CI system:
$ ansible cihosts -i config/central_ci_dynamic_hosts.py -m command -a <some shell command>
Running as docker container
This is a work in progress.
You may need to turn off selinux so docker can read the bindmounted files on the host.
[sudo] setenforce 0
There are a bunch of dependencies to mount. SSH keys are problematic.
[sudo] docker run -it \ -v /path/to/.ssh:/root/.ssh \ -v /path/to/UATFramework/resources.json:/uatframework/resources.json \ -v /path/to/UATFramework/config:/uatframework/config \ aweiteka/uatframework:wip [features/my.feature] [behave_opts]
General product documentation