berlius/universe repository overview

We currently support Linux and OSX running Python 2.7 or 3.5.

We recommend setting up a `conda environment <http://conda.pydata.org/docs/using/envs.html>`__
before getting started, to keep all your Universe-related packages in the same place.

Install Universe
~~~~~~~~~~~~~~~~
To get started, first install ``universe``:

.. code:: shell

    git clone https://github.com/openai/universe.git
    cd universe
    pip install -e .

If this errors out, you may be missing some required packages. Here's
the list of required packages we know about so far (please let us know
if you had to install any others).

On Ubuntu 16.04:

.. code:: shell

    pip install numpy
    sudo apt-get install golang libjpeg-turbo8-dev make

On Ubuntu 14.04:

.. code:: shell

    sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable  # for newer golang
    sudo apt-get update
    sudo apt-get install golang libjpeg-turbo8-dev make

On OSX:

You might need to install Command Line Tools by running:

.. code:: shell

    xcode-select --install

Or ``numpy``, ``libjpeg-turbo`` and ``incremental`` packages:

.. code:: shell

    pip install numpy incremental
    brew install golang libjpeg-turbo

Install Docker
~~~~~~~~~~~~~~

The majority of the environments in Universe run inside Docker
containers, so you will need to `install Docker
<https://docs.docker.com/engine/installation/>`__ (on OSX, we
recommend `Docker for Mac
<https://docs.docker.com/docker-for-mac/>`__). You should be able to
run ``docker ps`` and get something like this:

.. code:: shell

     $ docker ps
     CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Notes on installation

So we managed to run an agent, what did all the code actually
mean? We'll go line-by-line through the example.

* First, we import the `gym <https://github.com/openai/gym>`__ library,
  which is the base on which Universe is built. We also import
  ``universe``, which `registers
  <https://github.com/openai/universe/blob/master/universe/__init__.py>`__
  all the Universe environments.

.. code:: python

  import gym
  import universe # register the universe environments

* Next, we create the environment instance. Behind the scenes, ``gym``
  looks up the `registration
  <https://github.com/openai/universe/blob/master/universe/__init__.py>`__
  for ``flashgames.DuskDrive-v0``, and instantiates a `VNCEnv
  <https://github.com/openai/universe/blob/master/universe/envs/vnc_env.py#L88>`__
  object which has been `wrapped
  <https://github.com/openai/universe/blob/master/universe/wrappers/__init__.py#L42>`__
  to add a few useful diagnostics and utilities. The ``VNCEnv`` object
  is the *client* part of the environment, and it is not yet connected
  to a *remote*.

.. code:: python

  env = gym.make('flashgames.DuskDrive-v0')

* The call to ``configure()`` connects the client to a remote
  environment server. When called with ``configure(remotes=1)``,
  Universe will automatically create a Docker image running locally on
  your computer. The local client connects to the remote using VNC.
  (More information on client-remote communication can be found in the
  page on `universe internal communication protocols
  <doc/protocols.rst>`__. More on configuring remotes is at `remotes <doc/remotes.rst>`__.)

.. code:: python

  env.configure(remotes=1)

* When starting a new environment, you call ``env.reset()``. Universe
  environments run in real-time, rather than stepping synchronously
  with the agent's actions, so ``reset`` is asynchronous and returns
  immediately. Since the environment will not have waited to finish
  connecting to the VNC server before returning, the initial observations
  from ``reset`` will be ``None`` to indicate that there is
  not yet a valid observation.

  Similarly, the environment keeps running in the background even
  if the agent does not call ``env.step()``.  This means that an agent
  that successfully learns from a Universe environment cannot take
  "thinking breaks":  it must keep sending actions to the environment at all times.

  Additionally, Universe introduces the *vectorized* Gym
  API. Rather than controlling a single environment at a time, the agent
  can control a fixed-size vector of ``n`` environments, each with its
  own remote. The return value from ``reset`` is therefore a *vector*
  of observations. For more information, see the separate page on
  `environment semantics <doc/env_semantics.rst>`__)

.. code:: python

  observation_n = env.reset()

* At each ``step()`` call, the agent submits a vector of actions; one for
  each environment instance it is controlling. Each VNC action is a
  list of events; above, each action is the single event "press the
  ``ArrowUp`` key". The agent could press and release the key in one
  action by instead submitting ``[('KeyEvent', 'ArrowUp', True),
  ('KeyEvent', 'ArrowUp', False)]`` for each observation.

  In fact, the agent could largely have the same effect by just
  submitting ``('KeyEvent', 'ArrowUp', True)`` once and then calling
  ``env.step([[] for ob in observation_n])`` thereafter, without ever
  releasing the key using ``('KeyEvent', 'ArrowUp', False)``. The
  browser running inside the remote would continue to statefully
  represent the arrow key as being pressed. Sending other unrelated
  keypresses would not disrupt the up arrow keypress; only explicitly
  releasing the key would cancel it.  There's one slight subtlety:
  when the episode resets, the browser will reset, and will forget
  about the keypress; you'd need to submit a new ``ArrowUp`` at the
  start of each episode.

.. code:: python

  action_n = [[('KeyEvent', 'ArrowUp', True)] for ob in observation_n]

* After we submit the action to the environment and render one frame,
  ``step()`` returns a list of *observations*, a list of *rewards*, a
  list of *"done" booleans* indicating whether the episode has ended,
  and then finally an *info dictionary* of the form ``{'n': [{},
  ...]}``, in which you can access the info for environment ``i`` as
  ``info['n'][i]``.

  Each environment's ``info`` message contains useful diagnostic
  information, including latency data, client and remote timings,
  VNC update counts, and reward message counts.

.. code:: python

    observation_n, reward_n, done_n, info = env.step(action_n)
    env.render()

* We call ``step`` in what looks like a busy loop. In reality, there
  is a `Throttle
  <https://github.com/openai/universe/blob/master/universe/wrappers/__init__.py#L18>`__
  wrapper on the client which defaults to a target frame rate of 60fps, or one
  frame every 16.7ms. If you call it more frequently than that,
  ``step`` will `sleep
  <https://github.com/openai/universe/blob/master/universe/wrappers/throttle.py>`__
  with any leftover time.


Testing
=======

We are using `pytest <http://doc.pytest.org/en/latest/>`__ for tests. You can run them via:

.. code:: shell

    pytest

Run ``pytest --help`` for useful options, such as ``pytest -s`` (disables output capture) or ``pytest -k <expression>`` (runs only specific tests).

Additional documentation
========================

More documentation not covered in this README can be found in the
`doc folder <doc>`__ of this repository.

Getting help
============

If you encounter a problem that is not addressed in this README page
or in the `extra docs <doc>`__, then try our wiki page of `solutions
to common problems
<https://github.com/openai/universe/wiki/Solutions-to-common-problems>`__ -
and add to it if your solution isn't there!

You can also search through the `issues
<https://github.com/openai/universe/issues?utf8=%E2%9C%93&q=is%3Aissue>`__
on this repository and our `discussion board
<https://discuss.openai.com/c/Universe>`__ to see if another user has posted
about the same problem or to ask for help from the community.

If you still can't solve your problem after trying all of the above
steps, please post an issue on this repository.

What's next?
============

* Get started training RL algorithms! You can try out the `Universe Starter Agent <https://github.com/openai/universe-starter-agent>`_, an implementation of the `A3C algorithm <https://arxiv.org/abs/1602.01783>`_ that can solve several VNC environments.

* For more information on how to manage remotes, see the separate documentation page on `remotes <doc/remotes.rst>`__.

* Sign up for a `beta <https://docs.google.com/forms/d/e/1FAIpQLScAiW-kIS0mz6hdzzFZJJFlXlicDvQs1TX9XMEkipNwjV5VlA/viewform>`_ to get early access to upcoming Universe releases, such as tools to integrate new Universe environments or a dataset of recorded human demonstrations.


Changelog
---------
- 2017-02-08: The old location for wrappers.SafeActionSpace has been moved to wrappers.experimental.SafeActionSpace. SoftmaxClickMouse has also been moved to wrappers.experimental.SoftmaxClickMouse
- 2017-01-08: The wrappers.SafeActionSpace has been moved to wrappers.experimental.SafeActionSpace. The old location will remain with a deprecation warning until 2017-02-08.
- 2016-12-27: BACKWARDS INCOMPATIBILITY: The gym monitor is now a
  wrapper. Rather than starting monitoring as
  `env.monitor.start(directory)`, envs are now wrapped as follows:
  `env = wrappers.Monitor(env, directory)`. This change is on master
  and will be released with 0.21.0.