Sign inSign up

semc/appion-protomo

By semc

•Updated almost 7 years ago

A Docker-based appion-protomo distribution.

Image
0

2.2K

semc/appion-protomo repository overview

⁠appion-protomo v1.2.2

A Docker-based distribution of the Appion-Protomo fiducial-less tilt-series alignment suite.

Protomo aligns tilt-series based only on the accumulated SNR of the sample — no fiducials or features are used.

Click to see how Protomo alignment works seehow

⁠Prerequisites

  • A modern operating system (Linux, MacOS, Windows) with 16+ GB of RAM (32+ GB is highly recommended).

  • Tilt-series stacks and tilt image information (ideally in SerialEM or TOMO4 format, including stacks and info files).

  • Basic Unix/Linux knowledge.

Do you have Docker installed? If not, click here

⁠Linux/MacOS    (command line)

Click to expand

Download and install Docker 1.21 or greater for Linux⁠ or MacOS⁠.

Consider using a Docker 'convenience script' to install (search on your OS's Docker installation webpage).

Launch docker according to your Docker engine's instructions, typically docker start.

Note: You must have sudo or root access to install Docker. If you do not wish to run Docker as sudo/root, you need to configure user groups as described here: https://docs.docker.com/install/linux/linux-postinstall/⁠

Note: You may need to update your Docker installation by commandline before being able to install Appion-Protomo.

⁠Windows    (GUI & command line)

Click to expand

Download and install Docker Toolbox for Windows⁠.

Launch Kitematic.

If on first startup Kitematic displays a red error suggesting that you run using VirtualBox, do so.

Note: Docker Toolbox for MacOS⁠ has not yet been tested.

⁠Amazon EC2    (pre-installed!)

Click to expand

In Amazon EC2, search the public Amazon AMI instances for 'Appion-Protomo' and start an instance. This is a fully-configured Linux Docker Appion-Protomo with test datasets; everything is installed and ready to use!

⁠What is Docker?

This tutorial explains why Docker is used for Appion-Protomo.⁠


⁠Installation

Do you have Appion-Protomo installed in your Docker? If not, click here

⁠Linux/MacOS    (command line)

Click to expand

git clone http://github.com/nysbc/appion-protomo
cd appion-protomo
./run.sh

Note: The installation directory will contain the data directory. It is recommended that you reserve at least 10 GB of space for each tilt-series alignment.

This performs the following operations (click to expand):

  • Downloads the semc/appion-protomo docker image from Docker Hub⁠,

  • Creates a Docker volume to persist the Mariadb database,

  • Mounts ~/appion-protomo/emg/data on the host side to /emg/data inside the running container,

  • Mounts the mariadb-database Docker volume to /var/lib/mysql inside the running container,

  • Mounts the ~/appion-protomo directory to /local_data inside the running container,

  • Opens ports 80 for web traffic, 3306 for database traffic, and 5901 for VNC'ing into the running container,

  • Waits for the mysqld_safe database daemon to launch (for ~10 seconds, but could in rare instances take longer).

⁠Windows    (GUI & command line)

Click to expand

In the Kitematic search bar type: semc/appion-protomo.

Click Create on the semc appion-protomo repository and wait for the container to download and start.

Click for image seehow

In Windows, you may need to tweak your settings (click to expand)

If you needed to run Docker using VirtualBox, then you will need to increase the amount of RAM allocated to Docker:

  • Stop the Docker container,

  • Open VirtualBox,

  • Shut down the running virtual machine,

  • Edit the Settings for the virtual machine you just shut down,

    • Increase the RAM to 8+ GB,

  • Re-start the container.

Click for images seehow

  • Close Kitematic

  • Increase the allocated RAM for the virtual machine:

  • Optional: Increase the CPUs available for the virtual machine:

  • Start Kitematic:

⁠Amazon EC2    (pre-installed!)

Click to expand

Everything is already installed and ready to use!


⁠Usage

Is this your first time using Appion-Protomo in your Docker? If so, click here

⁠Launch Docker

Make sure Docker is running:

  • On CentOS or Ubuntu, type: sudo systemctl start docker.
  • On OSX with Docker Toolbox, launch the 'Docker Quickstart Terminal'.
  • On Windows, Kitematic launches Docker automatically.

⁠Linux/MacOS    (command line)

Click to expand

  • If you have not already launched the Appion-Protomo container, do:

./run.sh

  • You should run Appion-Protomo commands from inside a VNC viewer as appionuser and not as the root user. First, download and install a VNC viewer such as TigerVNC or RealVNC:

https://bintray.com/tigervnc/stable/tigervnc/1.9.0⁠

https://www.realvnc.com/en/connect/download/viewer/macos/⁠

Use your IP and TCP port like 192.168.99.100:5901 as the VNC address with password appion-protomo.

You might try docker-machine ip to find your IP.

Here are some more possible IP addresses/names: localhost, 0.0.0.0, 127.0.0.1, 192.168.0.1

  • To ssh into the container as root, do:

./exec.sh

Click for additional Docker Appion-Protomo container commands

  • To kill the container (this does not delete the container), do:

./kill.sh (Warning, this will kill any other Docker containers you have running as well, use with caution!)

  • To remove delete the container, but not the mounted data in /emg/data or the volume at /var/lib/mysql, do:

./rmContainers.sh (Warning: This will delete any other inactive containers you may have running as well!)

  • To delete the mariadb-database volume, do:

./rmVolume.sh (Warning: This will delete the database and anything you have uploaded in Appion! The volume will be recreated the next time you do ./run.sh, but you will have an empty Appion database!)

  • To re-build the appion-protomo image from scratch, do:

./build.sh (This will take a while!)

  • To access the Appion webviewer, navigate to:

http://192.168.99.100 (Your IP address will vary depending on your Docker installation. Possible addresses: localhost, 0.0.0.0, 127.0.0.1, 192.168.0.1)

⁠Windows    (GUI & command line)

Click to expand

  • To get to Appion-Protomo in your web browser, in Kitematic click on Settings the Hostname/Ports. Click on the IP next to Docker Port 80.

  • This should open your web browser to the Apache test page. Append /myamiweb/ to the URL to access Appion.

Click for images seehow

  • This should open up your web browser to the displayed address:

  • Append /myamiweb/ to the address:


  • To VNC into the container, download and install a VNC viewer such as TigerVNC or RealVNC:

https://bintray.com/tigervnc/stable/tigervnc/1.8.0⁠

https://www.realvnc.com/en/connect/download/viewer/windows/⁠

Use the IP and port found in the running Kitematic container's Settings >> Hostname/Ports next to Docker port 5901 as the VNC address with password appion-protomo.

Click for images seehow

  • Type this ip:port into your VNC viewer (password: appion-protomo):

  • The VNC viewer should open:

  • This allows you to run commands and use the manual tilt-series aligner.


⁠Example #1 starthere

Single particle tilt-series from EMPIAR-10129⁠)

Click to expand:

Appion-Protomo tilt-series alignment overview

Expected results after following this example tutorial

  • Initial tilt-series before any alignment:

  • Tilt-series (left) and reconstruction (right) after coarse alignment:

  • Tilt-series (left) and reconstruction (right) after Appion-Protomo refinement:


Upload first SerialEM-formatted tilt-series to a new session

Make sure your container is running via the ./run.sh script.

  • From inside the appion-protomo directory (on the host machine, not inside the container), do the following:
Download the following hemagglutinin SerialEM tilt-series (collected on an F20 with a DE-20; from [EMPIAR-10129]):

https://drive.google.com/open?id=1remm05G-R7w_6GLjW1R_jmV6Qzg0fcMx

tar -zxvf HAslow1.tar.gz <your appion-protomo/emg/data/tiltseries/ directory>

./exec.sh

> Now we are inside the container

For Windows you need to copy tilt-series to the container because there are no mounted drives. Click for image instructions seehow

  • Click on DOCKER CLI on the bottom-left of Kitematic:

  • Nativate to the directory containing your tilt-series, use docker ps to get your container id, and use docker cp to copy the directory's contents to the container:

  • Now your tilt-series should be in /home/appionuser/ in your container:


  • Navigate to your web portal at an address like

192.168.99.100/myamiweb

Select Project DB

Select Add a new project

  • Fill in the form for your project and click add

Select View Projects

Select the link on the name of your newly created project (NOT the pencil-editing icon)

Select create processing db and wait for the page to reload

Select upload images to new session

  • On the next page click the dropdown for Images grouped by and select SerialEM Tilt Series

Enter a description for Session Description

Enter the microscope's Cs value (2.0 mm for the provided hemagglutinin tilt-series)

Enter the SerialEM stack path

Enter the SerialEM mdoc path

Enter the voltage (200 kV for the provided hemagglutinin tilt-series)

Select Just Show Command at the bottom of the page.

  • On the next page, copy the entire uploadSerialEM.py command

Next, VNC to vnc://192.168.99.100:5901 (replace with your VNC ip:port) with password appion-protomo

Paste the command to a terminal prompt and hit enter

  • Wait for uploadSerialEM.py to finish processing.

Note: If uploadSerialEM.py gives an error regarding binning, then edit the file and change line 79 to: binning = 1 . This bug will be fixed in the next version.

Click for images seehow

  • The tilt-series has been uploaded. Now let's go see it:

  • All information from the SerialEM mdoc file has been imported - order of collection (for dose compensation), magnification, defocus, pixelsize, dose, and tilt angles. Tilt images are listed in chronological order from bottom to top.

Upload additional SerialEM-formatted tilt-series to an existing session

Place multiple SerialEM stacks and mdoc files (each pair with the same basename) into emg/data/tiltseries

Remove or move the previously-uploaded tilt-series stack and mdoc files.

  • Navigate to 192.168.99.100/myamiweb

Select Image Viewer

Make sure your project is selected in the dropdown (you should now see images in the image viewer panel)

If you don't see images then you first need to upload one tilt-series to a new session.

Click processing at the top

  • On the next page click Upload more images on the left

  • On the next page click the dropdown for Images grouped by and select SerialEM Tilt Series

Enter the SerialEM directory path

Enter the voltage

Select Just Show Command at the bottom of the page

  • On the next page, copy the entire uploadSerialEM.py command

Next, VNC to vnc://192.168.99.100:5901 with password appion-protomo

Paste the command to a terminal prompt and hit enter

  • Wait for uploadSerialEM.py to finish processing.

Click for images seehow

Coarse align tilt-series

  • Navigate to your session in the image viewer and click processing.

  • On the next page click Align Tilt-Series on the left.

  • On the next page click Protomo 2.4.1.

  • On the next page, Protomo Coarse Alignment, select the tilt-series you wish to align from the box.

    • This will fill in the Run name and present a number of parameters below.

Note: All parameters in the Appion-Protomo tilt-series alignment workflow contain extensive help files that are accessible by simply pointing at the parameter name.

For most samples select 10 nm as the Object Size in Parameter Presets.

Choose whether you wish to Estimate Defocus of the Untilted Plane or Dose Compensate the tilt-series.

Dose compensation often helps tilt-series alignment unless your sample is particularly thick, as is often the case with FIB-milled specimen.

Select Just Show Command at the bottom of the page.

  • On the next page, copy the entire protomo2aligner.py command and run it in the Docker container.

Click [Click here to View Coarse Alignment Results after protomo2aligner.py --coarse has been run] →.

  • On the next page, Protomo Coarse Alignment Summary, watch the Tilt-Series After Coarse Alignment video.

    • If there are any tilt images that are grossly shifted relative to the others then proceed to the next section Manually fix tilt-series if necessary. If not, proceed onto Refine tilt-series.

    • If there are any tilt images that contain objects that significantly occlude objects of interest, take note of their tilt angles.

Note: The objects in the tilt-series that move the least image-to-image will have a higher chance of being closest to the center of the tomogram after refinement. See the Critical Note in the upcoming 'Refine tilt-series' section.

Click for images seehow

  • Watch the tilt-series video to estimate the tilt azimuth (vertical is ±90°, horizontal is 0° or 180°)

  • This tilt-series has a tilt azimuth of approximately ±90°:

Manually fix tilt-series if necessary

  • If there are any tilt images that are grossly shifted relative to the others then continue through this section.

Copy the command at the bottom of the Protomo Coarse Alignment webpage beginning with /usr/bin/python.

  • VNC into your Docker container.

Paste the command into a terminal in the container.

Follow the bolded instructions that first appear after running the manual alignment command.

Practically, find an object that you think might be near the middle of the ice and align each image such that this object does not move on the screen.

Note: The objects in the tilt-series that move the least image-to-image will have a higher chance of being closest to the center of the tomogram after refinement. See the Critical Note in the upcoming 'Refine tilt-series' section.

Refine tilt-series

  • On the Protomo Coarse Alignment Summary webpage click on [Click here to continue onto Refinement if the Coarse Alignment results are satisfactory] →.

  • On the next page, select Basic or Advanced Settings Verbosity.

For most tilt-series, select 10 nm object(s) - Steep lowpass as the Object Size in Parameter Presets.

Select the largest Search Area that is in common between all tilt-images in the previously-viewed tilt-series in Parameter Presets.

  • If there are any tilt images that need to be removed, input their tilt angles in the General Parameters, as an Angle Limit and/or using Exclude Images by Tilt Angles.

  • If the tilt-series was manually aligned, choose More Manual for Begin with Coarse Alignment Results or Initial Alignment? in the General Parameters.

  • Under Window Parameters, input comma-separated values for the estimated Thickness(es), where the thickness corresponds to the thickness of the objects to be aligned to or their combined thickness.

Select Just Show Command at the bottom of the page.

  • On the next page, copy the entire protomo2multirefine.py or protomo2aligner.py command and run it in the Docker container.

On the left under Align Tilt-Series click ## runs processing or finished to access the main alignment summary webpage.

  • On the Protomo Alignment Summary webpage you can watch as each tilt-series aligns.

Note: The tilt-azimuth for each microscope relative to Appion-Protomo convention needs to be determined experimentally. Protomo will often refine the tilt azimuth properly after 10-50 iterations. Once this value is known, it should be inputted into all future tilt-series alignments for that microscope in the General Parameters during Coarse Alignment.

Note: The alignment thickness cannot be determined prior to tilt-series alignment. If a tilt-series alignment did not align as well as you wish, try re-aligning with additional alignment thickness values.

Critical Note: The objects in the search area (technically, search volume defines by the [search area] x [thickness]) in a Protomo reconstruction video are the objects that are being used for tilt-series refinement. The highest SNR objects are typically driven towards the middle of the reconstruction/search volume in the z-direction during iterative alignment. If the alignment thickness is not chosen appropriately, this can cause the following issues:

  • If the sample consists of single particles in 200 nm thick ice, for instance, where the particles are on both air-water interfaces, then an alignment thickness less than ~200 nm will likely align to only one of the layers of particles whereas an alignment thickness greater than ~200 nm will likely align to both layers of particles. In this scenario, if the particle is ~10 nm in extent, then two completely different alignment thicknesses might be found to be ideal: ~10 nm and ~200 nm. However, if the alignment using the ~10 nm alignment thickness is reconstructed, the layer of particles that was aligned to will be well resolved while the layer of particles that was not aligned to will be less well resolved. If the alignment using the ~200 nm alignment thickness is reconstructed, both layers of particles will be well resolved. Thus the ~200 nm alignment should be used and the ~10 nm alignment should probably be discarded even though they both report that they are well aligned.
  • If the sample consists of a roughly uniformly dense set of objects, such as a FIB-milled cellular lamella or reconstituted proteins on liposomes, then alignment thicknesses up to and including the thickness of the entire sample slab in the search area might align well, however the alignment closest to the thickness of the sample slab will likely align most accurately.
  • Alignment thicknesses chosen to be significantly larger than the sample thickness will likely align worse than alignment thicknesses that are roughly the same as the sample thickness.

Click for images seehow

  • From the Protomo Alignment Summary webpage you can easily assess the alignment quality of all of your tilt-series run with multiple different alignment thicknesses. You can easily navigate to the best iteration, defocus refinement, and the alignment summary webpage.

Assess refinement

A tilt-series refinement should be assessed in the following ways:

  1. (if possible) Watch the tilt-series video to see if the alignment is potentially good. Are any images clearly shifted or rotated improperly?
    • If so, then use the command at the bottom of the refinement iteration webpage to manually fix the bad tilt images and/or remove the bad tilt images. You will likely want to further refine the tilt-series before continuing.
  2. Watch the tilt-series correlation peak video at the top of the refinement iteration webpage - the small square video. All but one frame (usually the middle frame) should be black. The other frames should have a clear correlation peak in the center of the box.
    • If there are more than one black frames, then this means that the search area had to shrink during the refinement, which i

Tag summary

Content type

Image

Digest

Size

633.5 MB

Last updated

almost 7 years ago

docker pull semc/appion-protomo