Public Repository

Last pushed: a month ago
Short Description
The Pacifica Archive Interface provides REST API to an HSM archive
Full Description

Pacifica Archive Interface

This code is to provide the archive interface for the rest of the
Pacifica code base. This code consists of some very specific algorithms
and APIs to support data that might exist on tape or spinning disk.

Building and Installing

This code depends on the following libraries and python modules:

HPSS Client 7.4.1p2
Python JSON
Python CTypes
Python DocTest

This is a standard python distutils build process.

python ./ build
python ./ install

Running It

There are three ways of running the archive interface; POSIX, ORACLE_HMS_SIDEBAND and HPSS.

Posix File System Backend

python ./scripts/ -t posix -p 8080 -a --prefix /path

HPSS Archive Backend

python ./ -t hpss  -p 8080 -a --prefix /path


python ./ -t hmssideband  -p 8080 -a --prefix /path

Config file

You can also pass a config file via the --config option. This is the first option and has highest priority.
Second highest priority is the environment variable ARCHIVEI_CONFIG. This will be looked at if the --config
option was not used.
The final option is the application will default to config.cfg if neither of the first two options occured.

Config File Example:

Note here that the different backends use different config options. These are required for their respected
archive types.

use_id2filename = false

user = hpss.unix
auth = /var/hpss/etc/hpss.unix.keytab

sam_qfs_prefix = /tmp/path
schema = schema_name
host = host
user = user
password = pass
port = 3306

ID Mapping to File Names

The Pacifica software depends on a flat ID space for indexing files. This needs
to map to filenames on the backend storage in a nice way. To limit the number of
files in a single directory (or number of directories in a directory) we use the
algorithm in archiveinterface.id2filename. This takes a number and breaks it
into bytes. Each byte is then represented in hex and used to build the directory

For example id2filename(12345) becomes /39/3039 on the backend file system.

API Examples

Verify working

To verify the system is working do a GET against the system with no id specified.

curl -X GET

Sample output:

    "message": "Pacifica Archive Interface Up and Running"

Put a File

The path in the URL should be only an integer specifying a unique
file in the archive. Sending a different file to the same URL will
over-write the contents of the previous file. Setting the Last-
Modified header sets the mtime of the file in the archive and is

curl -X PUT -H 'Last-Modified: Sun, 06 Nov 1994 08:49:37 GMT' --upload-file /tmp/foo.txt

Sample output:

    "message": "File added to archive",
    "total_bytes": "24"

Get a File

The HTTP GET method is used to get the contents
of the specified file.

curl -o /tmp/foo.txt

Sample output (without -o option):
"Document Contents"

Status a File

The HTTP HEAD method is used to get a JSON document describing the
status of the file. The status includes, but is not limited to, the
size, mtime, ctime, whether its on disk or tape. The values can be found
within the headers.

curl -I -X HEAD

Sample output:

HTTP/1.0 204 No Content
Date: Fri, 07 Oct 2016 19:51:37 GMT
Server: WSGIServer/0.1 Python/2.7.5
X-Pacifica-Messsage: File was found
X-Pacifica-File: /tmp/12345
Content-Length: 18
Last-Modified: 1473806059.29
X-Pacifica-Ctime: 1473806059.29
X-Pacifica-Bytes-Per-Level: (18L,)
X-Pacifica-File-Storage-Media: disk
Content-Type: application/json

Stage a File

The HTTP POST method is used to stage a file for use. In posix this
equates to a no-op on hpss it stages the file to the disk drive.

curl -X POST

Sample Output:

    "file": "/12345",
    "message": "File was staged"

Move a File

The HTTP PATCH method is used to move a file.
The upload file contains the path to current file on archive
The Id at the end of the url is where the file will be moved to

curl -X PATCH --upload-file /tmp/foo.json

Sample Output:

    "message": "File Moved Successfully"

Sample Upload File

  "path": "/path/to/file/file.1"

Extending Supported Backends

Create a backend directory

Under pacifica-archiveinterface->archiveinterface->archivebackends add a
directory for the new backend type

Create Classes that Implement the Abstract Backend Class methods

Abstract backend classses can ge found under:
Descriptions of all the methods that need to be abstracted exists in the
comments above the class.

Update Backend Factory

Update the archive backend factory found here:
In this file is a load_backend_archive method. This method needs to have
its logic extended to support the new backend type. This also entails loading the appropriate files for this backend using import

Update Interface Server

Update the file to support the new backend choice.
File located: pacifica-archiveinterface->scripts->
In this file the type argument is defined with its supported types. Need to
extend that array to include the new backend type

Post Deployment Testing

Inside the post_deployment_test directory there is a file called
This file will run a series of tests against a deployed archive interface. The test are ordered so that they post, stage, status, and get files properly.
There are a few global variables at the top of the file that need to be adjusted to each deployment.

Variables to set in

export ARCHIVE_URL=''
  • ARCHIVE_URL is the URL to the newly deployed archive_interface


pytest -v post_deployment_tests/

Output will be the status of the tests against the archive interface

Docker Pull Command