Sign inSign up

datproject/public-archives

By datproject

•Updated about 9 years ago

public archive demo page

Image
0

10K+

datproject/public-archives repository overview

⁠Public Data Archives

WIP

⁠Goals
  • Highlight large archives hosted over Dat
  • Provide hosting for public data archives and other data of interest
  • Allow sponsors to have pages with logo + featured datasets
  • Page of services we can offer for data hosting, archiving, etc.

⁠Setup

Clone this repository, then run

npm install
cp config.defaults.yml config.development.yml

Modify config.development.yml to fit your needs, then start the server with npm start.

⁠Configuration

Each page can be organized into a collection via hypercore-archiver. Information about each collection is organized via the changed feed. Stats are generated and kept in a dat with toiletdb files.

port: 8080
db: ./db                # archive where stats are stored via toiletdb
archives: ./.archiver   # hypercore-archiver databases
feeds:
  featured:             # name of dataset collection
    key: '<key>'        # hypercore-archiver changes feed
    sparse: true        # whether to store the data
    dir: featured       # subdirectory under archives
  cdl:
    key: '<key>'
    sparse: true
    dir: cdl

⁠Adding A Collection Page

Each collection runs by tailing a hypercore-archiver feed. Tables can be created for each collection by following these steps:

  • Adding the changes feed to the config.yml file with the hypercore-archiver changes feed. Each feed has a name that can be used to access the stats database.
  • Add a page to the router:
app.get('/demo', function (req, res) {
  // dataset name from config
  dbs.demo.read(function (err, data) {
    if (err) return console.error(err)
    res.render('page', {
      page: './pages/demo', // template page in ./views
      title: 'Demo Page - Dat Project',
      archives: {
        // Info for displaying the dataset table
        name: 'Demo Datasets',
        description: `
          Description of the demo thing and what not.
        `,
        stats: data,
      }
    })
  })
})

⁠License

MIT⁠

Tag summary

Content type

Image

Digest

Size

304.1 MB

Last updated

about 9 years ago

docker pull datproject/public-archives