In today�s information age, data plays a pivotal role in many domains. VIFI offers a more effective solution to the traditional data fabric approach which will significantly reduce the time to conduct data analytics by:
Providing a standard workflow with templates that can be customized
Eases the time taken by data scientists to search, query and access data.
VIFI provides a unique and different perspective as it addresses a significant gap that remains un-addressed by existing projects.
Does not require data to be physically moved to a common location
Provides a framework for analytics at the data source.
Provides management for both public and non-public data with or without metadata available.
Supports authorization control for data and metadata access, and execution of models for end-users without exposing private datasets.
VIFI offers a more effective solution to the traditional data fabric approach. VIFI will significantly reduce the time to conduct data analytics by providing a standard workflow with templates that can be customized, and eases the time taken by data scientists to search, query and access data.
Each VIFI-node should have a "requests" folder that contains different directories for each use case (e.g., JPL directory for the earth science use case). Under each use case directory, the following directories exist:
in: This directory keeps incoming requests from VIFI users to be processed by current use case workflow.
finished: Upon successful completion of current VIFI request, results are moved to this directory. Each VIFI request has a separate folder under this "finished" directory.
failed: Upon failure in processing current VIFI request, results are moved to a separate directory in this folder for further investigation.
log: Keeps logging information of each VIFI request status (e.g., arrival, processing, success and failure)
"/user/test": Test ViFi RestAPI is working correctly. Response include a dictionary of {'vifi_restapi':'ok'}
"/user/snapshot": (Work-in-Porgress) Returns a snapshot of a specific request within a specific workflow domain.
"/user/getallresults": To get all compressed result files (timestamped or not) in the output directory (i.e., finished directory of the workflow). Form data includes "workflow" (workflow domain containing the request), "request_name" (the request within the workflow domain whose compressed results are to be retrieved. Generally, request_name and task ID should be the same), and "exclist" (List of files to be excluded from retrived files. This can be useful to avoid retrieving the same files again)
NOTE: to save the compressed results from "/user/getallresults", use a similar code to:
r=requests.get(<required paramters for /user/getallresults>)
with open('<file_name>.zip','wb') as f:
for i in r.iter_content():
f.write(i)
"/user/startrequest": To send a new request to be executed under a specific workflow domain. Form data includes "workflow" (workflow domain to receive the new request), Form files include "file" (the zipped request).
"/user/send": To send a file to a request under a specific workflow domain. Form data includes "workflow" (workflow domain containing the request), form files include "file" (the zipped request). "file" has the same name of the required request.
"/user/stop": To stop a specific request within a specific workflow domain. Form data includes "workflow" (workflow domain containing the request), and "request_name" (the request to be stopped within the speicifed workflow domain).
"/user/resume": To resume a previously stopped request within a specifi workflow domain. Form data includes "workflow" (workflow domain containing the request), and "request_name" (the request to be resumed within the speicifed workflow domain).
"/user/modify": (Work-in-Progress) To modify an existing request within a specified workflow domain.
"/user/finish": To consider a specific request within a speicific workflow domain as successfully finished. Form data includes "workflow" (workflow domain containing the request), and "request_name" (the request to be successfully finished within the speicifed workflow domain).
"/user/fail": To consider a specific request within a speicific workflow domain as failed. Form data includes "workflow" (workflow domain containing the request), and "request_name" (the request to be failed within the speicifed workflow domain).
VIFI follows a phased approach during devlopement. Thus, the more use cases we have, the more we learn to improve VIFI and identificy commoen requirements between different cases. Thus, we appreciate any use case that can benifit from VIFI (e.g., distributed big data nalaytics).