orngServerFiles: Orange's file repository

Module orngServerFiles allows users to download files from a common repository residing on the Orange server. It was designed to simplify the download and updates of external data sources for Orange Genomics add-on. Furthermore, an authenticated user can also manage the repository files with this module.

Orange server file repository was created with the intend to store larger files that do not come with Orange installation, but may be required from the user when running specific Orange functions. A typical example is Orange bioinformatics package, which relies on large data files storing genome information. These do not come pre-installed, but are rather downloaded from the server when needed and stored in the local repository. Module orngServerFiles provides low-level functionality to manage these files, and is used by Orange modules to locate the files from the local repository and update/download them when and if needed.

Each file managed by orngServerFiles is described by domain and the file name. Domains are like directories - a place where files are put in.

Local file management

Downloaded files are saved under Orange's settings directory, subdirectory buffer/bigfiles. For each new domain a subdirectory containing the downloaded files is created. With every download a corresponding info file is created with bearing the same name but added an extension plus ".info". Info fail contains file's title, tags, size and date and time of the upload.

Methods used for managing local files

download(domain, filename, serverfiles=None, callback=None, extract=True, verbose=True)
Downloads file from the repository to local orange installation. To download files as an authenticated user you should also pass an instance of ServerFiles class. Callback can be a function without arguments. It will be called once for each downloaded percent of file: 100 times for the whole file.
listdomains()
List all file domains in the local repository.
listfiles(domain)
List all files from a domain in a local repository.
info(domain, filename)
Returns a dictionary containing downloaded file information. Keys: title, tags, size, datetime.
allinfo(domain)
Goes through all files in a domain on a local repository and returns a dictionary, where keys are names of the files and values are their information.
search(substrings, caseSensitive=False, inTag=True, inTitle=True, inName=True)
Function searches for files in the local repository where all substrings in a list are contained in at least one chosen field (tag, title, name). It returns a list of tuples: first tuple element is the domain of the file, second its name.
localpath(domain [,filename])
Returns a path for the domain in the local repository. If argument filename is given, returns a path to corresponding file.
localpath_download(domain, filename, **kwargs)
Returns local path for the given domain and file. If file does not exist on the local machine, download it. Additional arguments are passed to the download function.
remove_domain(domain [,force])
Removes a domain. If force is True, domain is removed even if it is not empty (contains files).
remove(domain, filename)
Removes a file from repository.
update(domain, filename, access_code=None, verbose=True)
Downloads the corresponding file from the server and places it in the local repository, but only if the server copy of the file is newer or the local copy does not exist. An optional access_code can be given to allow server access if the file is protected.

ServerFiles: the repository

To work with the repository, you need to create an instance of ServerFiles object. To access the repository as an authenticated user, a username and password should be passed to the constructor. All password protected operations and transfers are secured by SSL; this secures both password and content.

Repository files are set as protected when first uploaded: only authenticated users can see them. They need to be unprotected for public use.

ServerFiles(username=None, password=None, access_code=None)
Creates a ServerFiles instance. Pass your username and password to use the repository as an authenticated user. If you want to use your access code (as an non-authenticated user), pass it also.

Non-authenticated users

download(domain, filename, target, callback=None)
Downloads file from the repository to a given target name. Callback can be a function without arguments. It will be called once for each downloaded percent of file: 100 times for the whole file.
listdomains()
List all domains on repository.
listfiles(domain)
List all files in a repository domain.
info(domain, filename)
Returns a dictionary containing repository file info. Keys: title, tags, size, datetime.
allinfo(domain)
Goes through all accessible files in a given domain and returns a dictionary, where key is file's name and value its info.
search(substrings, caseSensitive=False, inTag=True, inTitle=True, inName=True)
Function searches for files on the repository where all substrings in a list are contained in at least one choosen field (tag, title, name). It returns a list of tuples: first tuple element is the file's domain, second its name. As for now the search is performed locally, therefore information on files in repository is transfered on first call of this function.
downloadFH(domain, filename)
Returns a file handle to the file that we would like to download.

Authenticated users

Authenticated users can use methods described in "Non-authenticated users" section, but they can also use the functions that access and modify protected files.
create_domain(domain)
Creates a repository domain.
remove_domain(domain [,force])
Removes a domain. If force is True, domain is removed even if it is not empty (contains files).
remove(domain, filename)
Removes a file from the repository.
protection(domain, filename)
Returns file protection. Legend: "0" - public use, "1" - for authenticated users only, anything else represents a specific access code.
protect(domain, filename [, access_code])
Hide file from non-authenticated users. If an access code (string) is passed, the file will be available to authenticated users and non-authenticated users with that access code.
unprotect(domain, filename)
Put a file into public use.
upload(domain, filename, localfile [, title, tags])
Uploads a file "localfile" to the domain where it is saved with filename "filename". If file does not exist yet, set it as protected. Parameter localfile can be a file handle open for reading or a file name.

Examples

Downloading and listing files

Listing local files, files from the repository and downloading all available files from domain "demo".

import orngServerFiles repository = orngServerFiles.ServerFiles() print "My files", orngServerFiles.listfiles('demo') print "Repository files", repository.listfiles('demo') print "Downloading all files in domain 'test'" for file in repository.listfiles('demo'): print "Datetime for", file, repository.info('demo', file)["datetime"] orngServerFiles.download('demo', file) print "My files after download", orngServerFiles.listfiles('demo') print "My domains", orngServerFiles.listdomains()

Output for first run (and current repository state):

My files [] Repository files ['orngServerFiles.py', 'urllib2_file.py'] Downloading all files in domain 'test' Datetime for orngServerFiles.py 2008-08-20 12:25:54.624000 Datetime for urllib2_file.py 2008-08-20 12:25:54.827000 My files after download ['urllib2_file.py', 'orngServerFiles.py'] My domains ['demo']

Creating a domain, uploading files

A similar domain as "demo" in previous example can be built as follows. import orngServerFiles ordinary = orngServerFiles.ServerFiles() authenticated = orngServerFiles.ServerFiles(username, password) try: authenticated.remove_domain('demo2', force=True) except: pass authenticated.create_domain('demo2') authenticated.upload('demo2', 'orngServerFiles.py', 'orngServerFiles.py', \ title="orngServerFiles client", tags=["basic", "fileManagement"]) authenticated.upload('demo2', 'urllib2_file.py', 'urllib2_file.py') print "Non-authenticated users see:", ordinary.listfiles('demo2') print "Authenticated users see:", authenticated.listfiles('demo2') authenticated.unprotect('demo2', 'orngServerFiles.py') authenticated.unprotect('demo2', 'urllib2_file.py') print "Non-authenticated users now see:", ordinary.listfiles('demo2') print "orngServerFiles.py file info:" import pprint; pprint.pprint(ordinary.info('demo2', 'orngServerFiles.py'))

Output:

Ordinary users see: [''] Authenticated users see: ['orngServerFiles.py', 'urllib2_file.py'] Non-authenticated users now see: ['orngServerFiles.py', 'urllib2_file.py'] orngServerFiles.py file info: {'datetime': '2008-08-26 10:30:05.373000', 'size': '10165', 'tags': ['basic', 'fileManagement'], 'title': 'orngServerFiles client'}