Stoner.DataFolder

class Stoner.DataFolder(*args, **kargs)[source]

Provide an interface to manipulating lots of data files stored within a directory structure on disc.

By default, the members of the DataFolder are isntances of Stoner.Data. The DataFolder emplys a lazy open strategy, so that files are only read in from disc when actually needed.

Inheritance diagram of DataFolder

Methods

__init__(*args, **kargs)

add_group(key)

Add a new group to the current baseFolder with the given key.

all()

Iterate over all the files in the Folder and all it's sub Folders recursely.

append(value)

Append an item to the folder object.

clear()

Clear the subgroups.

compress([base, key, keep_terminal])

Compresses all empty groups from the root up until the first non-empty group is located.

concatenate([sort, reverse])

Concatentates all the files in a objectFolder into a single metadataObject like object.

count(name)

Provide a count method like a sequence.

extend(values)

S.extend(iterable) -- extend sequence by appending elements from the iterable

extract(*metadata, **kargs)

Extract metadata from each of the files in the terminal group.

fetch([futures])

Preload the contents of the DiskBasedFolderMixin.

file(name, value[, create, pathsplit])

recursely add groups in order to put the named value into a virtual tree of baseFolder.

filter([filter, invert, copy, recurse, prune])

Filter the current set of files by some criterion.

filterout(filter[, copy, recurse, prune])

Synonym for self.filter(filter,invert=True).

flatten([depth])

Compresses all the groups and sub-groups iunto a single flat file list.

gather([xcol, ycol])

Collect xy and y columns from the subfiles in the final group in the tree.

get(name[, default])

Return either a sub-group or named object from this folder.

getlist(**kargs)

Scan the current directory, optionally recursively to build a list of filenames.

group(key)

Sort Files into a series of objectFolders according to the value of the key.

index(name[, start, end])

Provide an index method like a sequence.

insert(ix, value)

Implement the insert method with the option to append as well.

items()

Return the key,value pairs for the subbroups of this folder.

keep_latest()

Filter out earlier revisions of files with the same name.

keys()

Return the keys used to access the sub-=groups of this folder.

make_name([value])

Construct a name from the value object if possible.

on_load_process(tmp)

Carry out processing on a newly loaded file to set means and extra metadata.

pop([name, default])

Return and remove either a subgroup or named object from this folder.

popitem()

Return the most recent subgroup from this folder.

prune([name])

Remove any empty groups from the objectFolder (and subgroups).

remove(value)

S.remove(value) -- remove first occurrence of value.

reverse()

S.reverse() -- reverse IN PLACE

save([root])

Save the entire data folder out to disc using the groups as a directory tree.

select(*args, **kargs)

Select a subset of the objects in the folder based on flexible search criteria on the metadata.

setdefault(k[, d])

Return or set a subgroup or named object.

slice_metadata(key[, output])

Return an array of the metadata values for each item/file in the top level group.

sort([key, reverse, recurse])

Sort the files by some key.

unflatten()

Take the file list an unflattens them according to the file paths.

unload([name])

Remove the instance from memory without losing the name in the Folder.

update(other)

Update this folder with a dictionary or another folder.

values()

Return the sub-groups of this folder.

walk_groups(walker, **kargs)

Walk through a heirarchy of groups and calls walker for each file.

zip_groups(groups)

Return a list of tuples of metadataObjects drawn from the specified groups.

Attributes

basenames

Return a list of just the filename parts of the objectFolder.

clone

Clone just does a deepcopy as a property for compatibility with Stoner.Core.DataFile.

debug

Just read the local debug value.

defaults

Build a single list of all of our defaults by iterating over the __mro__, caching the result.

depth

Give the maximum number of levels of group below the current objectFolder.

directory

Just alias directory to root now.

each

Return a Stoner.folders.each.item proxy object.

files

Return an iterator of potentially unloaded named objects.

groups

Subfolders are held in an ordered dictionary of groups.

instance

Return a default instance of the type of object in the folder.

is_empty

Return True if the folder is empty.

key

Override the parent class key to use the directory attribute.

layout

Return a tuple that describes the number of files and groups in the folder.

loaded

Iterate only over those members of the folder in memory.

loader

Return a callable that will load the files on demand.

ls

List just the names of the objects in the folder.

lsgrp

Return a list of the groups as a generator.

metadata

Return a Stoner.folders.metadata.MetadataProxy object.

mindepth

Give the minimum number of levels of group below the current objectFolder.

not_empty

Iterate over the objectFolder that checks whether the loaded metadataObject objects have any data.

not_loaded

Return an array of True/False for whether we've loaded a metadataObject yet.

objects

Return the objects in the folder are stored in a regexpDict.

pattern

Provide support for getting the pattern attribute.

root

Return the real folder root.

setas

Return the proxy for the setas attribute for each object in the folder.

shape

Return a data structure that is characteristic of the objectFolder's shape.

trunkdepth

Return the number of levels of group before a group with files is found.

type

Return the (sub)class of the Stoner.Core.metadataObject instances.