dataprocessor.pipes package

Submodules

dataprocessor.pipes.add_conf module

dataprocessor.pipes.add_conf.add_conf(node_list, path, key, value)

Add a new configure to the specified node.

Parameters:

path : str

Specify the node.

key : str

key of configure.

value : str

value of configure.

dataprocessor.pipes.add_conf.register(pipes_dics)

dataprocessor.pipes.add_node module

dataprocessor.pipes.add_node.add_node(node_list, path='.', node_type='run', children=[], name=None, parents=['..'], strategy='raise')

Add node.

A node is added to the node_list.

Parameters:

path : str, optional

path of node (default=”.”)

node_type : str, optional {“run”, “project”}

type of node (default=”run”)

name : str, optional

children : list, optional

parents : list, optional

strategy : str, optional

determines how to deal with conflict, i.e. there alread exists another node whose path is same as new one. See docstring of nodes.add.

Returns:

node_list

dataprocessor.pipes.add_node.register(pipes_dics)

dataprocessor.pipes.add_run module

dataprocessor.pipes.add_run.add_run(node_list, path, tag=None, name=None, comment='')

Add run node to node_list.

Parameters:

path : str

Path of node.

tag : str, optional

Project name or project path. See “add_tag”.

name : str, optional

Name of node. (default=basename of path)

comment : str, optional

Comment of node. (default=””)

dataprocessor.pipes.add_run.register(pipe_dics)

dataprocessor.pipes.add_tag module

dataprocessor.pipes.add_tag.add_tag(node_list, node_path, project_id)

Make the node belong to the project specified by project id.

We realize “tagging” feature by project nodes.

Parameters:

node_path : str

This path specify the unique node.

project_id: str

the name or path of project. The path is resolved by resolve_project_path.

dataprocessor.pipes.add_tag.register(pipe_dics)

dataprocessor.pipes.addcomment module

dataprocessor.pipes.addcomment.add(node_list, comment, node_path)

Add comment to node spedcified path.

Parameters:

comment : str

comment.

node_path : str

This path specify the unique node.

Examples

>>> node_list = [{"path": "/path/to/hoge"},
...              {"path": "/path/to/hogehoge"}]
>>> add(node_list, "some comments", "/path/to/hoge") == [
...     {"path": "/path/to/hoge", "comment": "some comments"},
...     {"path": "/path/to/hogehoge"}]
True
>>> add(node_list, "some comments aho", "/path/to/hogehom")
Traceback (most recent call last):
    ...
DataProcessorError: 'There is no node with specified path: /path/to/hogehom'

dataprocessor.pipes.addcomment.register(pipe_dics)

dataprocessor.pipes.configure module

dataprocessor.pipes.configure.get_filetype(path)

Get filetype from path (filename extension).

Parameters:

path: str

path to a file

Returns:

filetype as a string.

dataprocessor.pipes.configure.load(node_list, *args, **kwds)

Load configure

Parameters:

filename : str

filename of parameter configure file If file does not exist, add null list.

section : str

Specify section name in configure file.

Examples

>>> load(node_list, "configure.conf") 
>>> # Change load section.
>>> load(node_list, "configure.conf", "defaults") 

dataprocessor.pipes.configure.no_section(node_list, *args, **kwds)

Load configure

Parameters:

filename : str

filename of parameter configure file If file does not exist, add null list.

split_char : str

Specify the deliminator char.

comment_char : str

Specify the comment line signal char.

Examples

>>> no_section(node_list, "foo.conf") 
>>> # Change deliminator and comment line signal
>>> no_section(node_list, "foo.conf", split_char=":", comment_char="!")
... 

dataprocessor.pipes.configure.parse_ini(confpath, section)

Parse .ini and .conf to dictionary

Parameters:

confpath : str

Path to config file.

section : str

Specify section name in configure file.

Returns:

Specified section as a dictionary.

dataprocessor.pipes.configure.parse_yaml(confpath, section)

Parse .yaml to dictionary

Parameters:

confpath : str

Path to config file.

section : str

Specify section name in configure file.

Returns:

Specified section as a dictionary.

dataprocessor.pipes.configure.register(pipes_dics)

dataprocessor.pipes.figure module

dataprocessor.pipes.figure.register(pipe_dics)

dataprocessor.pipes.filter module

dataprocessor.pipes.filter.register(pipes_dics)

dataprocessor.pipes.ipynb module

dataprocessor.pipes.ipynb.add(node_list, path, parents=[])

Add ipynb file manually

dataprocessor.pipes.ipynb.gather(node_list, pattern='*.ipynb')

Search ipynb file by its extention

dataprocessor.pipes.ipynb.register(pipes_dics)

dataprocessor.pipes.ipynb.start(nl, ipynb_path)

Start existing .ipynb file at standing ipython notebook server

Parameters:

ipynb_path : str

path of existing .ipynb file

dataprocessor.pipes.jsonio module

dataprocessor.pipes.jsonio.register(pipes_dics)

dataprocessor.pipes.merge_nodelist module

dataprocessor.pipes.merge_nodelist.merge_nodelist(node_list, json_filename)

Merge another node_list.

Parameters:

json_fn : str

filename of JSON file including another node_list

dataprocessor.pipes.merge_nodelist.register(pipes_dics)

dataprocessor.pipes.move_node module

dataprocessor.pipes.move_node.move_node(node_list, from_path, dest)

Move node like as UNIX mv.

Change path of a node and related nodes. Coresponding directories are also moved or renamed.

Parameters:

from_path : str

path of node.

dest : str

path of destination.

Raises:

DataProcessorError

raised when path of ‘dest’ already exists and the path is a file.

dataprocessor.pipes.move_node.register(pipe_dics)

dataprocessor.pipes.normalize module

dataprocessor.pipes.normalize.normalize(node_list)

dataprocessor.pipes.normalize.register(pipe_dics)

dataprocessor.pipes.scan module

Scan directories as nodes.

dataprocessor.pipes.scan.directory(node_list, root, whitelist, followlinks=False)

Scan nodes from all directories under the directory ‘root’.

If one directory has properties of both of ‘run’ and ‘project’, type of the directory is set to ‘run’.

Parameters:

root : str

Scan directories recursively under the directory root.

whitelist : list of str or str

Run node has one or more file or directory which satisfies run_node_dir/whitelist. And project nodes satisfy project_dir/run_node_dir/whitelist. str can be specified by wildcard.

followlinks : {‘False’, ‘True’}, optional

Whether scan in symbolic link. Be aware that setting this to True may lead to infinite recursion.

Returns:

node_list

Examples

>>> # Initialize node_list.
>>> node_list = directory([], "scandir_path", ["data/hoge*", "*foo*"])

>>> # Rescan node_list.
>>> node_list = [
...     {'path': '/tmp/scan_dir/run0',
...      'parents': [],   # empty
...      'children': [],  # empty
...      'name': 'run0',
...      'type': 'run'}]
>>> node_list = directory([], "scandir_path", ["*.conf"])

dataprocessor.pipes.scan.register(pipe_dics)

dataprocessor.pipes.show module

dataprocessor.pipes.show.register(pipe_dics)

dataprocessor.pipes.show.show_projects(node_list, show_format='{path}')

Show project list.

Parameters:

show_format : str, optional

specify format you want to output. You should use new format function format. This string will be formatted by show_format.format(**node).

Returns:

node_list

dataprocessor.pipes.show.show_runs(node_list, project=None, show_format='{path}', parameters=[])

Show run list.

Parameters:

project : str or [str], optional

the path of project, whose runs will be shown.

show_format: str, optional

specify format you want to output. You should use new format function format. This string will be formatted by show_format.format(**node).

parameters: [str], optional

parameters to be displayed.

Returns:

node_list

dataprocessor.pipes.untag module

dataprocessor.pipes.untag.register(pipe_dics)

dataprocessor.pipes.untag.untag(node_list, path, project_id)

Untag the node.

Parameters:

path : str

Path of node.

project_id : str

the name or path of project. The path is resolved by resolve_project_path.

Raises:

DataProcessorError

raised when path of node or path specified by project id is not registered.

Module contents

@pipes