Get project in dataframe format.
If there are two or more project of specified name, the latter one is selected.
| Parameters: | project_path : str
properties : list of str (optional)
index : str or list of str (optional)
|
|---|---|
| Returns: | project : pandas.DataFrame |
Bases: exceptions.Exception
A runtime error occurred in DataProcessor
This exception is raised when invalid manipulation is done. This exception will be caught in dataprocessor.execute, and converted into InvalidJSONError.
Attributes
| msg | (str) A message for the error |
check whether maniplation is valid
| Parameters: | manip : list
|
|---|---|
| Raises: | InvalidJSONError
|
Examples
>>> def do_check_manip(manip):
... try:
... check_manip(manip)
... except InvalidJSONError as e:
... print("[%s] %s" % (e.name, e.msg))
>>> do_check_manip([{"nae": "save_json", # typo: "nae"
... "args": ["out.json"],
... "kwds": {"silent" : "True"}}])
[] pipe name is not defined
>>> do_check_manip([{"name": "save_json",
... "arg": ["out.json"], # typo: "arg"
... "kwds": {"silent" : "True"}}])
[save_json] attributes of pipes must be in 'name', 'args', 'kwds'
>>> do_check_manip([{"name": "save_json",
... "args": ["out.json"],
... "kwd": {"silent" : "True"}}]) # typo: "kwd"
[save_json] attributes of pipes must be in 'name', 'args', 'kwds'
>>> do_check_manip([{"name": "sav_json", # typo: "sav_json"
... "args": ["out.json"],
... "kwds": {"silent" : "True"}}])
[sav_json] invalid pipe name
>>> do_check_manip([{"name": "save_json",
... "args": [], # argument mismatch
... "kwds": {"silent" : "True"}}])
[save_json] the number of arguments mismatches
>>> do_check_manip([{"name": "load_json",
... "args": ["in.json"],
... "kwds": {"silent": "True"}}])
... # `load_json` does not have "kwds"
[load_json] pipe does not have 'kwds'
>>> do_check_manip([{"name": "save_json",
... "args": ["out.json"],
... "kwds": {"silnt" : "True"}}])
... # typo in keywords: "silnt"
[save_json] keyword argument 'silnt' does not exist
execute pipeline defined in manip
This function does not check whether manip is valid. Please check by .check_manip
| Parameters: | manip : list
node_list : list, optional
|
|---|---|
| Returns: | node_list : list |
Handling node for figures.
Bases: dataprocessor.exception.DataProcessorError
Exception about figure.
Return the path of a directory where the figure will be copied.
This is also an ID of the figure in DataProcessor.
| Parameters: | figure_path : str
figure_directory : str
|
|---|
Copy figure and incidentals and return corresponding node.
| Parameters: | figure_path : str
generators : list
figure_directory : str
parents : list
children : list, optional
|
|---|---|
| Raises: | DataProcessorFigureError
|
Register figure into node_list.
| Parameters: | node_list : list
figures : list
figure_directory : str
runs : list
generators : list
|
|---|
Filter by node_type.
| Parameters: | ntype : str
|
|---|---|
| Returns: | node_list |
| Raises: | DataProcessorError
|
Filter by prefix path.
In the case when pre_path is “/foo/bar”, following paths remain (not filtered):
, and following do not remain (filtered out):
| Parameters: | pre_path : str
|
|---|---|
| Returns: | node_list |
Bases: object
Generator of Zsh Completion Function.
Since this class uses some inner function of argparse, this does not work well in some version of argparse. This is tested only on argparse of python 2.7.5.
Attributes
| command_name | (str) command name of completion |
| parser | (argparse.ArgumentParser) parser of command |
Methods
| get() | returns string of Zsh completion function |
Bases: object
A data handler.
A utility class for save/load data into/from JSON file. See the following example. This class use JSON format for serializing.
Examples
>>> with DataHandler(filename, silent=True) as dh:
... dh.add({"path" : "/path/to/data1", "name" : "data1",
... "parents": [], "children": []})
Attributes
| node_list | (list) the managed node_list |
Methods
| get() | returns managed node_list |
| add(node, skip_validate_link) | add node into managed node_list |
| replace(node_list, skip_validate_link) | swap managed node_list and node_list in the argument |
| serialize() | serialize node_list into file. If you use with statement, you need not to call this. |
Add node into managed node_list.
| Parameters: | skip_validate_link : bool, optional
|
|---|
Bases: dataprocessor.io.DataHandler
A synced data handler.
This lock reading/writing JSON file.
Methods
| add(node[, strategy, skip_validate_link]) | Add node into managed node_list. |
| get() | |
| load() | |
| replace(node_list[, skip_validate_link]) | Swap node_list. |
| serialize() | |
| update(node_list[, skip_validate_link]) | Update node_list (use nodes.modest_update). |
Tools for manipulation of node_list.
Bases: dataprocessor.exception.DataProcessorError
Exception about Nodes processings.
Add a node into node_list.
This adds a node into node_list, and validate links in node[“children”] and node[“parents”]. In detail of validate link, see validate_link.
| Parameters: | node_list : list
node : dict
skip_validate_link : bool, optional
strategy : str, optional {“raise”, “update”, “modest_update”, “replace”}
|
|---|---|
| Raises: | DataProcessorNodesError:
DataProcessorError:
|
Examples
>>> node_list = [{"path": "/some/path", "children": [], "parents": []}]
>>> added_node = {"path": "/added/path", "children": ["/some/path"],
... "parents": []}
>>> add(node_list, added_node)
If skip_validate_link=True, snode_list[0][“parents”] is not filled.
Change node path.
| Parameters: | from_path : str
to_path : str
silent : bool or { ‘False’, ‘True’ }, optional
|
|---|---|
| Returns: | node_list |
| Raises: | DataProcessorError
|
Check if duplicated nodes exist.
only check their paths
| Parameters: | node_list : list
|
|---|---|
| Returns: | list of str
|
Examples
>>> node_list = [
... {"path": "/path/0", "parents": ["/path/1"],
... "children": ["/path/2", "/path/3"]},
... {"path": "/path/1", "parents": [],
... "children": ["/path/0"]},
... {"path": "/path/2", "parents": ["/path/0"],
... "children": []},
... {"path": "/path/2", "parents": ["/path/0"],
... "children": []}, # duplicated
... {"path": "/path/3", "parents": ["/path/0"],
... "children": [], "attr1": "value1"},
... {"path": "/path/3", "parents": ["/path/0"],
... "children": [], "attr1": "value2"} # duplicated
... ]
>>> check_duplicate(node_list)
['/path/2', '/path/3']
Search node from node_list by its path.
| Returns: | node or None
|
|---|
Examples
>>> node_list = [{"path": "/path/1"}, {"path": "/path/2"},
... {"path": "/path/3"}, {"path": "/path/4"}]
>>> get(node_list, "/path/3")
{'path': '/path/3'}
>>> node = get(node_list, "/path/5") # no node exists
>>> print(node)
None
Merge duplicate node.
If duplicated nodes are found, they will be merged. The latter node has priority in merging (see Examples)
| Parameters: | node_list : list
|
|---|---|
| Returns: | node_list : list |
Examples
>>> node_list = [
... {"path": "/path/1", "attr1": "value1"},
... {"path": "/path/1", "attr1": "value2"},
... ]
>>> merge_duplicate(node_list)
[{'path': '/path/1', 'attr1': 'value2'}]
Update node properties modestly
This keeps values as possible, besides the “update” strategy of add(...) simply uses dict.update(). If node in the arguments has empty value e.g. node = {“path”: “/path/0”, “comment” : “”}, this function does not overwrite existing comment of the node in node_list.
| Parameters: | node_list : list
node : dict
skip_validate_link : bool, optional
|
|---|---|
| Raises: | DataProcessorNodesError
|
Examples
>>> node_list = [{
... "path": "/path/0",
... "comment": "some comment",
... "children": [],
... "parents": [],
... }]
>>> node = {
... "path":"/path/0",
... "configure": {"A": 1.0},
... "comment": "",
... }
>>> modest_update(node_list, node)
>>> node_list == [{
... 'comment': 'some comment',
... 'path': '/path/0',
... 'parents': [],
... 'children': [],
... 'configure': {'A': 1.0}
... }]
True
Normalize node (i.e. fill necessary field).
| Parameters: | node: dict
|
|---|---|
| Returns: | dict
|
Examples
>>> node = {
... "path": "/path/0",
... "type": "run",
... "unknown": ["homhom"]
... }
>>> node = normalize(node)
>>> node == {
... "path": "/path/0",
... "type": "run",
... "name": "",
... "configure": {},
... "comment": "",
... "parents": [],
... "children": [],
... "unknown": ["homhom"]
... }
True
Remove node from node_list.
In detail of validate link, see validate_link.
| Parameters: | node_list : list
path : str
skip_validate_link : bool, optional
|
|---|---|
| Raises: | DataProcessorError
|
Examples
>>> node_list = [{"path": "/remove/path", "children": [],
... "parents": ["/some/path"]},
... {"path": "/some/path", "children": ["/remove/path"],
... "parents": []}]
>>> remove(node_list, "/remove/path")
If skip_validate_link=True, node_list[1][“chilrdren”] is not removed.
Validate the link of the node.
Check node[“children”] and node[“parents”] is correct. If a link is incomplete, it will fixed (see the following example).
| Parameters: | node_list : list
node : dict
|
|---|
Examples
Fill node_list[0]’s parents and node_list[1]’s children
>>> node_list = [
... {"path": "/path/0", "parents": [], "children": []},
... {"path": "/path/1", "parents": [], "children": []},
... {"path": "/path/2", "parents": ["/path/1"],
... "children": ["/path/0"]}]
>>> validate_link(node_list, node_list[2])
>>> node_list == [
... {"path": "/path/0", "parents": ["/path/2"], "children": []},
... {"path": "/path/1", "parents": [], "children": ["/path/2"]},
... {"path": "/path/2", "parents": ["/path/1"],
... "children": ["/path/0"]}]
True
Remove node_list[1]’s children.
>>> node_list = [
... {"path": "/path/0", "parents": [], "children": ["/path/1"]},
... {"path": "/path/1", "parents": ["/path/0"],
... "children": ["/not/exist"]}]
>>> validate_link(node_list, node_list[1], silent=True)
>>> node_list == [
... {"path": "/path/0", "parents": [], "children": ["/path/1"]},
... {"path": "/path/1", "parents": ["/path/0"], "children": []}]
True
This module provides wrapper functions which make easy to define a SIMD-like pipe, i.e. operates independently on each nodes.
The following two codes are equal without error handling:
>>> @wrap
... def pipe1(node, arg1, kwd1=""):
... print(node)
... node["attr1"] = "val"
... return node
>>> def pipe1(node_list, arg1, kwd1=""):
... for node in node_list:
... print(node)
... node["attr1"] = "val"
... return node_list
Error handling policy is following:
Simply, corresponds to the following conceptual code:
>>> for node in node_list:
... try:
... decorated(node)
... except DataProcessorError:
... continue
Bases: exceptions.Exception
Raised when the implementation of pipe is invalid
Attributes
| name | (str) The name of pipe in which error occurred. |
| msg | (str) A message for the error |
Create a pipe which operates on directory nodes
Create a pipe which operates on file nodes
Create a pipe which operates on nodes of specific type.
Examples
>>> nl = [{
... "path": "/path/1",
... "type": "project",
... "children": ["/path/2"],
... "parents": [],
... }, {
... "path": "/path/2",
... "type": "run",
... "children": [],
... "parents": ["/path/1"],
... }]
>>> @type("run")
... def run_pipe(node):
... node["comment"] = "RUN"
... return node
>>> @type("project")
... def project_pipe(node):
... node["comment"] = "PROJECT"
... return node
>>> nl = run_pipe(nl)
>>> nl = project_pipe(nl)
>>> nl == [{
... "path": "/path/1",
... "type": "project",
... "children": ["/path/2"],
... "parents": [],
... "comment": "PROJECT",
... }, {
... "path": "/path/2",
... "type": "run",
... "children": [],
... "parents": ["/path/1"],
... "comment": "RUN",
... }]
True
works on project nodes only.
Create a pipe from a function operates on a node.
This decorator makes easy to define a SIMD-like pipe, i.e. operates independently on each nodes.
Examples
The following two codes are equal without error handling:
>>> @wrap
... def pipe1(node, arg1, kwd1=""):
... print(node)
... node["attr1"] = "val"
... return node
>>> def pipe1(node_list, arg1, kwd1=""):
... for node in node_list:
... print(node)
... node["attr1"] = "val"
... return node_list
Configure manager of dataprocessor
Argument parser for executables in this project.
| Parameters: | rcpath : str, optional
options : [{str: dict}], optional
|
|---|---|
| Returns: | argparse.ArgumentParser |
| Raises: | DataProcessorRcError
|
Bases: dataprocessor.exception.DataProcessorError
Exception about configure file.
Get configure parser
| Parameters: | rcpath : str, optional
|
|---|---|
| Returns: | ConfigParser.SafeConfigParser
|
| Raises: | DataProcessorRcError
|
Get configure value
| Parameters: | section : str
key : str
|
|---|---|
| Returns: | str
|
| Raises: | DataProcessorRcError
|
Get configure value safely.
| Parameters: | section : str
key : str
default : any
|
|---|---|
| Returns: | str
|
Load node_list from default data.json.
| Parameters: | rcpath : str, optional
|
|---|---|
| Returns: | node_list |
| Raises: | DataProcessorRcError
|
Load configure into argparse.
| Parameters: | parser : argparse.ArgumentParser
section_name : str
options : {str: {str: str}}
|
|---|
Resolve project path from its path or name.
| Parameters: | name_or_path : str
create_dir : boolean
root : str, optional
basket_name : str, optional
rcpath : str, optional
|
|---|---|
| Returns: | path : str
|
| Raises: | DataProcessorRcError
DataProcessorError
|
Save node_list into default data.json with update strategy.
| Parameters: | rcpath : str, optional
|
|---|---|
| Raises: | DataProcessorRcError
|
Create a part of HTML of table.
Bases: object
Create table widget.
Create table from dictionary in node. If node has dictionary in its values, also search dictionary recursively. (e.g. node[key1][key2]...[keyn])
| Parameters: | table_type : {‘children’, ‘parents’}, optional
groups : list of dic, optional
|
|---|---|
| Raises: | DataProcessorError
|
Examples
If you have following list and you call as follows,
>>> nodelist = [{'path': '/tmp', 'children': ['/tmp/run1', '/tmp/run0']},
... {'path': '/tmp/run0', 'name': 'run0', 'comment': u'testあ',
... 'configure': {'nx':1, 'ny':2}},
... {'path': '/tmp/run1', 'name': 'run1',
... 'configure': {'nx': 10, 'ny': 20}}]
>>> tble = Table('/tmp', nodelist,
... groups=[{'dict_path': ['configure']},
... {'items': ['comment', 'path'], 'name': 'node'}])
>>> html_str = tble.render()
you can get html string of following formatted table.
| configure | node | |||
|---|---|---|---|---|
| nx | ny | comment | path | |
| run0 | 1 | 2 | testあ | /tmp/run0 |
| run1 | 10 | 20 | /tmp/run1 | |
Another example.
>>> node_list = [{"path": "/path/to",
... "children": ["/path/to/1", "/path/to/2"]},
... {"path": "/path/to/1", "name": "n1", "val1": 1,
... "hoge1": {"hoge2": {"hoge3": {"val4": 4}}}},
... {"path": "/path/to/2", "name": "h1", "val2": 2,
... "hoge1": {"hoge2": {"hoge3": {"val3": 4}}}}]
>>> tbl = Table("/path/to", node_list,
... groups=[{"items": ["val1", "val2"], "name": "some name"},
... {"dict_path": ["hoge1", "hoge2", "hoge3"]}])
>>> html_str = tbl.render()
| some name | hoge1/hoge2/hoge3 | |||
|---|---|---|---|---|
| val1 | val2 | val3 | val4 | |
| h1 | n | 2 | 4 | |
| n1 | 1 | 4 | ||
Methods
| render() | Get a piece of html for table. |
Utility of dataprocessor.
Some useful tools for dataprocessor are included.
Make arg boolen value.
| Parameters: | arg : bool, str, int, float |
|---|---|
| Returns: | bool |
Examples
>>> boolenize(True)
True
>>> boolenize(False)
False
>>> boolenize(1)
True
>>> boolenize(0)
False
>>> boolenize(0.0)
False
>>> boolenize("True")
True
>>> boolenize("other words")
True
>>> boolenize("False")
False
>>> boolenize("falSE")
False
>>> boolenize("false")
False
>>> boolenize("F")
False
>>> boolenize("f")
False
>>> boolenize("No")
False
>>> boolenize("NO")
False
>>> boolenize("no")
False
>>> boolenize("N")
False
>>> boolenize("n")
False
Check whether directory exists.
| Returns: | str
|
|---|---|
| Raises: | DataProcessorError
|
Check whether file exists.
| Returns: | str
|
|---|---|
| Raises: | DataProcessorError
|
Copy a file.
If to_path already exist, check whether it is same file. When there are same file, skip. When there are different, replace the file or change file name.
Moreover, when destination directory does not exist, create the direcotry.
| Parameters: | from_path : str to_path : str strategy : str, optional, {“interactive”, “replace”, “skip”, “error”}
|
|---|---|
| Raises: | DataProcessorError
|
Get absolute path of the directory.
If it does not exist, it will be created.
| Parameters: | silent : bool, optional
|
|---|---|
| Returns: | str
|
| Raises: | DataProcessorError
|
Get absolute path.
| Returns: | str
|
|---|---|
| Raises: | DataProcessorError
|
Read configure file without sections.
| Parameters: | filename : str
split_char : str, optional
comment_char : list of str, optional
|
|---|---|
| Returns: | dict
|
dataprocessor