Python¶

Generic plotting tools in Python Design discussion notes.

Visualization Schema Validator

We have developed functionality to validate HDF5 data against the formally specified visualization schema VizSchema. The validator is written in Python and is available from the txutils svn repository. The file txutils/vsh5plot/VizSchemaH5DataValidator.py contains the validator source code. The visualization schema is already embedded in it making it a self-contained tool, apart from its dependence on Python, the Tables (for reading HDF5 data) and the lxml (for parsing XML and XML Schema) Python libraries.

We do a lot of development using bilder and end up with Python installed outside of the system installations. That is why one needs often to clean the paths as follows:

$ export PATH=/bin:./:/usr/bin
$ export PYTHONPATH=

The validator works by using the formally specified visualization schema (in XML Schema syntax) and reading an HDF5 file on input. On output, it produces validation information for each data node in the HDF5 file's data tree. The complete usage information for the validator can be obtained by running:

$ python VizSchemaH5DataValidator.py -h

which produces the output:

Usage: python VizSchemaH5DataValidator.py [options] fileInHDF5Format

Options:
  -h, --help            show this help message and exit
  -f FILE, --file=FILE  write the validation report to FILE

If the -f FILE or the equivalent --file=FILE option is omitted, the output from the validator will be directed to standard out.

Example use on one of the HDF5 files from the vstests repository with the validator output saved in the file synergia_001.vsh5.vs:

python VizSchemaH5DataValidator.py -f synergia_001.vsh5.vs ~/txw/vstests/tests/synergia/synergia_001.vsh5

The output from the validation of this HDF5 data file is:

>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> Starting to validate h5 file: /Users/dad/txw/vstests/tests/synergia/synergia_001.vsh5
>> Using root VizSchema element with name -> vizSchema
>> 
>> Non VS h5 node: / with HDF attributes:
>> {}
>> 
>> Valid VS h5 node: /particles of vizSchema type: vsVariableWithMesh with validated attribs:
>> {'vsSpatialIndices': [array([0, 2, 4])], 'Name': ['/particles'], 'vsIndexOrder': ['compMajorC'], 'vsType': ['variableWithMesh']}
>>   This node does not have dependencies.
>> HDF5 node - all user attributes:
>> {'vsSpatialIndices': array([0, 2, 4]), 'vsIndexOrder': 'compMajorC', 'vsType': 'variableWithMesh'}
>> VS node attrib #: 4
>> H5 node attrib #: 3
>>  
>> # non VizSchema h5 data nodes found = 1
>> # invalid VizSchema h5 data nodes found = 0
>> # valid VizSchema h5 data nodes found = 1
>> 
>> Finished validating h5 file: /Users/dad/txw/vstests/tests/synergia/synergia_001.vsh5
>> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

The validator separates the HDF5 data nodes into three types: non VS (visualization schema), invalid VS, and valid VS nodes.

Any HDF5 data node that does not have any attributes starting with the 'vs' string is considered a non VS node. In the example validator output, the root HDF5 node (with path "/") is a non VS node. The validator also outputs the attributes of each HDF5 data node (as a python dictionary data structure). The root node in the example does not have any attributes as indicated by the "{}" output.

Any HDF5 data node that has at least one attribute starting the 'vs' string is an invalid VS node if it could not validate against any of the formally specified VizSchema elements. The (small) example above does not have invalid VS nodes.

For any HDF5 data node that validates against the VizSchema, the validator outputs a Python dictionary data structure with the validated attributes (represented as key in the dictionary) and their values. The path of the HDF5 node in the HDF5 file's data tree is used by the validator as the value for the 'Name' attribute. The validator wraps each valid HDF5 data node in a separate object of type ValState (defined in VizSchemaH5DataValidator.py). These ValState objects can then be used for further processing. The validator also reports the VizSchema type of each of the validated HDF5 nodes. In the example above, the valid HDF5 node is of type vsVariableWithMesh. If the VizSchema specifies any dependencies (even on non or invalid VS nodes), they are traced and their state of satisfaction is reported as well. The valid VS node in the example output is reported to have no dependencies specified for it in the VizSchema. For comparison with the validated attributes, the validator also output all of the attributes of the HDF5 node and their values (together with the total number of ValState and HDF5 node attributes.

Finally, the validator produces a summary for the total number of non, invalid, and valid VizSchema nodes.

For developers: the validator VizSchemaH5DataValidator.py provides control over the produced output using objects of its

class VsSchemaVerbose

and a set of verbosity levels. When an object of this type is created a verbosity level is provided during initialization. For example:

vsvprint = VsSchemaVerbose(2, vout)

where 2 is the provided verbosity level and vout is set either to stdout or to a file depending where a user of the validator has chosen to redirect the output. Then, any use of this object will produce output only if the provided verbosity is equal to or lower than the verbosity level given during initialization. For the example initialization shown, the following call

vsvprint(5, ('XML Schema namespace abbreviation used: %s' % self.xs))

will not produce output while the call

vsvprint(2, ("  Dependency for attribute %s is not satisfied" % dak))

will generate output.