6. API reference

Content below is automatically produced from docstrings in the sources, and needs more work...

6.1. Core

pdcalc	A best-effort attempt to build computation dependency-graphs from method with dict-like objects (such as pandas),
datamodel
processor	The core calculations required for transforming the Input-datamodel to the Output one.

6.2. ExcelRunner

FuefitExcelRunner

Sample xlwings script

6.3. Tests

cmdline_test

Check cmdline parsing and model building.

6.4. Module: fuefit.datamodel

fuefit.datamodel.base_model()

The base model for running a WLTC experiment.

It contains some default values for the experiment (ie the default ‘full-load-curve’ for the vehicles). But note that it this model is not valid - you need to override its attributes.

:return :json_tree: with the default values for the experiment.

fuefit.datamodel.jsonpointer_parts(jsonpointer)

Iterates over the jsonpointer parts.

Parameters:jsonpointer (str) – a jsonpointer to resolve within document Returns:a generator over the parts of the json-pointer Author:Julian Berman, ankostis

fuefit.datamodel.merge(a, b, path=, []list_merge_mode=<MergeMode.REPLACE: 1>, raise_struct_mismatches=False)

‘Merges b into a.

List merge modes: REPLACE, APPEND_HEAD, APPEND_TAIL, OVERLAP_HEAD, OVERLAP_TAIL

fuefit.datamodel.model_schema(additional_properties=False)

The json-schema for input/output of the fuefit experiment.

:return :dict:

fuefit.datamodel.resolve_jsonpointer(doc, jsonpointer, default=<object object at 0x7f59483905e0>)

Resolve a jsonpointer within the referenced doc.

Parameters:

• doc – the referrant document
• jsonpointer (str) – a jsonpointer to resolve within document

Returns:

the resolved doc-item or raises JsonPointerException

Author:

Julian Berman, ankostis

fuefit.datamodel.set_jsonpointer(doc, jsonpointer, value, object_factory=<class 'dict'>)

Resolve a jsonpointer within the referenced doc.

Parameters:

• doc – the referrant document
• jsonpointer (str) – a jsonpointer to the node to modify

Raises:

JsonPointerException (if jsonpointer empty, missing, invalid-contet)

6.5. Module: fuefit.processor

The core calculations required for transforming the Input-datamodel to the Output one.

Uses pandalon‘s automatic dependency extraction from calculation functions.

fuefit.processor._robust_residualfunc(coeffs, modelfunc, X, YData, is_robust=False, robust_prcntile=None)

A non-linear iteratively-reweighted least-squares (IRLS) residual function (objective-function) that robustly fits YData = modelfunc(X).

This method applies weights on each iteration so as to downscale any outliers and high-leverage data-points based on the ‘bisquare’ standardized adjusted residuals:[1]

rac{r}{K imes hat{sigma} imes sqrt{1 - h}}

where:

\(r\) (vector)

the residuals \(\hat{y} - y\)

\(K\) (scalar)

the robust percentile tuning constant used on each iteration to filter-out adjusted-standardized-weights above 1, expressed as the Bisquare M-estimator efficiency under Gaussian model.

\(\hat{\sigma}\) (scalar)

the robust estimate of the standard deviation of the residuals based on MAD[#]_ like this: \(\hat{\sigma}=1.4826 imes\operatorname{MAD}\)

\(h\) : (vector)

the hat vector, the diagonal of the hat matrix,[#]_ which is used to reduce the weight of high-leverage data points that are having a large effect on the least-squares fit.

param modelfunc:  The modeling function that accepts the dict of coeffs param nparray X:   param nparray YData:  measured-data points param boolean is_robust:  Whether to deleverage outlier YData. param float robust_prcntile:  The K percentile of the MAD, [default: 4.68, filters-out approximately 5% of the residuals as outliers]

See also

curve_fit, leastsq

[1]	http://www.mathworks.com/help/stats/robustfit.html

[2]	https://en.wikipedia.org/wiki/Median_absolute_deviation

[3]	https://en.wikipedia.org/wiki/Hat_matrix

fuefit.processor.eng_points_2_std_map(params, engine, eng_points)

A factory of the calculation functions for reaching to the data necessary for the Fitting.

The order of the functions below not important, and the actual order of execution is calculated from their dependencies, based on the data-frame’s column access.

fuefit.processor.engine_map_modelfunc(coeff_values, X)

The function that models the engine-map.

fuefit.processor.run(mdl, opts=None)

Parameters:

• mdl – the datamodel to process, all params and data
• opts (map) – flags controlling non-functional aspects of the process (ie error-handling and logging, gui, etc)

6.6. Module: fuefit.pdcalc

A best-effort attempt to build computation dependency-graphs from method with dict-like objects (such as pandas), inspired by XForms: http://lib.tkk.fi/Diss/2007/isbn9789512285662/article3.pdf

See also

Dependencies @calculation @calculation_factory [TODO] execute_funcs_map()

class fuefit.pdcalc.Dependencies

Bases: builtins.object

Discovers and stores the rough functions-relationships needed to produce ExecutionPlanner

The relation-tuples are “rough” in the sense that they may contain duplicates etc requiring cleanup by _consolidate_relations().

Usage:

Use the harvest_{XXX}() methods or the add_func_rel() method to gather dependencies from functions and function-facts and then use the build_plan() method to freeze them into a plan.

add_func_rel(item, deps=None, func=None)

Parameters:

• item (str) – a dotted.var
• deps – a string or a sequence of dotted.vars
• func – a standalone func, or a funcs_factory as a 2-tuple (func, index)

build_plan(sources, dests)

Builds an execution-plan that when executed with execute_plan() it will produce dests from sources.

Turns any stored-relations into a the full dependencies-graph then trims it only to those ‘’dotted.varname’’ nodes reaching from sources to dests.

Parameters:

• dests (sequence) – a list of ‘’dotted.varname’’s that will be the outcome of the calculations
• sources – a list of ‘’dotted.varname’’s (existent or not) that are assumed to exist when the execution wil start

Returns:

the new plan that can be fed to execute_plan()

Return type:

pd.Series

classmethod from_funcs_map(funcs_map, deps=None)

Factory method for building Dependencies by harvesting multiple functions and func_factories, at once

Parameters:

• funcs_map –

  a mapping or a sequence of pairs (what --> bool_or_null) with values:

  True

  when what is a funcs_factory, a function returning a sequence of functions processing the data

  False

  when what is a standalone_function, or

  None

  when what is an explicit relation 3-tuple (item, deps, func) to be fed directly to Dependencies.add_func_rel(), where:

  item

  a ‘’dotted.varname’’ string,

  deps

  a string or a sequence of ‘’dotted.varname’’ strings, and

  func

  a standalone func or a funcs_factory as a 2-tuple (func, index)

  Both items and deps are dot-separated sequence of varnames, such as ‘foo.bar’

• deps – if none, a new Dependencies instance is created to harvest relations

Returns:

a new (even inherited) or an updated Dependencies instance

Important

All functions must accept exactly the same args, or else the results will be undetermined and it will most-likely scream on execute_funcs_map.

class fuefit.pdcalc._DepFunc(func, is_funcs_factory=False, _child_index=None)

Bases: builtins.object

A wrapper for functions explored for relations, optionally allowing them to form a hierarchy of factories and produced functions.

It can be in 3 types:

• 0, standalone function: args given to function invocation are used immediatelt,
• 10, functions-factory: args are stored and will be used by the child-funcs returned,
• 20, child-func: created internally, and no args given when invoked, will use parent-factory’s args.

Use factory methods to create one of the first 2 types:

• pdcalc._wrap_standalone_func()
• pdcalc._wrap_funcs_factory()

fuefit.pdcalc._consolidate_relations(relations)

(item1, deps, func), (item1, ...) –> {item1, (set(deps), set(funcs))}

fuefit.pdcalc._filter_common_prefixes(deps)

deps: not-empty set

example::

deps = [‘a’, ‘a.b’, ‘b.cc’, ‘a.d’, ‘b’, ‘ac’, ‘a.c’] res = _filter_common_prefixes(deps) assert res == [‘a.b’, ‘a.c’, ‘a.d’, ‘ac’, ‘b.cc’]

fuefit.pdcalc._find_missing_input(calc_inp_nodes, graph)

Search for tentatively missing data.

fuefit.pdcalc._harvest_indexing(index)

Harvest any strings, slices, etc, assuming to be DF’s indices.

fuefit.pdcalc._harvest_mock_call(mock_call, func, deps_set, func_rels)

Adds a 2-tuple (indep, [deps]) into indeps with all deps collected so far when it visits a __setartr__.

fuefit.pdcalc._research_calculation_routes(graph, sources, dests)

Find nodes reaching ‘dests’ but not ‘sources’.

sources: a list of nodes (existent or not) to search for all paths originating from them dests: a list of nodes to search for all paths leading to them them return: a 2-tuple with the graph and its nodes topologically-ordered

fuefit.pdcalc._strip_magic_tail(path)

some.path___with_.__magics__ –> some.path

fuefit.pdcalc.calculation(deps=None)

A decorator that extracts dependencies from a function.

Parameters:deps – if None, any ₂ extracted deps are added in the calculation_deps module-variable.

Example:

@calculation
def foo(a, r):
    a['bc'] = a.aa
    r['r1'] = a.bb

@calculation
def bar(a, r):
    r['r2'] = a.bc

plan    = calculation_deps.build_plan(['a.ab', 'a.bc', 'r'], ['r.r2'])
planner.execute_funcs_map(plan, 1, 2)

fuefit.pdcalc.default_arg_paths_extractor(arg_name, arg, paths)

Dig recursively objects for indices to build the sources/dests sequences for Dependencies.build_plan()

It extracts indices recursively from maps and pandas The inner-ones paths are not added, ie df.some.key, but not df or df.some.

Note

for pandas-series their index (their infos-axis) gets appended only the 1st time (not if invoked in recursion, from DataFrame columns).

Parameters:paths (list) – where to add the extracted paths into

fuefit.pdcalc.execute_funcs_factory(funcs_fact, dests, *args, **kwargs)

A one-off way to run calculations from a funcions-factory (see execute_funcs_map())

fuefit.pdcalc.execute_funcs_map(funcs_map, dests, *args, **kwargs)

A one-off way to run calculations from a funcs_map as defined in Dependencies:from_funcs_map().

Parameters:

• funcs_map – a dictionary similar to the one used by Dependencies.from_funcs_map()
• dests (seq) –

  what is required as the final outcome of the execution, ie: for the func-functory:

  def some_funcs(foo, bar):
      def func_1():
          foo['a1'] = ...
      ...
      def func_n(): ...
          foo['a2'] = ...
          bar['b1'] = ...
  
      return [func_1, ..., func_n]
  

  the dests may be:

  ['foo.a1', 'foo.a2', 'bar.b1']
  

• args – the actual args to use for invoking the functions in the map, and the names of these args would come rom the first function to be invoked (which ever that might be!).

Note

Do not use it to run the calculations repeatedly. Preffer to cache the function-relations into an intermediate Dependencies instance, instead.

fuefit.pdcalc.tell_paths_from_named_args(named_args, arg_paths_extractor_func=<function default_arg_paths_extractor at 0x7f5942e9b378>, paths=None)

Builds the sources or the dests sequences params of Dependencies.build_plan() from a map of function-arguments

Parameters:named_args – an ordered map {param_name --> param_value} similar to that returned by inspect.signature(func).bind(*args).arguments: BoundArguments. Use the utility name_all_func_args() to generate such a map for some function. Returns:the paths updated with all ‘’dotted.vars’’ found

6.7. Module: fuefit.excel.FuefitExcelRunner

6.7.1. Sample xlwings script

The functions included provide for running a batch of experiments in an excel-table with json-pointer paths as headers (see accompanying .xlsm).

You can debug it by running it directly as a python script, as suggested by :

http://docs.xlwings.org/debugging.html

<< EDIT THIS SCRIPT TO PUT YOUR EXCEL/XLWINGS PYTHON-CODE, BELOW >>

fuefit.excel.FuefitExcelRunner.build_models(vehs_df)

Builds all input-dataframes as Experiment classes and returns them in a list of (veh_id, exp) pairs.

Parameters:vehs_df – A dataframe indexed by veh_id, and with columns json-pointer paths into the model Returns:a list of (veh_id, model) tuples

fuefit.excel.FuefitExcelRunner.read_input_as_df(table_ref_str)

Expects excel-table with jsonpointer paths as Header and 1st column named id as index, like this:

id     vehicle/test_mass  vehicle/p_rated           vehicle/gear_ratios 
veh_1               1500              100  [120.75, 75, 50, 43, 37, 32] 
veh_2               1600               80  [120.75, 75, 50, 43, 37, 32] 
veh_3               1200               60  [120.75, 75, 50, 43, 37, 32] 

fuefit.excel.FuefitExcelRunner.resolve_excel_ref(ref_str, default=<object object at 0x7f5948390620>)

if ref_str is an excel-ref, it returns the referred-contents as DataFrame or a scalar, None otherwise.

Excel-ref examples:

@a1
@E5.column
@some sheet_name!R1C5.TABLE
@1!a1:c5.table(header=False)
@3!a1:C5.horizontal(strict=True; atleast_2d=True)
@sheet-1!A1.table(asarray=True){columns=['a','b']}
@any%sheet^&name!A1:A6.vertical(header=True)        ## NOTE: Setting Range's `header` kw and 
                                                    #      DataFrame will parse 1st row as header

The excel-ref syntax is case-insensitive apart from the key-value pairs, which it is given in BNF-notation:

excel_ref   ::=  "@" 
                 [sheet "!"] 
                 cells 
                 ["." shape] 
                 ["(" range_kws ")"] 
                 ["{" df_kws "}"]
sheet       ::=  sheet_name | sheet_index
sheet_name  ::=  <any character>
sheet_index ::=  integer
cells       ::=  cell_ref [":" cell_ref]
cell_ref    ::=  A1_ref | RC_ref | tuple_ref
A1_ref      ::=  <ie: "A1" or "BY34">
RC_ref      ::=  <ie: "R1C1" or "R24C34">
tuple_ref   ::=  <ie: "(1,1)" or "(24,1)", the 1st is the row>
shape       ::=  "." ("table" | "vertical" | "horizontal")
range_kws   ::=  kv_pairs                    # keywords for xlwings.Range(**kws)
df_kws      ::=  kv_pairs                    # keywords for pandas.DataFrafe(**kws)
kv_pairs    ::=  <python code for **keywords ie: "a=3.14, f = 'gg'">

Note that the “RC-notation” is not converted, so Excel may not support it (unless overridden in its options).

fuefit.excel.FuefitExcelRunner.run_experiments(experiment_pairs)

Iterates veh_df and runs experiment_pairs.

Parameters:vehs_df – A dataframe indexed by veh_id, and with columns json-pointer paths into the model

6.8. Module: fuefit.test.cmdline_test

Check cmdline parsing and model building.