camonitor2table.py

a tool to convert outputs from camonitor to a table of values

Overview

Camonitor, when used with more than one process variable, prints each new value in a new line. Here is an example:

U3IV:AdiUn9PmsPosI             2023-01-25 14:22:04.022486 0
U3IV:AdiUn14PmsPosI            2023-01-25 14:22:04.022486 -3491
U3IV:AdiUn9PmsPosI             2023-01-25 14:22:15.022486 329
U3IV:AdiUn14PmsPosI            2023-01-25 14:22:15.022486 -4577
U3IV:AdiUn14PmsPosI            2023-01-25 14:22:15.122486 -7045
U3IV:AdiUn9PmsPosI             2023-01-25 14:22:15.122486 1111
U3IV:AdiUn9PmsPosI             2023-01-25 14:22:15.222486 1470
U3IV:AdiUn14PmsPosI            2023-01-25 14:22:15.222486 -8137
U3IV:AdiUn9PmsPosI             2023-01-25 14:22:15.322486 1459
U3IV:AdiUn14PmsPosI            2023-01-25 14:22:15.322486 -8121

Comparing several variables at the same time is sometimes made difficult by this. A table-like format would be better in this case. This program does just this, it creates a simple table from the output of a camonitor command that can for example be used for gnuplot. Here is an example of such a table:

Timestamp                  U3IV:AdiUn9PmsPosI U3IV:AdiUn14PmsPosI
2023-01-25 14:22:04.022486 0                  -3491
2023-01-25 14:22:15.022486 329                -4577
2023-01-25 14:22:15.122486 1111               -7045
2023-01-25 14:22:15.222486 1470               -8137
2023-01-25 14:22:15.322486 1459               -8121

Examples

Waveform support

Currently the program handles waveform records by adding a column for each element in the waveform record, here is an example, this line:

U3IV:DiagSpdSet                2023-02-20 16:44:58.859846 2 3.0 4.0

creates two columns, named "U3IV:DiagSpdSet_0" and "U3IV:DiagSpdSet_1" with the values 3.0 and 4.0. A Possible output of pretty_print() then loons like this:

Timestamp                  U3IV:DiagSpdSet_0 U3IV:DiagSpdSet_1
2023-02-20 16:44:58.859846 3.0               4.0

Reference of command line options

-h print a short help for all command line options.
--summary print a one-line summary of the scripts function
--doc create online help in restructured text. Use "./txtcleanup.py --doc | rst2html" to create html-help
-t, --test perform a simple self-test
--raw print the internal HashedList2D object, this is for debugging only.
--dump do not collect the data to a table but dump the data in camonitor format to the console. This may be useful if combined with some of the filter options or options that modify the timestamps or pv names.
-r, --rjust justify the values in each row to the right side. Note that the timestamps are always left justified except when the are converted to a floating point number (see --floattime).
-c, --columnformat FORMAT
 format columns with the given FORMAT. A FORMAT is a space separated list of format substrings that use the same conventions as C format strings. If only a single format is given, this is applied to all columns. If floattime is used, the same format is also applied to the timestamp field.
-s, --separator SEPARATOR
 specify the SEPARATOR that separates columns of the table. This string is also used to separate values when csv format is used (see --csv).
--csv produce a comma-separated list of values. Note that the separator character may be specified with the separator option (--separator).
--floattime STARTTIME
 convert timestamps floating point seconds when 0 corresponds to STARTTIME. If STARTTIME has the special value "FIRST", the first timestamp is taken as STARTTIME.
--from-time STARTTIME
 use only data where the timestamp is newer or equal to STARTTIME.
--to-time ENDTIME
 use only data where the timestamp is older or equal to ENDTIME.
--max-lines MAX
 stop after MAX lines have been fetched. This may be used for checking a command line with a very large file.
--filter-pv REGEXP
 select only PVs that match REGEXP.
--filter-complete
 select only rows where each column has a value.
--skip-flagged REGEXP
 Skip all lines where the flags match REGEXP, e.g. "UDF" skips all lines where the flags contain "UDF". If REGEXP has the value '.*', all lines with flags are removed.
--differentiate
 differentiate all values, that means that each value is replaced with the difference of this and the previous value for the same PV divided by the difference of the timestamp in seconds. The values must be numbers in order to be able to do this.
--fill fill empty places in the table with the first non-empty value in the same column from a row above.
--fill-interpolate
 fill empty places in the table with a linear interpolation taken from the rows above and below.
--add-seconds [SECONDS]
add the seconds given (a floating point value) to the timestamps.
--time-rebase [OLDTIME,NEWTIME]
Add an offset to all timestamps. The offset is calculated to ensure that OLDTIME is changed to NEWTIME.
-T --time-columnname COLUMNNAME
Set the name of the time column, The default "name for this column is 'Timestamp'.
--pvmap [PVMAP]
Defines a mapping that replaces a pv with a new name. A PVMAP is a string with the form 'OLDPV,NEWPV. You can specify more than one PVMAP.
--pvmaprx [REGEXP]
Apply a regular expression to each pv to modify it. The REGEXP should have the form '/match/replace/'. You can specify more than one REGEXP, in this case all are applied in the order you specify them. REGEXPs are applied after PVMAP changes (see above).
--progress show the progress of the program on stderr. 2 numbers are printed, the first is the current line in the data file, the second one is the number of fetched lines.
-f, --file FILE
 read the data from FILE. If this parameter is missing or if it is '-', read from stdin.