Changelog

List of changes in-between dclab releases.

version 0.24.1

  • enh: improve polygon filter speed by roughly two orders of magnitude with a cython version taken from scikit-image; there are only minor differences to the old implementation (top right point included vs. lower left point included), so this is not a breaking change (#23)

version 0.24.0

  • data: refurbished LUT for linear elastic spheres provided by Dominic Mokbel and Lucas Wittwer (based on the FEM simulation results from https://doi.org/10.6084/m9.figshare.12155064.v2); compared to the old LUT, there is a relative error in Young’s modulus below 0.1 %, which should not cause any breaking changes
  • data: updated isoelasticity lines (better spacing): analytical data was made available by Christoph Herold, numerical data was interpolated from the new LUT
  • data: added ‘scripts/extract_lut_and_iso.py’ for extracting Young’s modulus LUT and isoelastics from FEM simulation data provided by Lucas Wittwer; this is now the default method for extracting new LUTs and isoelastics
  • data: added ‘scripts/fem2rtdc.py’ for generating in-silico .rtdc datasets from FEM simulation data provided by Lucas Wittwer
  • fix: dclab-verify-dataset failed when the “logs” group was not present in HDF5 files
  • fix: use predefined chunks when writing HDF5 data to avoid exploding file sizes when writing one event at a time
  • fix: create a deep copy of the metadata dictionary when writing HDF5 data because it leaked to subsequent calls
  • ref: changed the way isoelasticity lines and and emodulus LUTs are stored and loaded (e.g. json metadata and a few more sanity checks)

version 0.23.0

  • feat: enable emodulus extrapolation for area_um/deform values outside of the given LUT.

version 0.22.7

  • enh: dclab-verify-dataset now also checks whether the sheath and sample flow rates add up to the channel flow rate
  • ref: Configuration does not anymore load unknown meta data keyword arguments, but ignores them. This implies that dclab-verify-dataset will not anymore check them actively. Instead, any warning issued when opening a file is added to the list of cues.
  • setup: bump nptdms to 0.23.0

version 0.22.6

  • fix: data export to HDF5 did not work when the “frame rate” is not given in the configuration

version 0.22.5

  • enh: add checks for valid keys in the Configuration dictionary of a dataset RTDCBase().config; unknown keys will issue an UnknownConfigurationKeyWarning (#58)
  • ref: moved rtdc_dataset.fmt_hdf5.UnknownKeyWarning to rtdc_dataset.config.UnknownConfigurationKeyWarning
  • ref: renamed rtdc_dataset.config.CaseInsensitiveDict to rtdc_dataset.config.ConfigurationDict and added option to check new keys

version 0.22.4

  • fix: disable computation of Young’s modulus for reservoir measurements (#75)
  • enh: new keyword argument req_func for AncillaryFeature to define additional logic for checking whether a feature is available for a given instance of RTDCBase.

version 0.22.3

  • enh: add data property to ICues (and use it when checking for compression)

version 0.22.2

  • fix: when computing the contour from the mask image, always use the longest contour - critical when the mask image contains artefacts
  • fix: minor issue with dclab-verify-dataset when nptdms was not installed and an exception occured
  • enh: dclab-verify-dataset shows some info on data compression

version 0.22.1

  • enh: remember working API Key
  • docs: document DCOR format

version 0.22.0

  • feat: implement DCOR client
  • enh: improved .rtdc file format detection (with wrong extension)

version 0.21.2

  • enh: dclab-verify-dataset now also checks HDF5 “mask” feature attributes
  • setup: bump h5py to 2.10.0 (need <object>.attrs.get_id)

version 0.21.1

  • fix: correct type of HDF5 image attributes for “mask” feature

version 0.21.0

  • feat: implement new CLI dclab-repack
  • fix: don’t write “logs” group to HDF5 files if there aren’t any
  • fix: support HDF5 files that have no “logs” group
  • docs: fix docstring of dclab-join

version 0.20.8

  • fix: regression where old .tmds data could not be opened if they did not contain the “area_msd” feature
  • fix: convert bytes logs to string in fmt_hdf5
  • enh: support len(ds.logs) for fmt_hdf5
  • enh: replace “info” by “build” in CLI job info

version 0.20.7

  • fix: ensure file extension is .rtdc in dclab-join
  • fix: correct “frame” and “index_online” features when exporting to hdf5
  • enh: allow to set metadata dictionary in dclab.cli.join

version 0.20.6

  • fix: typo in contour check resulted in small tolerance

version 0.20.5

  • fix: be more trustful when it comes to contour data in the tdms file format; instead of raising errors, issue warnings (#72)

version 0.20.4

  • ref: move integrity checks to new class check.IntegrityChecker
  • docs: document remaining dictionaries in dclab.dfn

version 0.20.3

  • docs: fix bad anchors

version 0.20.2

  • ref: using temperature values outside the range for viscosity computation now issues a warning instead of raising an error; warnings were added for the CellCarrier buffers
  • fix: handle number detection correctly in get_emodulus

version 0.20.1

  • fix: always return an array when computing the KDE
  • ref: make accessible static function RTDCBase.get_kde_spacing

version 0.20.0

  • feat: compute elastic modulus from “temp” feature (#51)
  • enh: computing isoelastics from datasets can use [setup]: “temperature” to compute the viscosity/emodulus (#51)
  • enh: define new meta data key [setup]: “temperature”
  • docs: add an advanced section on Young’s modulus computation (#51)

version 0.19.1

  • fix: hierarchy children did not pass force argument to hierarchy parent when apply_filter is called
  • fix: revert histogram2d “density” argument to “normed” to support numpy 1.10 (Shape-Out 1)
  • fix: implement unambiguous RTDCBase.__repr__

version 0.19.0

  • feat: added better contour spacing computation based on percentiles (dclab.kde_methods.bin_width_percentile)
  • feat: add feature “index_online” which may be missing values (#71)
  • feat: implement __getstate__ and __setstate__ for polygon filters
  • fix: write UTF-8 BOM when exporting to .tsv
  • enh: add check whether unique_id exists in PolygonFilter

version 0.18.0

  • fix: correctly handle filtering when features are removed from a dataset
  • ref: move dclab.rtdc_dataset.util to dclab.util
  • ref: minor cleanup in computation of viscosity (support lower-case medum values, add dclab.features.emodulus_viscosity.KNOWN_MEDIA)
  • ref: cleanup dclab.rtdc_dataset.filter (use logical operators, correctly display nan-warning messages, keep track of polygon filters, add consistency checks, improve readability)

version 0.17.1

  • maintenance release

version 0.17.0

  • feat: add command line script for compressing HDF5 (.rtdc) data “dclab-compress”
  • enh: record warnings under “/log” for all command line scripts
  • enh: set gzip data compression for all command line scripts

version 0.16.1

  • fix: circumvent UnicodeDecodeErrors which occured in frozen macOS binaries that use dclab
  • enh: support subsecond accuracy in the the configuration key [experiment] time (e.g. “HH:MM:SS.SSS” instead of “HH:MM:SS”)
  • enh: store the correct, relative measurement time in dclab-join (#63)

version 0.16.0

  • fix: RTDCBase.downsample_scatter with ret_mask=True did not return boolean array of len(RTDCBase) as indicated in the docs
  • ref: remove RTDCBase._plot_filter, which was confusing anyway
  • ref: deprecate usage of RTDCBase._filter

version 0.15.0

  • feat: add method RTDCBase.reset_filter
  • feat: implement RTDCBase.features_loaded
  • feat: allow to instantiate RTDC_Hierarchy without initially applying the filter
  • fix: non-scalar columns of RTDC_Hierarchy did not implement len()
  • docs: add an example script dedicated to data plotting
  • ref: remove circular references between Filter and RTDCBase

version 0.14.8

  • fix: Ignore feature “trace” when the trace folder exists but is empty (HDF5 format)
  • fix: If no contour can be found, raise an error before other ancillary features produce cryptic errors

version 0.14.7

  • enh: allow to add meta data when exporting to .fcs or .tsv (dclab version is saved by default)
  • setup: bump fcswrite from 0.4.1 to 0.5.0

version 0.14.6

  • fix: improved handling of tdms trace data (split trace with fixed samples per event to avoid ValueError when exporting to hdf5)
  • fix: transposed roi size x/y config value when exporting to hdf5

version 0.14.5

  • cli: write warning messages to logs in tdms2rtdc
  • ref: increase verbosity of warning messages

version 0.14.4

  • fix: discard trace data when “samples per event” has multiple values for tdms data
  • fix: prefer image shape over config keywords when determining the shape of the event mask and check the shape in dclab-verify-dataset
  • fix: avoid ContourIndexingError by also searching the neighboring (+2/-2) events when the contour frame number does not match (#67)

version 0.14.3

  • enh: explicitly check contour data when testing whether to include the first event in tdms2rtdc

version 0.14.2

  • ref: convert said ValueError to ContourIndexingError

version 0.14.1

  • fix: ValueError when verifying contour frame index due to comparison of float with int

version 0.14.0

  • feat: new command line script for creating a scalar-feature-only dataset with all available ancillary features “dclab-condense”
  • enh: enable scalar feature compression for hdf5 export
  • docs: fix doc string for dclab-tdms2rtdc (–include-initial-empty-image falsely shown as “enabled by default”)

version 0.13.0

  • feat: allow to obtain a mask representing the filtered data with the ret_mask kwarg in RTDCBase.get_downsampled_scatter
  • feat: allow to force-exclude invalid (inf/nan) events when downsampling using the remove_invalid keyword argument
  • feat: exclude empty initial images in dclab-tdms2rtdc; they may optionally be included with “–include-initial-empty-image”
  • feat: new property RTDCBase.features_innate (measuerd feature)
  • enh: log which ancillary features were computed in dclab-tdms2rtdc (#65)
  • enh: improved tdms meta data import (also affects dclab-tdms2rtdc)
  • enh: update channel count and samples per event when writing hdf5 data
  • enh: dclab-verify-dataset now recognizes invalid tdms data
  • enh: several other improvements when reading tdms data
  • enh: group meta data in log files (dclab-tdms2rtdc and dclab-join)
  • fix: correctly handle hdf5 export when the image or contour columns have incorrect sizes (affects dclab-tdms2rtdc)
  • fix: ignore empty configuration values when loading tdms data
  • fix: image/contour files were searched recursively instead of only in the directory of the tdms file
  • fix: check for precence of “time” feature before using it to correct measurement date and time
  • fix: ancillary feature computation for brightness had wrong dependency coded (contour instead of mask)
  • fix: ancillary feature computation when contour data is involved lead to error, because LazyContourList did not implement identifier (see #61)
  • ref: remove NoContourDataWarning for tdms file format
  • tests: improve dataset checks (#64)

version 0.12.0

  • feat: add command line script for joining measurements “dclab-join” (#57)
  • feat: make log files available as RTDCBase.logs
  • feat: include log data in “dclab-join” and “dclab-tdms2rtdc”
  • fix: features property for tdms file format falsely contains the keys “contour”, “image”, “mask”, and “trace” when they are actually not available.
  • enh: support loading TDMS data using the ‘with’ statement
  • docs: add example for joining measurements
  • docs: other minor improvements
  • setup: add Python 3.7 wheels for Windows (#62)
  • setup: remove Python 2 wheels for macOS

version 0.11.1

  • docs: add example for fluorescence trace visualization
  • docs: restructure advanced usage section
  • ref: make dclab in principle compatible with imageio>=2.5.0; Dependencies are pinned due to segfaults during testing
  • setup: make tdms format support and data export dependency optional; To get the previous behavior, use pip install dclab[all]

version 0.11.0

  • feat: compute contours lazily (#61)

version 0.10.5

  • setup: migrate to PEP 517 (pyproject.toml)

version 0.10.4

  • enh: ignore defective feature “aspect” from Shape-In 2.0.6 and 2.0.7
  • enh: support loading HDF5 data using the ‘with’ statement (e.g. with dclab.new_dataset(rtdc_path) as ds:)

version 0.10.3

  • fix: add numpy build dependency (setup_requires)

version 0.10.2

  • fix: HDF5-export did not re-enumerate “index” feature

version 0.10.1

  • fix: support nan-valued events when computing quantile levels in submodule kde_contours

version 0.10.0

  • BREAKING CHANGE: Change np.meshgrid indexing in RTDCBase.get_kde_contour from “xy” to “ij”
  • feat: new submodule kde_contours for computing kernel density contour lines at specific data percentiles (#60)
  • fix: range for contour KDE computation did not always contain end value (RTDCBase.get_kde_contour)
  • fix: positions keyword argument in RTDCBase.get_kde_scatter was not correctly scaled in the logarithmic case
  • ref: cleanup and document PolygonFilter.point_in_poly
  • ref: move skimage code to separate submodule “external”
  • ref: drop dependency on statsmodels and move relevant code to submodule “external”

version 0.9.1

  • fix: all-zero features were treated as non-existent due to relic from pre-0.3.3 era
  • fix: correct extraction of start time from tdms format (1h offset from local time and measurement duration offset)
  • fix: correct extraction of module composition from tdms format (replace “+” with “,”)
  • enh: add configuration key mapping for tdms format to simplify conversion to hdf5 format (see fmt_tdms.naming)
  • enh: do not add laser info for unused lasers for tdms format
  • enh: dclab-verify-dataset checks for image attribute dtype
  • enh: include original software version when exporting to rtdc format

version 0.9.0

  • feat: add new feature: gravitational force, temperature, and ambient temperature
  • ref: remove unused has_key function in rtdc_dataset.config.CaseInsensitiveDict
  • setup: require numpy>=1.10.0 because of equal_nan argument in allclose

version 0.8.0

  • fix: usage of “xor” (^) instead of “or” (|) in statistics
  • feat: support remove_invalid=False in downsampling.downsample_rand (#27)
  • feat: add keyword arguments xscale and yscale to improve data visualization in RTDCBase.get_downsampled_scatter, RTDCBase.get_kde_contour, and RTDCBase.get_kde_scatter (#55)
  • enh: make downsampling code more transparent
  • BREAKING CHANGE: low-level downsampling methods refactored
    • downsampling.downsample_grid: removed keyword argument remove_invalid, because setting it to False makes no sense in this context
    • downsampling.downsample_rand: changed default value of remove_invalid to False, because this is more objective
    • rename keyword argument retidx to ret_idx
    • these changes do not affect any other higher level functionalities in dclab.rtdc_dataset or in Shape-Out

version 0.7.0

  • feat: add new ancillary feature: principal inertia ratio (#46)
  • feat: add new ancillary feature: absolute tilt (#53)
  • feat: add computation of viscosity for water (#52)

version 0.6.3

  • fix: channel width not correctly identified for old tdms files

version 0.6.2

  • ci: automate release to PyPI with appveyor and travis-ci

version 0.6.0

  • fix: image export as .avi did not have option to use unfiltered data
  • fix: avoid a few unicode gotchas
  • feat: use Doane’s formula for kernel density estimator defaults (#42)
  • docs: usage examples, advanced scripting, and code reference update (#49)

version 0.5.2

  • Migrate from os.path to pathlib (#50)
  • fmt_hdf5: Add run index to title

version 0.5.1

  • Setup: add dependencies for statsmodels
  • Tests: filter known warnings
  • fmt_hdf5: import unknown keys such that “dclab-verify-dataset” can complain about them

version 0.5.0

  • BREAKING CHANGES:
    • definitions.feature_names now contains non-scalar features (inlcuding “image”, “contour”, “mask”, and “trace”). To test for scalar features, use definitions.scalar_feature_names.
    • features bright_* are computed from mask instead of from contour
  • Bugfixes:
    • write correct event count in exported hdf5 data files
    • improve implementation of video file handling in fmt_tdms
  • add new non-scalar feature “mask” (#48)
  • removed configuration key [online_contour]: “bin margin” (#47)
  • minor improvements for the tdms file format

version 0.4.0

  • Bugfix: CLI “dclab-tdms2rtdc” did not work for single tdms files (#45)
  • update configuration keys:
    • added new keys for [fluorescence]
    • added [setup]: “identifier”
    • removed [imaging]: “exposure time”, “flash current”
    • removed [setup]: “temperature”, “viscosity”
  • renamed feature “ncells” to “nevents”

version 0.3.3

  • ref: do not import missing features as zeros in fmt_tdms
  • CLI:
    • add tdms-to-rtdc converter “dclab-tdms2rtdc” (#36)
    • improve “dclab-verify-dataset” user experience
  • Bugfixes:
    • “limit events” filtering must be integer not boolean (#41)
    • Support opening tdms files with capitalized “userDef” column names
    • OSError when trying to open files from repository root

version 0.3.2

  • CLI: add rudimentary dataset checker “dclab-verify-dataset” (#37)
  • Add logic to compute parent/root/child event indices of RTDC_Hierarchy
    • Hierarchy children now support contour, image, and traces
    • Hierarchy children now support and remember manual filters (#22)
  • Update emodulus look-up table with larger values for deformation
  • Implement pixel size correction for emodulus computation
  • Allow to add pixelation error to isoelastics (add_px_err=True) (#28)
  • Bugfixes:
    • Pixel size not read from tdms-based measurements
    • Young’s modulus computation wrong due to faulty FEM simulations (#39)

version 0.3.1

  • Remove all-zero dummy columns from dict format
  • Implement hdf5-based RT-DC data reader (#32)
  • Implement hdf5-based RT-DC data writer (#33)
  • Bugfixes:
    • Automatically fix inverted box filters
    • RTDC_TDMS trace data contained empty arrays when no trace data was present (trace key should not have been accessible)
    • Not possible to get isoelastics for circularity

version 0.3.0

  • New fluorescence crosstalk correction feature recipe (#35)
  • New ancillary features “fl1_max_ctc”, “fl2_max_ctc”, “fl3_max_ctc” (#35)
  • Add priority for multiple ancillary features with same name
  • Bugfixes:
    • Configuration key values were not hashed for ancillary features
  • Code cleanup:
    • Refactoring: Put ancillary columns into a new folder module
    • Refactoring: Use the term “feature” consistently
    • Unify trace handling in dclab (#30)
    • Add functions to convert input config data

version 0.2.9

  • Bugfixes:
    • Regression when loading configuration strings containing quotes
    • Parameters missing when loading ShapeIn 2.0.1 tdms data

version 0.2.8

  • Refactor configuration class to support new format (#26)

version 0.2.7

  • New submodule and classes for managing isoelastics
  • New ancillary columns “inert_ratio_raw” and “inert_ratio_cvx”
  • Bugfixes:
    • Typo when finding contour data files (tdms file format)
  • Rrefactoring:
    • “features” submodule with basic methods for ancillary columns

version 0.2.6

  • Return event images as gray scale (#17)
  • Bugfixes:
    • Shrink ancillary column size if it exceeds dataset size
    • Generate random RTDCBase.identifier (do not use RTDCBase.hash) to fix problem with identical identifiers for hierarchy children
    • Correctly determine contour data files (tdms file format)
    • Allow contour data indices larger than uint8

version 0.2.5

  • Add ancillary columns “bright_avg” and “bright_sd” (#18, #19)
  • Standardize attributes of RTDCBase subclasses (#12)
  • Refactoring:
    • New column names and removal of redundant column identifiers (#16)
    • Minor improvements towards PEP8 (e.g. #15)
    • New class for handling filters (#13)
  • Bugfixes:
    • Hierarchy child computed all ancillary columns of parent upon checking availability of a column

version 0.2.4

  • Replace OpenCV with imageio
  • Add (ancilliary) computation of volume (#11)
  • Add convenience methods for Configuration
  • Refactoring (#8):
    • Separate classes for .tdms, dict-based, and hierarchy datasets
    • Introduce “_events” attribute for stored data
    • Data columns (including image, trace, contour) are accessed via keys instead of attributes.
    • Make space for new hdf5-based file format
    • Introduce ancilliary columns that are computed on-the-fly (new “_ancilliaries” attribute and “ancilliary_columns.py”)

version 0.2.3

  • Add look-up table for elastic modulus (#7)
  • Add filtering option “remove invalid events” to remove nan/inf
  • Support nan and inf in data analysis
  • Improve downsampling performance
  • Refactor downsampling methods (#6)

version 0.2.2

  • Add new histogram-based kernel density estimator (#2)
  • Refactoring:
    • Configuration fully handled by RTDC_DataSet module (#5)
    • Simplify video export function (#4)
    • Removed “Plotting” configuration key
    • Removed .cfg configuration files

version 0.2.1

  • Support npTDMS 0.9.0
  • Add AVI-Export function
  • Add lazy submodule for event trace data and rename RTDC_DataSet.traces to RTDC_DataSet.trace
  • Add “Event index” column

version 0.2.0

  • Compute sensible default configuration parameters for KDE estimation and contour plotting
  • Speed-up handling of contour text files
  • Add support for “User Defined” column in tdms files

version 0.1.9

  • Implement hierarchical instantiation of RTDC_DataSet
  • Bugfix: Prevent instances of PolygonFilter that have same id
  • Load InertiaRatio and InertiaRatioRaw from tdms files

version 0.1.8

  • Allow to instantiate RTDC_DataSet without a tdms file
  • Add statistics submodule
  • Bugfixes:
    • Faulty hashing strategy in RTDC_DataSet.GetDownSampledScatter
  • Code cleanup (renamed methods, cleaned structure)
  • Corrections/additions in definitions (fRT-DC)

version 0.1.7

  • Added channel: distance between to first fl. peaks
  • Added fluorescence channels: peak position, peak area, number of peaks
  • Allow to disable KDE computation
  • Add filter array for manual (user-defined) filtering
  • Add config parameters for log axis scaling
  • Add channels: bounding box x- and y-size
  • Bugfixes:
    • cached.py did not handle None
    • Limiting number of events caused integer/bool error

version 0.1.6

  • Added RTDC_DataSet.ExportTSV for data export
  • Bugfixes:
    • Correct determination of video file in RTDCDataSet
    • Fix multivariate KDE computation
    • Contour accuracy for Defo overridden by that of Circ

version 0.1.5

version 0.1.4

  • Added support for 3-channel fluorescence data (FL-1..3 max/width)

version 0.1.2

  • Add support for limiting amount of data points analyzed with the configuration keyword “Limit Events”
  • Comments refer to “events” instead of “points” from now on