DC data I/O

When working with DC data, you will inevitably run into the situation where you would like to write some part of dataset (be it an .rtdc file or data on DCOR) to a new file. Depending on the situation, one or more of the following subsection will probably cover what you need.

Exporting data

The RTDCBase class has the attribute RTDCBase.export which allows to export event data to several data file formats. See Export for more information.

In [1]: import dclab

In [2]: ds = dclab.new_dataset("data/example.rtdc")

# Restrict the deformation to 0.15
In [3]: ds.config["filtering"]["deform min"] = 0

In [4]: ds.config["filtering"]["deform max"] = .15

In [5]: ds.apply_filter()

In [6]: print("Deformation mean before filtering:", ds["deform"][:].mean())
Deformation mean before filtering: 0.0287258

In [7]: print("Deformation mean after filtering:", ds["deform"][ds.filter.all].mean())
Deformation mean after filtering: 0.026480934

# Export to .tsv
In [8]: ds.export.tsv(path="export_example.tsv",
   ...:               features=["area_um", "deform"],
   ...:               filtered=True,
   ...:               override=True)
   ...: 

# Export to .rtdc
In [9]: ds.export.hdf5(path="export_example.rtdc",
   ...:                features=["area_um", "aspect", "deform"],
   ...:                filtered=True,
   ...:                override=True)
   ...: 

Note that data exported as HDF5 files can be loaded with dclab (reproducing the previously computed statistics - without filters).

In [10]: ds2 = dclab.new_dataset("export_example.rtdc")

In [11]: ds2["deform"][:].mean()
Out[11]: np.float32(0.026480934)

Writing to an .rtdc file

If you would like to create your own .rtdc files, you can make use of the RTDCWriter class.

In [12]: with dclab.RTDCWriter("my-data.rtdc", mode="reset") as hw:
   ....:     hw.store_metadata({"experiment": {"sample": "my sample",
   ....:                                       "run index": 1}})
   ....:     hw.store_feature("deform", np.random.rand(100))
   ....:     hw.store_feature("area_um", np.random.rand(100))
   ....: 

In [13]: ds_custom = dclab.new_dataset("my-data.rtdc")

In [14]: print(ds_custom.features)
['area_um', 'deform', 'index']

In [15]: print(ds_custom.config["experiment"])
{'event count': 100, 'run index': 1, 'sample': 'my sample'}

The mode argument defines how data should be written to the file. In the above example, any existing data in my-data-rtdc is deleted. If you are writing to an existing file, you may use mode=”append” to append events to existing features or mode=”replace” to replace entire features with new data. Use “append” if you are continuously writing new events to a file. Use “replace” if you are rewriting feature data (e.g. in a script that computes features for all events).

Warning

If you are using the wrong mode, you can introduce inconsistencies. Say you are computing custom feature data and you changed the algorithm that does the computation. You have an existing .rtdc file with the outdated userdef1 feature. You would like to replace this data with the data from the updated algorithm. If you open that file with mode=”append” and write userdef1 to it, then the userdef1 feature will have twice the length. You should used mode=”replace” instead.

Copying (parts of) a dataset

In some situations, you would only like to copy an entire feature column from one dataset to a new file without modification. The copier submodule enables this on a low-level.

Use the rtdc_copy() method to create a compressed version of a DC dataset opened as an HDF5 file (RTDC_HDF5 or RTDC_S3).
Use the h5ds_copy() method to copy parts of an HDF5 dataset to another HDF5 file, with the option to enforce compression (if the source h5py.Dataset is not compressed properly already).