Skip to content

README.md

Latest commit

 

History

History
104 lines (78 loc) · 8.95 KB

README.md

File metadata and controls

104 lines (78 loc) · 8.95 KB

The goal of this project is to provide tools for working with large network traffic datasets and to facilitate research in the traffic classification area. The core functions of the cesnet-datazoo package are:

  • A common API for downloading, configuring, and loading of four public datasets of encrypted network traffic.
  • Extensive configuration options for:
    • Selection of train, validation, and test periods.
    • Selection of application classes and splitting classes between known and unknown.
    • Data transformations, such as feature scaling.
  • Built on suitable data structures for experiments with large datasets. There are several caching mechanisms to make repeated runs faster, for example, when searching for the best model configuration.
  • Datasets are offered in multiple sizes to give users an option to start the experiments at a smaller scale (also faster dataset download, disk space, etc.). The default is the S size containing 25 million samples.

🧠 🧠 See a related project CESNET Models providing pretrained neural networks for traffic classification. 🧠 🧠

📓 📓 Example Jupyter notebooks are included in a separate Traffic Classification Examples repository. 📓 📓

🚀 🚀 Transfer Learning Codebase for reproducing experiments from our paper — covering ten downstream traffic classification tasks with three transfer approaches (k-NN, linear probing, and full model fine-tuning). 🚀 🚀

Datasets

The cesnet-datazoo package currently provides four datasets with details in the following table (you might need to scroll the table horizontally to see all datasets).

  1. CESNET-TLS22
  2. CESNET-QUIC22
  3. CESNET-TLS-Year22
  4. CESNET-QUICEXT-25
Name CESNET-TLS22 CESNET-QUIC22 CESNET-TLS-Year22 CESNET-QUICEXT-25
Protocol TLS QUIC TLS QUIC
Published in 2022 2023 2024 2025
Collection duration 2 weeks 4 weeks 1 year 1 year
Collection period 4.10.2021 - 17.10.2021 31.10.2022 - 27.11.2022 1.1.2022 - 31.12.2022 1.6.2024 - 31.5.2025
Application count 191 102 180 50
Available samples 141392195 153226273 507739073 194296462
Available dataset sizes XS, S, M, L XS, S, M, L XS, S, M, L XS, S, M, L
Cite 10.1016/j.comnet.2022.109467 10.1016/j.dib.2023.108888 10.1038/s41597-024-03927-4 In preparation
Zenodo URL zenodo.org/record/7965515 zenodo.org/record/7963302 zenodo.org/records/10608607 zenodo.org/records/17249078
Related papers 10.23919/TMA58422.2023.10199052 10.1145/3768988

Installation

Install the package from pip with:

pip install cesnet-datazoo

or for editable install with:

pip install -e git+https://github.com/CESNET/cesnet-datazoo

Examples

Initialize dataset to create train, validation, and test dataframes

from cesnet_datazoo.datasets import CESNET_QUIC22
from cesnet_datazoo.config import DatasetConfig, AppSelection

dataset = CESNET_QUIC22("/datasets/CESNET-QUIC22/", size="XS")
dataset_config = DatasetConfig(
    dataset=dataset,
    apps_selection=AppSelection.ALL_KNOWN,
    train_period_name="W-2022-44",
    test_period_name="W-2022-45",
)
dataset.set_dataset_config_and_initialize(dataset_config)
train_dataframe = dataset.get_train_df()
val_dataframe = dataset.get_val_df()
test_dataframe = dataset.get_test_df()

The DatasetConfig class handles the configuration of datasets, and calling set_dataset_config_and_initialize initializes train, validation, and test sets with the desired configuration. Data can be read into Pandas DataFrames as shown here or via PyTorch DataLoaders. See CesnetDataset reference.

See more examples in the documentation.

Papers

Acknowledgments

This project was supported by the Ministry of the Interior of the Czech Republic, grant No. VJ02010024: Flow-Based Encrypted Traffic Analysis.