Skip to content

Add QNN model #151

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xpai merged 9 commits into from
May 25, 2026
Merged

Add QNN model #151

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
26f0b8b
Update README.md
salmon1802 May 23, 2025
b45381d
Update README.md
salmon1802 May 23, 2025
3ad4b23
Add files via upload
salmon1802 May 23, 2025
71f5d10
Add QNN model
salmon1802 May 23, 2025
17cd9c8
Update README.md
salmon1802 May 23, 2025
678cb5e
Add QNN.py
salmon1802 May 24, 2025
702b96b
Add QNN_alpha.py
salmon1802 May 24, 2025
d87257e
Add QNN model
salmon1802 May 27, 2025
c9e7dfc
Merge remote-tracking branch 'origin/main' into salmon1802/main
xpai May 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ Click-through rate (CTR) prediction is a critical task for various industrial ap
| 38 | SIGIR'23 | [EulerNet](./model_zoo/EulerNet) | [EulerNet: Adaptive Feature Interaction Learning via Euler's Formula for CTR Prediction](https://dl.acm.org/doi/10.1145/3539618.3591681) :triangular_flag_on_post:**Huawei** | [:arrow_upper_right:](https://github.com/Ethan-TZ/EulerNet/tree/main/%23Code4FuxiCTR%23) | `torch` |
| 39 | CIKM'23 | [GDCN](./model_zoo/GDCN) | [Towards Deeper, Lighter and Interpretable Cross Network for CTR Prediction](https://dl.acm.org/doi/pdf/10.1145/3583780.3615089) :triangular_flag_on_post:**Microsoft** | | `torch` |
| 40 | ICML'24 | [WuKong](./model_zoo/WuKong) | [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://arxiv.org/abs/2403.02545) :triangular_flag_on_post:**Meta** | [:arrow_upper_right:](https://github.com/reczoo/BARS/tree/main/ranking/ctr/WuKong) | `torch` |
| 41 | KDD'25 | [QNN-α](./model_zoo/QNN) | [Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Prediction](https://arxiv.org/abs/2505.17999) :triangular_flag_on_post:**Huawei** | [:arrow_upper_right:](https://github.com/salmon1802/QNN/tree/main/checkpoints) | `torch` |
|<tr><th colspan=6 align="center">:open_file_folder: **Behavior Sequence Modeling**</th></tr>|
| 42 | KDD'18 | [DIN](./model_zoo/DIN) | [Deep Interest Network for Click-Through Rate Prediction](https://www.kdd.org/kdd2018/accepted-papers/view/deep-interest-network-for-click-through-rate-prediction) :triangular_flag_on_post:**Alibaba** | [:arrow_upper_right:](https://github.com/reczoo/BARS/tree/main/ranking/ctr/DIN) | `torch` |
| 43 | AAAI'19 | [DIEN](./model_zoo/DIEN) | [Deep Interest Evolution Network for Click-Through Rate Prediction](https://arxiv.org/abs/1809.03672) :triangular_flag_on_post:**Alibaba** | [:arrow_upper_right:](https://github.com/reczoo/BARS/tree/main/ranking/ctr/DIEN) | `torch` |
Expand Down
81 changes: 81 additions & 0 deletions model_zoo/QNN/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Prediction

## Model Overview

Click-through rate (CTR) prediction aims to accurately estimate user click behavior by leveraging multiple features, including user profiles, item attributes, and contextual information.
Existing CTR prediction models typically employ the Hadamard product for feature interaction, but the underlying mechanism remains insufficiently explored.
From the perspective of quadratic neural networks (QNN), we conducts both theoretical and empirical analyses of the Hadamard product-based feature interaction mechanism, and provides a novel interpretive framework to systematically explain its effectiveness and limitations.
Furthermore, we innovatively introduce multi-head Khatri–Rao products as an efficient alternative to the Hadamard product and propose a Self-Ensemble Loss, which further improves model performance without increasing inference latency.

<div align="center">
<img src="https://github.com/user-attachments/assets/0dfa50ce-db90-4abc-8e54-d7639f649545" width="600" alt="QNN model"/>
</div>



## Requirements

We have tested FinalMLP with the following requirements.

```python
python: 3.8
pytorch: 1.10
fuxictr: 2.0.1
```

## Configuration Guide


The `dataset_config.yaml` file contains all the dataset settings as follows.

| Params | Type | Default | Description |
| ----------------------------- | ---- | ------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| data_root | str | | the root directory to load and save data data |
| data_format | str | | input data format, "h5", "csv", or "tfrecord" supported |
| train_data | str | None | training data path |
| valid_data | str | None | validation data path |
| test_data | str | None | test data path |
| min_categr_count | int | 1 | min count to filter category features, |
| feature_cols | list | | a list of features with the following dict keys |
| feature_cols::name | str\|list | | feature column name in csv. A list is allowed in which the features have the same feature type and will be expanded accordingly. |
| feature_cols::active | bool | | whether to use the feature |
| feature_cols::dtype | str | | the input data dtype, "int"\|"str" |
| feature_cols::type | str | | feature type "numeric"\|"categorical"\|"sequence"\|"meta" |
| label_col | dict | | specify label column |
| label_col::name | str | | label column name in csv |
| label_col::dtype | str | | label data dtype |



The `model_config.yaml` file contains all the model hyper-parameters as follows.

| Params | Type | Default | Description |
| ----------------------- | --------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| model | str | "QNN_alpha" | model name,  which should be same with model class name |
| dataset_id | str | "TBD" | dataset_id to be determined |
| loss | str | "binary_crossentropy" | loss function |
| metrics | list | ['logloss', 'AUC'] | a list of metrics for evaluation |
| task | str | "binary_classification" | task type supported: ```"regression"```, ```"binary_classification"``` |
| optimizer | str | "adam" | optimizer used for training |
| learning_rate | float | 1.0e-3 | learning rate |
| embedding_regularizer | float\|str | 0 | regularization weight for embedding matrix: L2 regularization is applied by default. Other optional examples: ```"l2(1.e-3)"```, ```"l1(1.e-3)"```, ```"l1_l2(1.e-3, 1.e-3)"```. |
| net_regularizer | float\|str | 0 | regularization weight for network parameters: L2 regularization is applied by default. Other optional examples: ```"l2(1.e-3)"```, ```"l1(1.e-3)"```, ```"l1_l2(1.e-3, 1.e-3)"```. |
| batch_size | int | 10000 | batch size, usually a large number for CTR prediction task |
| embedding_dim | int | 32 | embedding dimension of features. Note that field-wise embedding_dim can be specified in ```feature_specs```. |
| num_layers | int | 3 | number of network layers |
| num_row | int | 3 | number of rows of Khatri-Rao Product, hyperparameter M in paper |
| net_dropout | float | 0 | dropout rate in QNN |
| batch_norm | bool | False | whether using BN in QNN |
| num_heads | int | 1 | number of heads used for Khatri-Rao Product |
| epochs | int | 100 | the max number of epochs for training, which can early stop via monitor metrics. |
| shuffle | bool | True | whether shuffle the data samples for each epoch of training |
| seed | int | 2021 | the random seed used for reproducibility |
| monitor | str\|dict | 'AUC' | the monitor metrics for early stopping. It supports a single metric, e.g., ```"AUC"```. It also supports multiple metrics using a dict, e.g., {"AUC": 2, "logloss": -1} means ```2*AUC - logloss```. |
| monitor_mode | str | 'max' | ```"max"``` means that the higher the better, while ```"min"``` denotes that the lower the better. |
| model_root | str | './checkpoints/' | the dir to save model checkpoints and running logs |
| early_stop_patience | int | 2 | training is stopped when monitor metric fails to become better for ```early_stop_patience=2```consective evaluation intervals. |
| save_best_only | bool | True | whether to save the best model checkpoint only |
| eval_steps | int\|None | None | evaluate the model on validation data every ```eval_steps```. By default, ```None``` means evaluation every epoch. |


#### For reproducing the results, please refer to https://github.com/salmon1802/QNN/tree/main/checkpoints
7 changes: 7 additions & 0 deletions model_zoo/QNN/config/dataset_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
### Tiny data for tests only
tiny_npz:
data_root: ../../data/
data_format: npz
train_data: ../../data/tiny_npz/train.npz
valid_data: ../../data/tiny_npz/valid.npz
test_data: ../../data/tiny_npz/test.npz
36 changes: 36 additions & 0 deletions model_zoo/QNN/config/model_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
Base:
model_root: './checkpoints/'
num_workers: 8
verbose: 1
early_stop_patience: 2
pickle_feature_encoder: True
save_best_only: True
eval_steps: null
debug_mode: False
group_id: null
use_features: null
feature_specs: null
feature_config: null

QNN_alpha_default: # This is a config template
model: QNN_alpha
dataset_id: TBD
loss: 'binary_crossentropy'
metrics: ['logloss', 'AUC']
task: binary_classification
optimizer: adam
learning_rate: 1.e-3
embedding_regularizer: 1.e-5
net_regularizer: 0
batch_size: 10000
embedding_dim: 16
num_layers: 3
num_row: 3
net_dropout: 0.1
num_heads: 1
batch_norm: True
epochs: 100
shuffle: True
seed: 2025
monitor: {'AUC': 1, 'logloss': 0}
monitor_mode: 'max'
3 changes: 3 additions & 0 deletions model_zoo/QNN/fuxictr_version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# pip install -U fuxictr
import fuxictr
assert fuxictr.__version__ == "2.0.1"
Loading
Loading