Skip to content

Commit 3ca2394

Browse files
vijaydwivedi75vijaydwivedi75
committed
initial commit
0 parents  commit 3ca2394

66 files changed

Lines changed: 4509 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#common
2+
**/*.DS_Store
3+
**/*.ipynb_checkpoints/
4+
**/__pycache__
5+
out/
6+
7+
gmail.py
8+
9+
#SBM datasets
10+
data/SBMs/*.pkl
11+
12+
#Superpixels datasets
13+
data/superpixels/*.pkl
14+
data/superpixels/*.zip
15+
PATH/
16+
17+
#ZINC dataset
18+
data/molecules/*.pkl
19+
data/molecules/*.pickle
20+
data/molecules/*.zip
21+
data/molecules/zinc-full/*.pkl
22+
23+
#TSP
24+
data/TSP/*.txt
25+
data/TSP/*.pkl
26+
data/TSP/*.zip
27+
data/TSP/pyconcorde/
28+
29+
#COLLAB
30+
data/COLLAB/*
31+
dataset/
32+
33+
#CSL
34+
data/CSL/*.pkl
35+
data/CSL/*.zip
36+
data/CSL/*.pt
37+
38+

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2020 Vijay Prakash Dwivedi, Xavier Bresson
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
2+
3+
# Graph Transformer Architecture
4+
5+
Source code for the paper "**A Generalization of Transformer Networks to Graphs**" by _[Vijay Prakash Dwivedi](https://github.com/vijaydwivedi75) and [Xavier Bresson](https://github.com/xbresson)_, at **AAAI'21 Workshop on Deep Learning on Graphs: Methods and Applications (DLG-AAAI'21)**.
6+
7+
We propose a generalization of transformer neural network architecture for arbitrary graphs: **Graph Transformer**. <br>Compared to the [Standard Transformer](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf), the highlights of the presented architecture are:
8+
9+
- The attention mechanism is a function of neighborhood connectivity for each node in the graph.
10+
- The position encoding is represented by Laplacian eigenvectors, which naturally generalize the sinusoidal positional encodings often used in NLP.
11+
- The layer normalization is replaced by a batch normalization layer.
12+
- The architecture is extended to have edge representation, which can be critical to tasks with rich information on the edges, or pairwise interactions (such as bond types in molecules, or relationship type in KGs. etc).
13+
14+
<br>
15+
16+
<p align="center">
17+
<img src="./docs/graph_transformer.png" alt="Graph Transformer Architecture" width="800">
18+
<br>
19+
<b>Figure</b>: Block Diagram of Graph Transformer Architecture
20+
</p>
21+
22+
23+
## 1. Repo installation
24+
25+
This project is based on the [benchmarking-gnns](https://github.com/graphdeeplearning/benchmarking-gnns) repository.
26+
27+
[Follow these instructions](./docs/01_benchmark_installation.md) to install the benchmark and setup the environment.
28+
29+
30+
<br>
31+
32+
## 2. Download datasets
33+
34+
[Proceed as follows](./docs/02_download_datasets.md) to download the datasets used to evaluate Graph Transformer.
35+
36+
37+
<br>
38+
39+
## 3. Reproducibility
40+
41+
[Use this page](./docs/03_run_codes.md) to run the codes and reproduce the published results.
42+
43+
44+
<br>
45+
46+
## 4. Reference
47+
48+
Coming soon!
49+
50+
<!-- ```
51+
@article{dwivedi2020graph,
52+
title={A Generalization of Transformer Networks to Graphs},
53+
author={Dwivedi, Vijay Prakash and and Bresson, Xavier},
54+
journal={arXiv preprint arXiv:2012.xxxxx},
55+
year={2020}
56+
}
57+
```
58+
-->
59+
60+
61+
<br><br><br>
62+

‎configs/SBMs_GraphTransformer_CLUSTER_500k_full_graph_BN.json‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"gpu": {
3+
"use": true,
4+
"id": 0
5+
},
6+
7+
"model": "GraphTransformer",
8+
"dataset": "SBM_CLUSTER",
9+
10+
"out_dir": "out/SBMs_full_NoPE_BN/",
11+
12+
"params": {
13+
"seed": 41,
14+
"epochs": 1000,
15+
"batch_size": 10,
16+
"init_lr": 0.0005,
17+
"lr_reduce_factor": 0.5,
18+
"lr_schedule_patience": 10,
19+
"min_lr": 1e-6,
20+
"weight_decay": 0.0,
21+
"print_epoch_interval": 5,
22+
"max_time": 24
23+
},
24+
25+
"net_params": {
26+
"L": 10,
27+
"n_heads": 8,
28+
"hidden_dim": 80,
29+
"out_dim": 80,
30+
"residual": true,
31+
"readout": "mean",
32+
"in_feat_dropout": 0.0,
33+
"dropout": 0.0,
34+
"layer_norm": false,
35+
"batch_norm": true,
36+
"self_loop": false,
37+
"lap_pos_enc": false,
38+
"wl_pos_enc": false,
39+
"full_graph": true
40+
}
41+
}

‎configs/SBMs_GraphTransformer_CLUSTER_500k_full_graph_LN.json‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"gpu": {
3+
"use": true,
4+
"id": 0
5+
},
6+
7+
"model": "GraphTransformer",
8+
"dataset": "SBM_CLUSTER",
9+
10+
"out_dir": "out/SBMs_full_NoPE_LN/",
11+
12+
"params": {
13+
"seed": 41,
14+
"epochs": 1000,
15+
"batch_size": 10,
16+
"init_lr": 0.0005,
17+
"lr_reduce_factor": 0.5,
18+
"lr_schedule_patience": 10,
19+
"min_lr": 1e-6,
20+
"weight_decay": 0.0,
21+
"print_epoch_interval": 5,
22+
"max_time": 24
23+
},
24+
25+
"net_params": {
26+
"L": 10,
27+
"n_heads": 8,
28+
"hidden_dim": 80,
29+
"out_dim": 80,
30+
"residual": true,
31+
"readout": "mean",
32+
"in_feat_dropout": 0.0,
33+
"dropout": 0.0,
34+
"layer_norm": true,
35+
"batch_norm": false,
36+
"self_loop": false,
37+
"lap_pos_enc": false,
38+
"wl_pos_enc": false,
39+
"full_graph": true
40+
}
41+
}

‎configs/SBMs_GraphTransformer_CLUSTER_500k_sparse_graph_BN.json‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"gpu": {
3+
"use": true,
4+
"id": 0
5+
},
6+
7+
"model": "GraphTransformer",
8+
"dataset": "SBM_CLUSTER",
9+
10+
"out_dir": "out/SBMs_sparse_NoPE_BN/",
11+
12+
"params": {
13+
"seed": 41,
14+
"epochs": 1000,
15+
"batch_size": 32,
16+
"init_lr": 0.0005,
17+
"lr_reduce_factor": 0.5,
18+
"lr_schedule_patience": 10,
19+
"min_lr": 1e-6,
20+
"weight_decay": 0.0,
21+
"print_epoch_interval": 5,
22+
"max_time": 24
23+
},
24+
25+
"net_params": {
26+
"L": 10,
27+
"n_heads": 8,
28+
"hidden_dim": 80,
29+
"out_dim": 80,
30+
"residual": true,
31+
"readout": "mean",
32+
"in_feat_dropout": 0.0,
33+
"dropout": 0.0,
34+
"layer_norm": false,
35+
"batch_norm": true,
36+
"self_loop": false,
37+
"lap_pos_enc": false,
38+
"wl_pos_enc": false,
39+
"full_graph": false
40+
}
41+
}

‎configs/SBMs_GraphTransformer_CLUSTER_500k_sparse_graph_LN.json‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
{
2+
"gpu": {
3+
"use": true,
4+
"id": 0
5+
},
6+
7+
"model": "GraphTransformer",
8+
"dataset": "SBM_CLUSTER",
9+
10+
"out_dir": "out/SBMs_sparse_NoPE_LN/",
11+
12+
"params": {
13+
"seed": 41,
14+
"epochs": 1000,
15+
"batch_size": 32,
16+
"init_lr": 0.0005,
17+
"lr_reduce_factor": 0.5,
18+
"lr_schedule_patience": 10,
19+
"min_lr": 1e-6,
20+
"weight_decay": 0.0,
21+
"print_epoch_interval": 5,
22+
"max_time": 24
23+
},
24+
25+
"net_params": {
26+
"L": 10,
27+
"n_heads": 8,
28+
"hidden_dim": 80,
29+
"out_dim": 80,
30+
"residual": true,
31+
"readout": "mean",
32+
"in_feat_dropout": 0.0,
33+
"dropout": 0.0,
34+
"layer_norm": true,
35+
"batch_norm": false,
36+
"self_loop": false,
37+
"lap_pos_enc": false,
38+
"wl_pos_enc": false,
39+
"full_graph": false
40+
}
41+
}

‎configs/SBMs_GraphTransformer_LapPE_CLUSTER_500k_full_graph_BN.json‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
{
2+
"gpu": {
3+
"use": true,
4+
"id": 0
5+
},
6+
7+
"model": "GraphTransformer",
8+
"dataset": "SBM_CLUSTER",
9+
10+
"out_dir": "out/SBMs_full_LapPE_BN/",
11+
12+
"params": {
13+
"seed": 41,
14+
"epochs": 1000,
15+
"batch_size": 10,
16+
"init_lr": 0.0005,
17+
"lr_reduce_factor": 0.5,
18+
"lr_schedule_patience": 10,
19+
"min_lr": 1e-6,
20+
"weight_decay": 0.0,
21+
"print_epoch_interval": 5,
22+
"max_time": 24
23+
},
24+
25+
"net_params": {
26+
"L": 10,
27+
"n_heads": 8,
28+
"hidden_dim": 80,
29+
"out_dim": 80,
30+
"residual": true,
31+
"readout": "mean",
32+
"in_feat_dropout": 0.0,
33+
"dropout": 0.0,
34+
"layer_norm": false,
35+
"batch_norm": true,
36+
"self_loop": false,
37+
"lap_pos_enc": true,
38+
"pos_enc_dim": 10,
39+
"wl_pos_enc": false,
40+
"full_graph": true
41+
}
42+
}

‎configs/SBMs_GraphTransformer_LapPE_CLUSTER_500k_full_graph_LN.json‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
{
2+
"gpu": {
3+
"use": true,
4+
"id": 0
5+
},
6+
7+
"model": "GraphTransformer",
8+
"dataset": "SBM_CLUSTER",
9+
10+
"out_dir": "out/SBMs_full_LapPE_LN/",
11+
12+
"params": {
13+
"seed": 41,
14+
"epochs": 1000,
15+
"batch_size": 10,
16+
"init_lr": 0.0005,
17+
"lr_reduce_factor": 0.5,
18+
"lr_schedule_patience": 10,
19+
"min_lr": 1e-6,
20+
"weight_decay": 0.0,
21+
"print_epoch_interval": 5,
22+
"max_time": 24
23+
},
24+
25+
"net_params": {
26+
"L": 10,
27+
"n_heads": 8,
28+
"hidden_dim": 80,
29+
"out_dim": 80,
30+
"residual": true,
31+
"readout": "mean",
32+
"in_feat_dropout": 0.0,
33+
"dropout": 0.0,
34+
"layer_norm": true,
35+
"batch_norm": false,
36+
"self_loop": false,
37+
"lap_pos_enc": true,
38+
"pos_enc_dim": 10,
39+
"wl_pos_enc": false,
40+
"full_graph": true
41+
}
42+
}

0 commit comments

Comments
 (0)