It seems that F1 score can not match the paper, for example, I set epoch=200, layers=96 64 32, learning rate=0.001, but on BIT-OTC dataset the F1 score can only reach 0.802.

I was wondering if there is anything wrong about my experimental settings?

Before starting, thank you for developing the SGCN model. When I tried to run the code, I faced a problem with input data.

In your example data(bitcoin_otc), all ids have a numerical value. But my data has string data that need to convert to numerical ID. I guess that I should add a converter such as key and library, I'm not sure where should I modify it. Can you suggest the method for running such cases?

Hi Benedek,

Thanks for releasing the code!

I find that the F1 value in the result stays the same. Could you give me some advice on how I should correct it?

Looking forward to your reply. Thanks!

Hello @benedekrozemberczki :

I found out a little bug in file ./src/signedsageconvolution.py

the code below in your file

edge_index = add_self_loops(edge_index, num_nodes=x.size(0))

should be replaced by

edge_index, _ = add_self_loops(edge_index, num_nodes=x.size(0))

i.e. just add , _ after edge_index

All the add_self_loops lines in the file ./src/signedsageconvolution.py are modified as I typed above.

(perhaps it is just a slip of the finger... sorry, i just know a slip of the tongue...)

and finally, it works.

My Environment is:

Ubuntu 16.04
Python 3.7.3
PyTorch 1.1.0
PyTorch_Geometric 1.2.1
PyTorch_Scatter 1.2.0

Yours, @wmf1997

(.venv) [email protected]:~/ub16_prj/SGCN$ python src/main.py Traceback (most recent call last): File "src/main.py", line 3, in from sgcn import SignedGCNTrainer File "/home/ub16hp/ub16_prj/SGCN/src/sgcn.py", line 12, in from torch_geometric.nn import SAGEConv File "/home/ub16hp/ub16_prj/SGCN/.venv/lib/python3.5/site-packages/torch_geometric/nn/init.py", line 1, in from .conv import * # noqa File "/home/ub16hp/ub16_prj/SGCN/.venv/lib/python3.5/site-packages/torch_geometric/nn/conv/init.py", line 1, in from .spline_conv import SplineConv File "/home/ub16hp/ub16_prj/SGCN/.venv/lib/python3.5/site-packages/torch_geometric/nn/conv/spline_conv.py", line 3, in from torch_spline_conv import SplineConv as Conv ImportError: No module named 'torch_spline_conv' (.venv) [email protected]:~/ub16_prj/SGCN$

Dear bendek, Greetings, I have a question about the calculation of AUPR in SGCN. I'm trying to get AUPR score by input the below code in utils.py:

from sklearn.metrics import roc_auc_score, f1_score, average_precision_score, precision_recall_curve, auc, plot_precision_recall_curve

def calculate_auc(targets, predictions, edges):
    """
    Calculate performance measures on test dataset.
    :param targets: Target vector to predict.
    :param predictions: Predictions vector.
    :param edges: Edges dictionary with number of edges etc.
    :return auc: AUC value.
    :return f1: F1-score.
    """
    targets = [0 if target == 1 else 1 for target in targets]
    auc_score = roc_auc_score(targets, predictions)
    #precision, recall, thresholds = precision_recall_curve(targets, predictions)
    #aupr=auc(recall, precision)
    pred = [1 if p > 0.5 else 0 for p in  predictions]
    f1 = f1_score(targets, pred)
    #precision, recall, thresholds = precision_recall_curve(targets, pred)
    #aupr=auc(recall, precision)
    pos_ratio = sum(pred)/len(pred)
    return auc_score, aupr, f1, pos_ratio

But I'm not sure where should I put the new code (#). Does AUPR need to get predictions value or pred value? Also, what the different between predictions and pred? I guess 'predictions' is probability value and 'pred' is binary value, is it right?

Sorry for frequent edit, but why did you set targets = [0 if target == 1 else 1 for target in targets]? I think that it means give opposite labels to positive and negative edges. (i.e., positive=0, negative=1) I look forward to your reply. Sincerely, Songyeon

The first commit: Fix negative sampling. Avoid sampling positive edges and viewing them as non-edges, and avoid sampling negative edges and viewing them as non-edges. Borrow utility functions from torch_geometric. The trick to avoiding sampling an edge that exists is to map the existent links and the sampled links to some unique numbers (by multiplying the source node with the total number of nodes, then add the target node, ref: https://github.com/rusty1s/pytorch_geometric/blob/712f581642bccc225189fc3eb364ce0642b915a3/torch_geometric/utils/negative_sampling.py#L95) After sampling and mapping, we use NumPy.isin to pick out the invalid links (assumed to be non-existent, but exists), then resample iteratively until no invalid samples exist.

The second commit: Add a bias term in regression to predict link signs. This is to learn some prior distribution of different types of links: positive, negative and none. The bias term can be helpful in dealing with unbalanced links. In the data sets used in SGCN, most links do not exist, and positive links are much more frequent than negative ones.

Fix classifier so that it will not always predict the majority class, also add one column to the output table, change README correspondingly as well.

There are some problems using a threshold determined by the ratio of negative links: On the one hand, you need to know the ground truth negative link ratio, which is a kind of cheating. On the other hand, due to the high imbalance of positive and negative links, your threshold would be close to zero, resulting in voting the majority class all the time.

thanks for your code！ I have a little question,

self.y should bu related to negative_edges and test_negative_edges, when we calculate "calculate_regression_loss" function, pos = torch.cat((self.positive_z_i, self.positive_z_j), 1) # [14624, 128] neg = torch.cat((self.negative_z_i, self.negative_z_j), 1) # [2522, 128]