An Information-Theoretic Analysis of Temporal GNNs

Abstract

Temporal Graph Neural Networks, a new and trending area of machine learning, suffers from a lack of formal analysis. In this paper, information theory is used as the primary tool to provide a framework for the analysis of temporal GNNs. For this reason, the concept of information bottleneck is used and adjusted to be suitable for a temporal analysis of such networks. To this end, a new definition for Mutual Information Rate is provided, and the potential use of this new metric in the analysis of temporal GNNs is studied.

I Introduction

The study of Graph Neural Networks (GNNs) is a rapidly growing area of machine learning and artificial intelligence. First introduced by [1] back in 2009, it has now become one of the trendiest areas in machine learning research. The importance of this field becomes more evident by observing the amount of graphical data structures around us. Social networks, communication networks, and biological networks are only a few examples of such graphs. Graphs can be used to represent the relationship present between data in many different fields of science, and therefore GNNs have been used to solve problems in areas such as modern recommender systems, computer vision, natural language processing (NLP), software mining, bioinformatics, and urban intelligence [2].

Additionally, most of these graphs are dynamic by nature. This means that we expect them to change and evolve over time. For instance, new nodes and links are always being formed or deleted in social media networks. Because of this reason, Temporal GNNs have recently attracted a lot of attention from the machine learning community. Temporal GNNs are used to study and learn from graphical data and their evolution through time, whereas standard GNNs learn from single graphs. A survey on existing methods on this topic has been recently published [3], and even newer methods have been introduced since then [4].

Generally speaking, there is a lack of theoretical formulations to assess the performance of GNNs. Consequently, the temporal version of GNNs also suffers from the same lack of theoretical assessment in a more serious manner. Because of this, we propose an information-theoretic approach to analysing the performance of temporal graph learning algorithms. The information bottleneck principle is used as the core idea, while the temporal graph evolutions are modeled using stochastic processes. Mutual Information Rate (MIR) is then used as the main metric for analysing the performance of algorithms that work on temporal graphs. Additionally, a new and more robust metric to replace MIR is introduced, and it is discussed how NECs can be used for analysing temporal GNNs.

II The information bottleneck problem for temporal GNNs

The information-bottleneck problem [5] is often used as the information-theoretic formulation of Machine Learning. This method models the machine learning algorithm using an encoder-decoder pair, and formulates the problem as a trade-off between the compression ratio of the input and the accuracy of the output. We show the input to the algorithm with $X$, the desired output with $Y$, and the actual output of the algorithm with $\hat{Y}$. Additionally, we use $Z$ to show the output of the encoder. This will be the output of the last layer if we are working with a neural network [6]. Fig. 1 shows these notations in the context of a learning algorithm.

Figure 1: Model of an ML algorithm

Using the model of Fig. 1, the information bottleneck problem can be used to model the ML problem as below.

$$\begin{split}\textrm{inf}&\quad I(X;Z)\\ \textrm{subject to}&\quad I(Y;Z)\geq\alpha,\end{split}$$ (1)

where $I$ is used to show mutual information as per Shannon’s definition [7, Ch. 2], and $\alpha$ is the parameter that sets the accuracy of the algorithm.

However, Eq. 1 can not be used in the context of dynamic graphs. This is because when dealing with temporal graphs, we should look at sequences of the input and output, rather than single ones. In this case, rather than assuming the variables in Eq. 1 to be single graphs, we will consider them to be stochastic processes. In other words we have

$$\begin{split}Y&=\{Y_{1},Y_{2},\ldots,Y_{n}\},\\ X&=\{X_{1},X_{2},\ldots,X_{n}\},\\ Z&=\{Z_{1},Z_{2},\ldots,Z_{n}\},\\ \hat{Y}&=\{\hat{Y}_{1},\hat{Y}_{2},\ldots,\hat{Y}_{n}\},\\ \end{split}$$ (2)

where the following Markov chain governs each $i$, which is a time snapshot of the network.

$$Y_{i}\rightarrow X_{i}\rightarrow Z_{i}\rightarrow\hat{Y}_{i}$$ (3)

Eq. 1 tells us that in order to assess an ML algorithm we can look at the mutual information between $X$ and $Z$, and $Y$ and $Z$, respectively. However, in a dynamic graph setup, we have $n$ time snapshots of these random variables. The natural question that arises in these scenario is the value of $i$ to look at. In this case, we will use the concept of MIR [8].

When working with stochastic processes, we can look at the mutual information between pairs of corresponding variables, or $I(X_{1},\ldots,X_{n};Y_{1},\ldots,Y_{n})$. However, the value of this can theoretically be unlimited as the process grows large. Therefore, we rely on the definition of MIR, which measures average mutual information per unit of time defined as below.

Definition 1 (Mutual Information Rate (MIR)).

The mutual information rate between two stochastic processes $X$ and $Y$ is defined as below.

$$\textrm{MIR}=\lim_{n\to\infty}\frac{1}{n}I\left(X_{1},X_{2},\ldots,X_{n};Y_{1},Y_{2},\ldots,Y_{n}\right)$$

Having defined the variables in Fig. 1 and using the notions from the information bottleneck principle, we propose the following metrics to be used for the assessment of temporal graph learning algorithms.

• Accuracy: $\textrm{MIR}(Y;Z)$

• Compression: $\textrm{MIR}(X;Z)$

• Overall score: $\textrm{MIR}(Y;Z)/\textrm{MIR}(X;Z)$

Note that there exist numerous methods for estimating the MIR of a pair of stochastic processes. For instance, the mutual information estimation setup proposed in [5][p. 17] can be used. However, because of the existence of $2n$ random variables in the definition of MIR, especially as $n$ grows large, we propose an alternative metric for measuring the rate of mutual information.

In the remainder of this section, we will use the following notation for better readability when referring to subsets of a stochastic process.

$$X_{i:j}\equiv\{X_{i},X_{i+1},\ldots,X_{j}\}$$ (4)

III Alternative Mutual Information Rate (AMIR)

The current definition for MIR makes its applications to stochastic processes very limited, as it involves the mutual information of the entire process as it grows large. This notion of MIR might not be feasible to study for real-world processes, as estimating the mutual information of such big stochastic processes might not be accurate. Because of this, we propose the definition of a novel relative metric for the mutual information rate between processes. This new metric is called alternative mutual information rate, or AMIR for short. Even though a similar metric exists for entropy rate of stochastic processes, the absence of this metric for analysing mutual information can be felt. This novel metric analyses the mutual information of single random variables, given the history of the sequence. We believe that this novel metric can be estimated more accurately, especially if the given stochastic processes are Markov chains.

Firstly, note that the definition of MIR as stated in Definition 1 is very similar to that of entropy rate [7, Ch. 4]. Additionally, there exists a relative version of entropy rate, which is shown to be equal to entropy rate for stationary processes [7, Thm. 4.2.1]. This new relative version of the entropy rate had the benefit of being easier to work with and use in our estimations and proofs. Based on the similarities between the concepts of entropy rate and MIR, we search for a relative format for MIR as well. This conditional concept of MIR can help us in many applications that involve the correlation of two stochastic processes, where we want to study the limit of the behaviour of single random variables in a conditional manner. Based on this, we provide the following definition for this alternative version of MIR.

Definition 2 (Alternative Mutual Information Rate (AMIR)).

The Alternative Mutual Information Rate (AMIR) between two stochastic processes $X$ and $Y$ is defined as

$$\text{AMIR}(X;Y)=\lim_{n\to\infty}I(X_{n};Y_{n}|X_{1:n-1},Y_{1:n-1}),$$ (5)

when it exists.

In the remainder of this section, we will look at the properties of AMIR, its existence, and its relationship with MIR. We first start with the following theorem.

Theorem 3.

For two stationary processes $X$ and $Y$, AMIR exists.

Proof.

It is well known that for two stationary processes $X$ and $Y$, $\text{MIR}(X;Y)$ exists [8]. For AMIR, we can write

$$\begin{split}\text{AMIR}(X;Y)&=\lim_{n\to\infty}I(X_{n};Y_{n}|X_{1:n-1},Y_{1:n-1})\\ &=\lim_{n\to\infty}H(X_{n}|X_{1:n-1},Y_{1:n-1})\\ &+H(Y_{n}|X_{1:n-1},Y_{1:n-1})\\ &-H(X_{n},Y_{n}|X_{1:n-1},Y_{1:n-1}).\\ \end{split}$$ (6)

Firstly, note that based on the stationarity of $X$, $Y$, and $(X,Y)$, we have the following equation [7, Ch. 4].

$$\lim_{n\to\infty}H(X_{n},Y_{n}|X_{1:n-1},Y_{1:n-1})=\lim_{n\to\infty}\frac{1}{n}H(X_{1:n},Y_{1:n})$$ (7)

Eq. 7 shows that $H(X_{n},Y_{n}|X_{1:n-1},Y_{1:n-1})$ tends towards the entropy rate of $(X,Y)$, and has a limit. Therefore, in order to show that the limit for AMIR exists, we only need to show that the other two limits in Eq. 6 exist. For $H(X_{n}|X_{1:n-1},Y_{1:n-1})$ we have

$$\begin{split}H(X_{n}|X_{1:n-1},Y_{1:n-1})&\leq H(X_{n}|X_{2:n-1},Y_{2:n-1})\\ &=H(X_{n-1}|X_{1:n-2},Y_{1:n-2}),\end{split}$$ (8)

where the first line comes from the fact that conditioning reduces entropy, and the second line can be concluded from the stationarity of both processes. Therefore, $H(X_{n}|X_{1:n-1},Y_{1:n-1})$ forms a non-increasing sequence of positive numbers, which has a limit. A similar approach can be used to prove that $H(Y_{n}|X_{1:n-1},Y_{1:n-1})$ also converges. Therefore, AMIR also has a limit and is well-defined for stationary processes. ∎

Additionally, we can state the following corollary for Theorem 3.

Corollary 4.

For stationary processes $X$ and $Y$ we always have

$$\text{AMIR}(X;Y)\leq\text{MIR}(X;Y).$$ (9)

Proof.

We can use Eq. 6 and write the following chain of equations.

$$\begin{split}\text{AMIR}(X;Y)&=\lim_{n\to\infty}H(X_{n}|X_{1:n-1},Y_{1:n-1})\\ &+H(Y_{n}|X_{1:n-1},Y_{1:n-1})\\ &-H(X_{n},Y_{n}|X_{1:n-1},Y_{1:n-1})\end{split}$$ (10)

$$\begin{split}&=\lim_{n\to\infty}H(X_{n}|X_{1:n-1})\\ &-I(X_{n};Y_{1:n-1})\\ &+H(Y_{n}|Y_{1:n-1};X_{1:n-1})\\ &-I(Y_{n};X_{1:n-1})\\ &-H(X_{n},Y_{n}|X_{1:n-1},Y_{1:n-1})\end{split}$$ (11)

$$\begin{split}&=H(X)+H(Y)-H(X,Y)\\ &-I(X_{n};Y_{1:n-1})-I(Y_{n};X_{1:n-1})\end{split}$$ (12)

$$\begin{split}&=\text{MIR}(X;Y)\\ &-I(X_{n};Y_{1:n-1})-I(Y_{n};X_{1:n-1})\end{split}$$ (13)

Based on the fact that mutual information is non-negative, we can easily conclude the claim of the corollary based on Eq. 13. ∎

Additionally, the conditions under which MIR and AMIR are equal are stated in the following corollary.

Corollary 5.

For stationary processes $X$ and $Y$, MIR and AMIR are equal if and only if the following conditions are met.

1. Given $X_{1:n-1}$, $X_{n}$ and $Y_{1:n-1}$ are independent.

2. Given $Y_{1:n-1}$, $Y_{n}$ and $X_{1:n-1}$ are independent.

Proof.

As $I(X_{n};Y_{1:n-1}|X_{1:n-1})$ and $I(Y_{n};X_{1:n-1}|Y_{1:n-1})$ are non-negative, the only way that the values of MIR and AMIR can be equal according to Eq. 13 is if these two values are zero. For $I(X_{n};Y_{1:n-1}|X_{1:n-1})$ we can write

$$\begin{split}&I(X_{n};Y_{1:n-1}|X_{1:n-1})=0\\ &\Rightarrow H(X_{n}|X_{1:n-1})-H(X_{n}|X_{1:n-1},Y_{1:n-1})=0\\ &\Rightarrow H(X_{n}|X_{1:n-1})=H(X_{n}|X_{1:n-1},Y_{1:n-1}),\end{split}$$ (14)

which translates into $X_{n}$ and $Y_{1:n-1}$ being independent given $X_{1:n-1}$. The other condition can be obtained by setting $I(Y_{n};X_{n-1}|Y_{n-1})$ to zero. ∎

It can be seen that in addition to the processes being stationary, the conditions in Corollary 5 need to be met in order for MIR and AMIR to be equal. This is unlike the equality of entropy rate and relative entropy rate, where the only condition was the stationarity of the process. However, it can also be seen that the conditions for the equality of these two metrics are not very strict, and can be assumed for many real-life pairs of stochastic processes.

IV Benefits of AMIR

In this section, we will briefly mention and discuss two advantages of AMIR over MIR.

IV-A Rate of convergence

Existing methods for estimating mutual information of random variables rely on samples from the distribution of those random variables [9, 10]. Therefore, as the space of possible data points grows, more sample are needed to accurately estimate this metric. Looking at the definition of MIR and AMIR, it can be seen that the space of both metrics includes $(|X||Y|)^{n}$ possibilities. However, there are usually assumptions in place about the memory of the system, which dictates how many samples in the future will be affected by the current sample. Based on this, there usually exists a constant C for which we can assume

$$I(X_{n};Y_{n}|X_{1:n-1},Y_{1:n-1})\\ =I(X_{n};Y_{n}|X_{n-C:n-1},Y_{n-C:n-1}).$$ (15)

This assumption will reduce the size of the sample space to $(|X||Y|)^{C+1}$, which lowers the number of samples needed to estimate AMIR and increase the convergence rate.

IV-B Number of sample sequences needed

It must be noted that in order to estimate MIR, we need samples of $(X_{n},Y_{n})$, which means samples of both sequences. Additionally, the number of required samples for accurate estimation will increase exponentially with $n$. However, with AMIR, one sequence for $(X,Y)$ will suffice if the memory of the system is assumed to be limited. In other words, with every new sample for the sequence, we will have a new sample for estimating $I(X_{n},Y_{n}|X_{n-C:n-1},Y_{n-C:n-1})$, which removes the need for having numerous sequences.

V AMIR of Markov chains

In this section, we will consider the Mutual Information Rate of stationary Markov chains. As a special case of stochastic processes, Markov chains have applications in many different areas.

Consider $X$ and $Y$ to be stationary Markov chains. Let $\mu_{X}$, $\mu_{Y}$, and $\mu_{XY}$ be the stationary distributions of $X$, $Y$, and $(X,Y)$, respectively. Additionally, let $P$, $Q$, and $R$ be the transition probability matrices for $X$, $Y$, and $(X,Y)$, respectively. To calculate the MIR of $X$ and $Y$, we can write

$$\begin{split}\text{MIR}(X;Y)&=H(X)+H(Y)-H(X,Y)\\ &=-\sum_{ij}\mu_{X,i}P_{ij}\log P_{ij}\\ &-\sum_{ij}\mu_{Y,i}Q_{ij}\log Q_{ij}\\ &+\sum_{ij}\mu_{XY,i}R_{ij}\log R_{ij}.\end{split}$$ (16)

Note that the entropy rate of stationary Markov chains can be found using [7, Thm. 4.2.4].

Additionally, it can be seen that the condition of causality holds for most of the processes that are simulated using Markov chains. For instance, Markov chains have recently been used for temporal alignment of medical signals in dementia [11]. In this study, it is safe to assume that future signals will not affect the past, and therefore the assumption of causality holds. Consequently, it is safe to state that for most of the Markov chains we work with, MIR and AMIR will be equal, but it is always necessary to check the condition of Corollary 5.

Even though both MIR and AMIR can be calculated using Eq. 16, we do not necessarily always have all the parameters of the Markov chain, and therefore can not use Eq. 16 directly. Therefore, we need to rely on estimation. We believe that in estimation, AMIR can prove to be a more robust and reliable estimator compared to MIR, as much less parameters are involved. Observe that for MIR, we need to estimate $I(X_{1},\ldots,X_{n};Y_{1},\ldots,Y_{n})$, whereas the value to estimate for AMIR in Markov chains is simply $I(X_{n};Y_{n}|X_{n-1},Y_{n-1})$.

VI Simulation results

In this section, we will perform simulations on a number of stochastic processes to compare the performance of MIR and AMIR as metrics of mutual information to study stochastic processes.

VI-A Gaussian processes

In this section, we simulate MIR and AMIR for two correlated Gaussian processes. For this purpose, we chose the following processes.

• $X_{i}\sim\mathcal{N}(0,\,1)$

• $Y_{i}\sim 0.8X_{i}+0.6\mathcal{N}(0,\,1)$

It can be checked that as the variables in these processes are i.i.d, they satisfy the conditions of Corollary 5, and MIR and AMIR are equal for these two processes. Additionally, both MIR and AMIR are simply equal to the mutual information of a pair of these random variables, which can be calculated as below.

$$\begin{split}I(X_{i},Y_{i})&=H(Y_{i})-H(Y_{i}|X_{i})\\ &=H(\mathcal{N}(0,\,1))-H(\mathcal{N}(0,\,0.36))\\ &=0.5\log\frac{1}{0.36}\end{split}$$ (17)

Fig. 2 shows the result of this simulation for estimating MIR and AMIR. Fig. 2(a) shows the estimate for MIR and AMIR as the sequence grows large, alongside the actual calculated value for the mutual information. Fig. 2(b) shows the error in the estimate as the difference between the estimated values for MIR and AMIR with the actual value. It can be seen that even though AMIR shows more fluctuations, it is faster in converging to the theoretical limit. This is in accordance with our predictions about the behaviour of AMIR.

(a) The estimations for MIR and AMIR

(b) Error in the convergence of MIR and AMIR

Figure 2: Estimating MIR and AMIR

VI-B Hidden Markov Model

In this section, we created and simulated a two state Hidden Markov Model (HMM) $(X,Y)$ with the following properties.

$$\begin{cases}\mathbf{Z}_{X}=\begin{bmatrix}0.8&0.2\\ 0.4&0.6\end{bmatrix}\\ p(Y_{i}|X_{i})=\begin{bmatrix}0.7&0.3\\ 0.3&0.7\end{bmatrix}\end{cases}$$ (18)

Firstly, it must be noted that this pair of stationary processes does not satisfy the conditions of Corollary 5. This is because it can be checked that $Y_{i}$ is not independent from $X_{i-1}$. Therefore, we expect MIR and AMIR to be different for this pair of processes. Based on the parameters given in Eq. 18, the values for MIR and AMIR can be calculated, and have the following values.

$$\begin{cases}\text{MIR}(X;Y)=0.1035\text{ bits}\\ \text{AMIR}(X;Y)=0.0892\text{ bits}\\ \end{cases}$$ (19)

We then ran the simulation for one random instance of the HMM with a length of 4000, in order to estimate the value for AMIR using its definition. The results of this simulation are illustrated in Fig. 3. The first thing that can be observed is the fact that the estimated AMIR is clearly tending towards its theoretical limit, as proven in Theorem 3. Secondly, it can be seen that as stated in Corollary 4, the value for AMIR is less than MIR. Ultimately, this result is showcasing one of the main advantages of AMIR, which is that it can be estimated from having a single sequence from both processes. However, to estimate MIR from observed data, we are in need of numerous instances of both sequences.

Figure 3: Estimating AMIR of an HMM

VII Application of Network Evolution Chains to temporal GNNs

It can be seen that due to the similarities between the concepts of MIR and entropy rate, Network Evolution Chains [12] can be used to simulate the sequence of input graphs, and the information-theoretic analysis done in [12] can be employed to study temporal GNNs. Using Network Evolution Chains to model the input sequence of graphs will impose a Markov chain assumption on the input, which will result in the following simplification for MIR.

$$\textrm{MIR}=\lim_{n\to\infty}I\left(X_{n};Y_{n}|X_{n-1},Y_{n-1}\right)$$ (20)

Additionally, there are parallels to all the concepts discussed in [12] in terms of MIR. For instance, an equivalent of the AEP can be stated for MIR in the following format.

$$\frac{1}{n}\log\frac{p(X_{1:n},Y_{1:n})}{p(X_{1:n})p(Y_{1:n})}\rightarrow\text{MIR}(X;Y)\quad\text{with probability 1}$$ (21)

The proof for Eq. 21 can be done in a similar way to the original AEP, which can be found in [7, Thm. 16.8.1].

If the input graphs to a temporal GNN are modeled using an NEC, then it satisfies the conditions for the AEP for MIR. Additionally, as the relative version of MIR, AMIR can be used to calculate the MIR of the process, just like the approach done in [12]. This then gives rise to the extension of all the concepts discussed in [12] in the context of temporal GNNs and MIR. This goes beyond the scope of this paper, and is left as a future work on this subject.

VIII Conclusion

In this paper, we provided the information-theoretic fundamentals of the analysis of temporal GNNs. After emphasizing on the importance and relevance of GNNs, we argued why there has recently been a lot of research on the temporal analysis of such networks. Because of the lack of a theoretical framework for analysing temporal GNNs, we introduced an information-theoretic methodology to assess the performance of temporal GNNs. Finally, we argued why simulating the temporal evolution of graphs using NECs can further improve the benefits of the introduced model, and expand its applications.