The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2024)

3.1 Preliminaries

We study decentralized partially observable Markov decision processes (DEC-POMDP) [24] with shared rewards. A DEC-POMDP is defined by $⟨𝒮,𝒜,O,R,P,n,\gamma ⟩$. $𝒮$ is the state space. $𝒜$ is the shared action space for each agent $i$. $o_i=O(s;i)$ is the local observation for agent $i$ at global state $s$. $P(s^{\prime }|s,A)$ denotes the transition probability from $s$ to $s^{\prime }$ given the joint action $A=(a_1,\mathrm{\dots },a_n)$ for all $n$ agents. $R(s,A)$ denotes the shared reward function. $\gamma$ is the discount factor.Agents use a policy ${\pi }_{\theta }(a_i|o_i)$ parameterized by $\theta$ to produce an action $a_i$ from the local observation $o_i$, andjointly optimize the discounted accumulated reward $J(\theta )={𝔼}_{A^t,s^t}\left[{\sum }_t{\gamma }^{t}R(s^t,A^t)\right]$ where $A^t=(a_1^t,\mathrm{\dots },a_n^t)$ is the joint action at time step $t$.

3.2 MAPPO and IPPO

Our implementation of PPO in multi-agent settings closely resembles the structure of PPO in single-agent settings by learning a policy ${\pi }_{\theta }$ and a value function $V_{\varphi }(s)$; these functions are represented as two separate neural networks. $V_{\varphi }(s)$ is used for variance reduction and is only utilized during training; hence, it can take as input extra global information not present in the agent’s local observation, allowing PPO in multi-agent domains to follow the CTDE structure. For clarity, we refer to PPO with centralized value function inputs as MAPPO (Multi-Agent PPO), and PPO with local inputs for both the policy and value function as IPPO (Independent PPO). We note that both MAPPO and IPPO operate in settings where agents share a common reward, as we focus only on cooperative settings.

3.3 Implementation Details

4.1 Testbeds, Baselines, and Common Experimental Setup

Testbed Environments: We evaluate the performance of MAPPO and IPPO on four cooperative benchmark – the multi-agent particle-world environment (MPE), the StarCraft multi-agent challenge (SMAC), the Hanabi challenge, and Google Research Football (GRF) – and compare these methods’ performance to popular off-policy algorithms which achieve state of the art results in each benchmark. Detailed descriptions of each testbed can be found in Appendix B.

Empirical Findings: In the majority of environments, PPO achieves results better or comparable to the off-policy methods with comparable sample efficiency.

4.2 MPE Testbed

Experimental Setting: We consider the three cooperative tasks proposed in [22]: the physical deception task (Spread), the simple reference task (Reference), and the cooperative communication task (Comm). As the MPE environment does not provide a global input, we follow [22] and concatenate all agents’ local observations to form a global state which is utilized by MAPPO and the off-policy methods. Furthermore, Comm is the only task without hom*ogenous agents; hence, we do not utilize parameter sharing for this task. All results are averaged over ten seeds.

Experimental Results: The performance of each algorithm at convergence is shown in Fig. 1. MAPPO achieves performance comparable and even superior to the off-policy baselines; we particularly see that MAPPO performs very similarly to QMix on all tasks and exceeds the performance of MADDPG in the Comm task, all while using a comparable number of environment steps. Despite not utilizing global information, IPPO also achieves similar or superior performance to centralized off-policy methods. Compared to MAPPO, IPPO converges to slightly lower final returns in several environments (Comm and Reference).

4.3 SMAC Testbed

Experimental Setting: We evaluate MAPPO with two different centralized value function inputs – labeled AS and FP – that combines agent-agnostic global information with agent-specific local information. These inputs are described fully in Section 5. All off-policy baselines utilize both the agent-agnostic global state and agent-specific local observations as input. Specifically, for agent $i$, the local Q-network (which computes actions at execution) takes in only the local agent-specific observation $o_i$ as input while the global mixer network takes in the agent-agnostic global state $s$ as input. For each random seed, we follow the evaluation metric proposed in [37]: we compute the win rate over 32 evaluation games after each training iteration and take the median of the final ten evaluation win-rates as the performance for each seed.

Experimental Results: We report the median win rates over six seeds in Table 1, which compares the PPO-based methods to QMix and RODE. Full results are deferred to Table 2 and Table 3 in Appendix. MAPPO, IPPO, and QMix are trained until convergence or reaching 10M environment steps. Results for RODE are obtained using the statistics from [37]. We observe that IPPO and MAPPO with both the AS and FP inputs achieve strong performance in the vast majority of SMAC maps. In particular, MAPPO and IPPO perform at least as well as QMix in most maps despite using the same number of samples. Comparing different value functions inputs, we observe that the performance of IPPO and MAPPO is highly similar, with the methods performing strongly in all but one map each. We also observe that MAPPO achieves performance comparable or superior to RODE’s in 10 of 14 maps while using the same number of training samples. With more samples, the performance of MAPPO and IPPO continue to improve and ultimately match or exceed RODE’s performance in nearly every map. As shown in Appendix D.1,MAPPO and IPPO perform comparably or superior to other other off-policy methods such as QPlex, CWQMix, and AIQMix in terms of both final performance and sample-efficiency.

Overall, MAPPO’s effectiveness in nearly every SMAC map suggests that simple PPO-based algorithms can be strong baselines in challenging MARL problems.

4.4 Google Football Testbed

Experimental Setting: We evaluate MAPPO in several GRF academy scenarios, namely 3v.1, counterattack (CA) easy and hard, corner, pass-shoot (PS), and run-pass-shoot (RPS). In these scenarios, a team of agents attempts to score a goal against scripted opponent player(s). As the agents’ local observations contain a full description of the environment state, there is no distinction between MAPPO and IPPO; for consistency, we label the results with PPO in Table 2 as “MAPPO”. We utilize GRF’s dense-reward setting in which all agents share a single reward which is the sum of individual agents’ dense rewards. We compute the success rate over 100 rollouts of the game and report the average success rate over the last 10 evaluations, averaged over 6 seeds.

Experimental Results: We compare MAPPO with QMix and several SOTA methods, including CDS, a method that augments the environment reward with an intrinsic reward, and TiKick, an algorithm which combines online RL fine-tuning and large-scale offline pre-training. All methods except TiKick are trained for 25M environment steps in all scenarios with the exception of CA (hard) and Corner, in which methods are trained for 50M environment steps.

We generally observe in Table 2 that MAPPO achieves comparable or superior performance to other off-policy methods in all settings, despite not utilizing an intrinsic reward as is done in CDS. Comparing MAPPO to QMix, we observe that MAPPO clearly outperforms QMix in each scenario, again while using the same number of training samples. MAPPO additionally outperforms TiKick on 4/5 scenarios, despite the fact that TiKick performs pre-training on a set of human expert data.

4.5 Hanabi Testbed

Experimental Setting: We evaluate MAPPO and IPPO in the full-scale Hanabi game with varying numbers of players (2-5 players). We compare MAPPO and IPPO to strong off-policy methods, namely Value Decomposition Networks (VDN) and Simplified Action Decoder (SAD), a Q-learning variant that has been successful in Hanabi. All methods do not utilize auxiliary tasks. Because each agent’s local observation does not contain information about the agent’s own cards²²2The local observations in Hanabi contain information about the other agent’s cards and game state., MAPPO utilizes a global-state that adds the agent’s own cards to the local observation as input to its value function. VDN agents take only the local observations as input. SAD agents take as input not only the local observation provided by the environment, but also the greedy actions of other players in the past time steps (which is not used by MAPPO and IPPO). Due to algorithmic restrictions, no additional global information is utilized by SAD and VDN during centralized training. We follow [15] and report the average returns across at-least 3 random seeds as well as the best score achieved by any seed. The returns are averaged over 10k games.

Experimental Results: The reported results for SAD and VDN are obtained from [15]. All methods are trained for at-most 10B environment steps. As demonstrated in Table 3, MAPPO is able to produce results comparable or superior to the best and average returns achieved by SAD and VDN in nearly every setting, while utilizing the same number of environment steps. This demonstrates that even in environments such as Hanabi which require reasoning over other players’ intents based on their actions, MAPPO can achieve strong performance, despite not explicitly modeling this intent.

IPPO’s performance is comparable with MAPPO’s in the 2-agent setting. However, as the agent number grows, MAPPO shows a clear margin of improvement over both IPPO and off-policy methods, which suggests that a centralized critic input can be crucial.

5.1 Value Normalization

Through the training process of MAPPO, value targets can drastically change due to differences in the realized returns, leading to instability in value learning. To mitigate this issue, we standardize the targets of the value function by using running estimates of the average and standard deviation of the value targets. Concretely, during value learning, the value network regresses to normalized target values. When computing the GAE, we use the running average to denormalize the output of the value network so that the value outputs are properly scaled. We find that using value normalization never hurts training and often improves the final performance of MAPPO significantly.

Empirical Analysis: We study the impact of value-normalization in the MPE spread environment and several SMAC environments - results are shown in Fig. 2. In Spread, where the episode returns range from below -200 to 0, value normalization is critical to strong performance. Value normalization also has positive impacts on several SMAC maps, either by improving final performance or by reducing the training variance.

Suggestion 1: Utilize value normalization to stabilize value learning.

5.2 Input Representation to Value Function

The fundamental difference between many multi-agent CTDE PG algorithms and fully decentralized PG methods is the input to the value network. Therefore, the representation of the value input becomes an important aspect of the overall algorithm. The assumption behind using centralized value functions is that observing the full global state can make value learning easier. An accurate value function further improves policy learning through variance reduction.

Past works have typically used two forms of global states. [22] use a concatenation of local observations (CL) global state which is formed by concatenating all local agent observations. While it can be used in most environments, the CL state dimensionality grows with the number of agents and can omit important global information which is unobserved by all agents; these factors can make value learning difficult. Other works, particularly those studying SMAC, utilize an Environment-Provided global state (EP) which contains general global information about the environment state [11]. However, the EP state typically contains information common to all agents and can omit important local agent-specific information. This is true in SMAC, as shown in Fig. 4.

To address the weaknesses of the CL and EP states, we allow the value function to leverage both global and local information by forming an Agent-Specific Global State (AS) which creates a global state for agent $i$ by concatenating the EP state and $o_i$, the local observation for agent $i$. This provides the value function with a more comprehensive description of the environment state. However, if there is overlap in information between $o_i$ and the EP global state, then the AS state will have redundant information which unnecessarily increases the input dimensionality to the value function. As shown in Fig. 4, this is the case in SMAC. To examine the impact of this increased dimensionality, we create a Featured-Pruned Agent-Specific Global State (FP) by removing repeated features in the AS state.

Emperical Analysis: We study the impact of these different value function inputs in SMAC, which is the only considered benchmark that provides different options for centralized value function inputs. The results in Fig. 3 demonstrate that using the CL state, which is much higher dimensional than the other global states, is ineffective, particularly in maps with many agents. In comparison, using the EP global state achieves stronger performance but notably achieves subpar performance in more difficult maps, likely due to the lack of important local information. The AS and FP global states both achieve strong performance, with the FP state outperforming AS states on only several maps. This demonstrates that state dimensionality, agent-specific features, and global information are all important in forming an effective global state. We note that using the FP state requires knowledge of which features overlap betweenthe EP state and the agents’ local observations, and evaluate MAPPO with this state to demonstrate that limiting the value function input dimensionality can further improve performance.

Suggestion 2: When available, include both local, agent-specific features and global features in the value function input. Also check that these features do not unnecessarily increase the input dimension.

5.3 Training Data Usage

An important feature of PPO is the use of importance sampling for off-policy corrections, allowing sample reuse.[14] suggest splitting a large batch of collected samples into mini-batches and training for multiple epochs. In single-agent continuous control domains, the common practice is to split a large batch into about 32 or 64 mini-batches and train for tens of epochs. However, we find that in multi-agent domains, MAPPO’s performance degrades when samples are re-used too often. Thus, we use 15 epochs for easy tasks, and 10 or 5 epochs for difficult tasks. We hypothesize that this pattern could be a consequence of non-stationarity in MARL: using fewer epochs per update limits the change in the agents’ policies, which could improve the stability of policy and value learning.Furthermore, similar to the suggestions by [17], we find that using more data to estimate gradients typically leads to improved practical performance. Thus, we split the training data into at-most two mini-batches and avoid mini-batching in the majority of situations.

Experimental Analysis: We study the effect of training epochs in SMAC maps in Fig. 5(a). We observe detrimental effects when training with large epoch numbers: when training with 15 epochs, MAPPO consistently learns a suboptimal policy, with particularly poor performance in the very difficult MMM2 and Corridor maps. In comparison, MAPPO performs well using 5 or 10 epochs.The performance of MAPPO is also highly sensitive to the number of mini-batches per training epoch. We consider three mini-batch values: 1, 2, and 4. A mini-batch of 4 indicates that we split the training data into 4 mini-batches to run gradient descent.Fig. 5(b) demonstrates that using more mini-batches negatively affects MAPPO’s performance: when using 4 mini-batches, MAPPO fails to solve any of the selected maps while using 1 mini-batch produces the best performance on 22/23 maps. As shown in Fig. 6, similar conclusions can be drawn in the MPE tasks. In Reference and Comm, the simplest MPE tasks, all chosen epoch and minibatch values result in the same final performance, and using 15 training epochs even leads to faster convergence.However, in the harder Spread task, we observe a similar trend to SMAC: fewer epochs and no mini-batch splitting produces the best results.

Suggestion 3: Use at most 10 training epochs on difficult environments and 15 training epochs on easy environments. Additionally, avoid splitting data into mini-batches.

5.4 PPO Clipping

Another core feature of PPO is the use of clipped importance ratio and value loss to prevent the policy and value functions from drastically changing between iterations. Clipping strength is controlled by the $ϵ$ hyperparameter: large $ϵ$ values allow for larger updates to the policy and value function.Similar to the number of training epochs, we hypothesize that policy and value clipping can limit the non-stationarity which is a result of the agents’ policies changing during training. For small $ϵ$, agents’ policies are likely to change less per update, which we posit improves overall learning stability at the potential expense of learning speed. In single-agent settings, a common $ϵ$ value is 0.2 [9, 1].

Experimental Analysis: We study the impact of PPO clipping strengths, controlled by the $ϵ$ hyperparameter, in SMAC (Fig. 7). Note that $ϵ$ is the same for both policy and value clipping. We generally that with small $ϵ$ terms such as 0.05, MAPPO’s learning speed is slowed in several maps, including hard maps such as MMM2 and 3s5z vs. 3s6z. However, final performance when using $ϵ=0.05$ is consistently high and the performance is more stable, as demonstrated by the smaller standard deviation in the training curves. We also observe that large $ϵ$ terms such as 0.2, 0.3, and 0.5, which allow for larger updates to the policy and value function per gradient step, often result in sub-optimal performance.

Suggestion 4: For the best PPO performance, maintain a clipping ratio $ϵ$ under 0.2; within this range, tune $ϵ$ as a trade-off between training stability and fast convergence.

5.5 PPO Batch Size

During training updates, PPO samples a batch of on-policy trajectories which are used to estimate the gradients for the policy and value function objectives. Since the number of mini-batches is fixed in our training (see Sec. 5.3), a larger batch generally will result in more accurate gradients, yielding better updates to the value functions and policies. However, the accumulation of the batch is constrained by the amount of available compute and memory: collecting a large set of trajectories requires extensive parallelism for efficiency and the batches need to be stored in GPU memory. Using an unnecessarily large batch-size can hence be wasteful in terms of required compute and sample-efficiency.

Experimental Analysis: The impact of various batch sizes on both final task performance and sample-efficiency is demonstrated in Fig. 8. We observe that in nearly all cases, there is a critical batch-size setting - when the batch-size is below this critical point, the final performance of MAPPO is poor, and further tuning the batch size produces the optimal final performance and sample-efficiency.However, continuing to increase the batch size may not result in improved final performance and in-fact can worsen sample-efficiency.

Suggestion 5: Utilize a large batch size to achieve best task performance with MAPPO. Then, tune the batch size to optimize for sample-efficiency.

Acknowledgments

This research is supported by NSFC (U20A20334, U19B2019 and M-0248), Tsinghua-Meituan Joint Institute for Digital Life, Tsinghua EE Independent Research Project, Beijing National Research Center for Information Science and Technology (BNRist), Beijing Innovation Center for Future Chips and 2030 Innovation Megaprojects of China (Programme on New Generation Artificial Intelligence) Grant No. 2021AAA0150000.

C.1 Implementation

All algorithms utilize parameter sharing - i.e., all agents share the same networks - in all environments except for the Comm scenario in the MPE. Furthermore, we tune the architecture and hyperparameters of MADDPG and QMix, and thus use different hyperparameters than the original implementations. However, we ensure that the performance of the algorithms in the baselines matches or exceeds the results reported in their original papers.

For each algorithm, certain hyperparameters are kept constant across all environments; these are listed in Tables 7 and 8 for MAPPO, QMix, and MADDPG, respectively. These values are obtained either from the PPO baselines implementation in the case of MAPPO, or from the original implementations for QMix and MADDPG. Note that since we use parameter sharing and combine all agents’ data, the actual batch-sizes will be larger with more agents.

In these tables, “recurrent data chunk length” refers to the length of chunks that a trajectory is split into before being used for training via BPTT (only applicable for RNN policies). “Max clipped value loss” refers to the value-clipping term in the value loss. “Gamma” refers to the discount factor, and “huber delta” specifies the delta parameter in the Huber loss function. “Epsilon” describes the starting and ending value of $ϵ$ for $ϵ$-greedy exploration, and “epsilon anneal time” refers to the number of environment steps over which $ϵ$ will be annealed from the starting to the ending value, in a linear manner. “Use feature normalization” refers to whether the feature normalization is applied to the network input.

C.2 Parameter Sharing

In the main results which are presented, we utilize parameter sharing - a technique which has been shown to be beneficial in a variety of state-of-the-art methods [5, 33] in all algorithms for a fair comparison. Specifically, both the policy and value network parameters are shared across all agents. In this appendix section, we include results which demonstrate the benefit of parameter sharing. Table 4 shows median evaluation win rate (with standard deviation in parantheses) on selected SMAC maps over 6 random seeds. MAPPO-Ind is MAPPO denotes MAPPO without parameter sharing - e.g., each agent has a separate policy and value function network. We observe that MAPPO with parameter sharing outperforms MAPPO without parameter sharing by a clear margin, supporting our decision to adopt parameter sharing in all PPO experiments and all baselines used in our results. A more theoretical analysis of the effect of parameter sharing can be found in [https://doi.org/10.48550/arxiv.2206.07505].

C.3 Death Masking

In SMAC, it is possible through the course of an episode for certain agents to become inactive, or “die” while other agents remain active in the environment. In this setting, while the local observation for a dead agent becomes all zeros except for the agent’s ID, the value-state still contains other nonzero features about the environment. When computing the GAE for an agent during training, it is unclear how to handle timesteps in which the agent is dead. We consider four options: (1) in which we replace the value state for a dead agent with a zero state containing the agent ID (similar to it’s local observation). We refer to this as “death masking”; (2) MAPPO without death masking, i.e., still using the nonzero global state as value input; (3) completely drop the transition samples after an agent dies (note that we still need to accumulate rewards after the agent dies to correctly estimate episode returns); and (4) replacing the global state with a pure zero-state which does not include the agent ID.Fig. 10 demonstrates that variant (1) significantly outperforms variants (2) and (3), and consistently achieves overall strong performance. Including the agent id in the death mask, as is done in variant (1), is particularly important in maps which agents may take on different roles, as demonstrated by the superior performance of variant (1) compared to variant (4), which does not contain the agent ID in the death-mask zero-state, in the 3s5z vs. 3s6z map.

Justification of Death MaskingLet ${\mathrm{𝟎}}_a$ be a zero vector with agent a’s agent ID appended to the end.The use of agent ID leads to an agent-specific value function depending on an agent’s type or role. It has been empirically justified that such an agent-specific feature is particularly helpful when the environment contains heterogeneous agents.

We now provide some intuition as to why using ${\mathrm{𝟎}}_{𝒂}$ as the critic input when agents are dead appears to be a better alternative to using the usual agent-specific global state as the input to the value function. Note that our global state to the value network has agent-specific information, such as available actions and relative distances to other agents. When an agent dies, these agent-specific features become zero, while the remaining agent-agnostic features remain nonzero - this leads to a drastic distribution shift in the critic input compared to states in which the agent is alive. In most SMAC maps, an agent is dead in only a small fraction of the timesteps in a batch (about 20%); due to their relative infrequency in the training data the states in which an agent is dead will likely have large value prediction error.Moreover, it is also possible that training on these out of distribution inputs harms the feature representation of the value network.

Although replacing the states at which an agent is dead with a fixed vector ${\mathrm{𝟎}}_{𝒂}$ also results in a distribution shift, the replacement results in there being only 1 vector which captures the state at which an agent is dead - thus, the critic is more likely to be able to fit the average post-death reward for agent $a$ to the input ${\mathrm{𝟎}}_{𝒂}$. Our ablation on the value function fitting error provide some weight to this hypothesis.

Another possible mechanism of handling agent deaths is to completely skip value learning in states in which an agent is dead, by essentially terminating an agent’s episode when it dies. Suppose the game episode is $T$ and the agent dies at timestep $d$. If we are not learning on dead state then, in order to correctly accumulate the episode return, we need to replace the reward $r_d$ at timestep $d$ by the total return $R_d$ at time $d$, i.e., $r_d\leftarrow R_d={\sum }_{t=d}^T{\gamma }^{t-d}{r}_t$. We would then need to compute the GAE only on those states in which the agent is alive. While this approach is theoretically correct (we are simply treating the state where the agent died as a terminal state and assigning the accumulated discounted reward as a terminal reward), it can have negative ramifications in the policy learning process, as outlined below.

The GAE is an exponentially weighted average of $k$-step returns intended to trade off between bias and variance. Large $k$ values result in a low bias, but high variance return estimate, whereas small $k$ values result in a high bias, low variance return estimate. However, since the entire post death return $R_d$ replaces the single timestep reward $r_d$ at timestep $d$, computing the 1-step return estimate at timestep $d$ essentially becomes a ($T-d$)-step estimate, eliminating potential benefits of value function truncation of the trajectory and potentially leading to higher variance. This potentially dampens the benefit that could come from using the GAE at the timesteps in which an agent is dead.

We analyze the impact of the death masking by comparing different ways of handling dead agents, including: (1) our death masking, (2) using global states without death masking and (3) ignoring dead states in value learning and in the GAE computation. We first examine the median win rate with these different options in Fig. 19 and 21. It is evident that our method of death masking, which uses ${\mathrm{𝟎}}_{𝒂}$ as the input to the critic when an agent is dead, results in superior performance compared to other options.

Fig. 22 also demonstrates that using the death mask results in a lower values loss in the vast majority of SMAC maps, demonstrating that the accuracy of the value predictions improve when using the death mask. While the arguments here are intuitive the clear experimental benefits suggest that theoretically characterizing the effect of this method would be valuable.

C.4 Hyperparameters

Tables 4-16 describe the common hyperparameters, hyperparameter grid search values, and chosen hyperparmeters for MAPPO, QMix, and MADDPG in all testing domains. Tables 9, 10, 11, and 12 describe common hyperparameters for different algorithms in each domain. Tables 13, 14, and 15 describe the hyperparameter grid search procedure for the MAPPO, QMix, and MADDPG algorithms, respectively. Lastly, Tables 16, 17, 18 and 19 describe the final chosen hyperparameters among fine-tuned parameters for different algorithms in MPE, SMAC, Hanabi, and GRF, respectively.

For MAPPO, “Batch Size” refers to the number of environment steps collected before updating the policy via gradient descent. Since agents do not share a policy only in the MPE speaker-listener, the batch size does not depend on the number of agents in the speaker-listener environment. “Mini-batch” refers to the number of mini-batches a batch of data is split into, “gain” refers to the weight initialization gain of the last network layer for the actor network. “Entropy coef” is the entropy coefficient $\sigma$ in the policy loss. “Tau” corresponds to the rate of the polyak average technique used to update the target networks, and if the target networks are not updated in a “soft” manner, the “hard interval” hyperparameter specifies the number of gradient updates which must elapse before the target network parameters are updated to equal the live network parameters. “Clip” refers to the $ϵ$ hyperparameter in the policy objective and value loss which controls the extent to which large policy and value function changes are penalized.

MLP network architectures are as follows:all MLP networks use “num fc” linear layers, whose dimensions are specified by the “fc layer dim” hyperparameter. When using MLP networks, “stacked frames” refers to the number of previous observations which are concatenated to form the network input: for instance, if “stacked frames” equals 1, then only the current observation is used as input, and if “stacked frames” is 2, then the current and previous observations are concatenated to form the input. For RNN networks, the network architecture is “num fc” fully connected linear layers of dimension “fc layer dim”, followed by “num GRU layers” GRU layers, finally followed by “num fc after” linear layers.

D.1 Additional SMAC Results

Results of all algorithms in all SMAC maps can be found in Tab. 5 and 6.

As MAPPO does not converge within 10M environment steps in the 3s5z vs. 3s6z map, Fig. 11 shows the performance of MAPPO in 3s5z vs. 3s6z when run until convergence. Fig. 12 presents the evaluation win of MAPPO with different value inputs (FP and AS), decentralized PPO (IPPO), QMix, and QMix with a modified global state input to the mixer network, which we call QMix (MG). Specifically, QMix(MG) uses a concatenation of the default environment global state, as well as all agents’ local observations, as the mixer network input.

Fig. 13 compares the results of MAPPO(FP) to various off-policy baselines, including QMix(MG), RODE, QPLEX, CWQMix, and AIQMix, in many SMAC maps. Both QMIX and RODE utilize both the agent-agnostic global state and agent-specific local observations as input. Specifically, for agent $i$, the local Q-network (which computes actions at execution) takes in only the local agent-specific observation $o_i$ as input while the global mixer network takes in the agent-agnostic global state $s$ as input. This is also the case for the other value-decomposition methods presented in Appendix Table 1 (QPLEX, CWQMix, and AIQMix).

D.2 Additional GRF Results

Fig. 14 compares the results of MAPPO to various baselines, including QMix, CDS, and TiKick, in 6 academy scenarios.