LLM-PySC2 - LLM StarCraft II Learning Environment

LLM-PySC2 is
NKU Robot Autonomy and Human-AI Collaboration Group and
NUDT Laboratory for Big Data and Decision's Python component of the StarCraft II LLM Decision Environment.
It wraps Deepmind's PySC2 Learning Environment API
in to a LLM energized Multi-Agent Decision Environment.
This is a collaboration between NKU and NUDT to develop StarCraft II into a rich environment for LLM research.
LLM-PySC2 provides an interface for LLM agents to interact with StarCraft 2,
getting textual/multimodal observations and dealing with textual actions.

We also make it possible for LLMs to make decisions in SMAC tasks, which make it possible to compare LLM with RL method.

Features

1. Support for more than 5 series of mainstream LLMs;
2. Macro decision-making and Micro operation available;
3. Multi-Agent(even heterogeneous agent) collaboration, point-to-point communication, domain communication;
4. Rich experimental tasks, consist of 24 llm_pysc2 experiments with 9 llm_smac tasks (more to be added);
5. Support for parallel experiments, multi-agent multi-thread concurrently query;
6. Diversified observational information, text observation, feature maps and StarCraft2 wiki data;
7. Complete StarCraft2 action space, with support for nearly all kinds of unit skills/abilities;
8. Automatic economic management in complete game;
9. High quality logger and data recorder.

About

If you use the LLM-PySC2 environment or LLM-SMAC tasks in your research,
please cite our github pager or
LLM StarCraft II Pre-Print Paper after preprint paper published.
You can also contact us by e-mail [email protected] or [email protected].

Arxiv paper: https://arxiv.org/abs/2411.05348 (Citation: https://ui.adsabs.harvard.edu/abs/2024arXiv241105348L/exportcitation)

Quick Start Guide

Get StarCraft II

LLM-PySC2 depends on the full StarCraft II game and only works with versions that
include the API, which is 3.16.1 and above.

Linux

Follow Blizzard's documentation to
get the linux version. By default, LLM-PySC2 expects the game to live in
~/StarCraftII/. You can override this path by setting the SC2PATH
environment variable or creating your own run_config.

Windows/MacOS

Install of the game as normal from Battle.net. Even the
Starter Edition will work.
If you used the default install location LLM-PySC2 should find the latest binary.
If you changed the install location, you might need to set the SC2PATH
environment variable with the correct location.

LLM-PySC2 should work on MacOS and Windows systems running Python 3.8+,
but has only been thoroughly tested on Linux. We welcome suggestions and patches
for better compatibility with other systems.

Get LLM-PySC2

download the LLM-PySC2 code from our github page LLM-PySC2.

use pip install to initialize the environment:

shell
$ conda create --name YOUR_ENV_NAME python==3.9
$ conda activate YOUR_ENV_NAME
$ pip install -e .

you can use mirrors like pip install -e . -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple to speed up downloading.

Get the maps

We have placed the required maps in the project folder:

text
llm_pysc2/maps/llm_pysc2
llm_pysc2/maps/llm_smac

You need to copy and paste these 2 folders into the Maps folder of the StarCraft2 program. Generally, the folder path is:

C:\Program Files (x86)\StarCraft II\Maps

and finally looks like:

text
C:\Program Files (x86)\StarCraft II\Maps\llm_pysc2
C:\Program Files (x86)\StarCraft II\Maps\llm_smac

If you used a custom path in installation, you may need to find the Map folder to finish the step.

Get llm api key

If you do not know how to get api_key, you can contact us to obtain a temporary gpt-3.5-turbo api_key with 2M tokens for free.

You need to write your api_key in ./llm_pysc2/agents/configs/config.ProtossAgentConfig before test the llm:

text
class ProtossAgentConfig(AgentConfig):
    def __init__(self):
        super(ProtossAgentConfig, self).__init__()
        self.race = 'protoss'
        self.model_name = 'gpt-3.5-turbo'
        self.api_base = 'YOUR_API_BASE'
        self.api_key = 'YOUR_API_KEY'
        ...

or set api_key like what we do in ./llm_pysc2/bin/experiment_llm_pysc2.py:

config.reset_llm(model_name, api_base, api_key)

if you do not have api_key but still want to test the environment,
you can set config.LLM_SIMULATION_TIME = 5 to simulate a 5-second response large model
and continue the tutorial below.

Test the environment

After specify your LLM api_key, api_base and model_name, you can run our experiments to test LLM and
both the llm_pysc2 tasks and llm-smac tasks:

shell
$ python -m llm_pysc2.bin.experiment_llm_pysc2
$ python -m llm_pysc2.bin.experiment_llm_smac

These two script will load gpt-3.5 energized agents and use pure text observation to make decisions.
If you want to use multimodal LLMs like gpt-4v, you can set config.ENABLE_IMAGE_RGB = True to
activate image observations.

Also, you can use --parallel parameter (or edit files in ./llm_pysc2/bin) to run several games at the same time:

shell
$ python -m pysc2.bin.agent --map pvz_task4_level1 --agent_race protoss --parallel 2 --agent llm_pysc2.bin.experiment_llm_pysc2.MainAgentLLMPysc2
$ python -m pysc2.bin.agent --map pvz_task4_level1 --agent_race protoss --parallel 4 --agent llm_pysc2.bin.experiment_llm_pysc2.MainAgentLLMPysc2

which may significantly improve experimental efficiency.

Experiments

We provided two series of experiment tasks:

text
(1) llm_pysc2 experiments: a series of pvz combat, with 8 different settings and 3 levels of difficult for each.
(2) llm_smac experiments: same as original SMAC tasks, units control by LLM agent instead of RL agent.

You can run these experiments in ./llm_pysc2/bin/llm_pysc2 and ./llm_pysc2/bin/llm_smac.

llm_smac experiments

llm_smac experiments are original smac tasks, use the same map and setting of smac tasks. Consider that
the control of the Zerg and Terran is still ongoing, we will add more smac tasks in the future
(executable files in ./llm_pysc2/bin/llm_smac):

text
2s3z                3s5z                1c3s5z              
3s_vs_3z            3s_vs_4z            3s_vs_5z            
2c_vs_64zg          2s_vs_1sc           3s5z_vs_3s6z        
(more to be added in future version)

llm_pysc2 experiments

llm_pysc2 experiments contains 6 pvz combat with 3 level of difficulties, concentrate more attention to larger
scenarios and the use of unit skills, provide higher complexity and operability:

Each of them has three different difficulties. Simply, level-1 can be used for demo and method debug;
level-2 is a standard difficulty, can be used for policy training, and level-3 can be used as an experimental field
for trained method evaluating, which is quite difficult, and can be served as a good OOD evaluating environment.

More details can be seen in ./docs/llm_pysc2/experiments.md.

Customize your Agent/Tasks

Customize LLM Interaction Process

If you want to redefine a SubAgent's interaction process with the large model, you can redefine the query function of
a SubAgent. View relevant code of class Customized_LLMAgent(LLMAgent) in ./llm_pysc2/agents/llm_pysc2_agent.py:

text
class Customized_LLMAgent(LLMAgent)
    def query(self, obs) -> None:

Customize MainAgent

Main agent is used to interact with pysc2 and does not directly make decisions, so it is not recommended to modify it.
You can affect the main agent by modifying the config just like what we do in.

text
llm_pysc2/bin/experiment_llm_pysc2.py
llm_pysc2/bin/experiment_llm_smac.py

Customize SubAgents

MainAgent is only an objects used for scheduling cameras, managing internal data and interacting with the env.
While SubAgents(of the MainAgent) query llm to obtain text actions, plays the role of decision maker.

SubAgent is configured in configurations(llm_pysc2/agents/configs).
For example, ./llm_pysc2/agents/configs/llm_smac/config_2s3z.py defines a MainAgent with only one sub agent named
CombatGroupSmac:

text
class ConfigSmac_2s3z(ProtossAgentConfig):

  def __init__(self):
    super(ConfigSmac_2s3z, self).__init__()
    ...
    self.AGENTS = {
      'CombatGroupSmac': {
        'describe': "Protoss military commander, controls units to fight against enemy. ",
        'llm': {...},
        'team': [
          {'name': 'Zealot-1', 'unit_type': [units.Protoss.Zealot],
           'game_group': 1, 'select_type': 'group'},
          {'name': 'Zealot-2', 'unit_type': [units.Protoss.Zealot],
           'game_group': 2, 'select_type': 'group'},
          {'name': 'Stalker-1', 'unit_type': [units.Protoss.Stalker],
           'game_group': 4, 'select_type': 'group'},
        ],
        'action': {
            units.Protoss.Zealot: PROTOSS_BASIC_ACTION_SMAC,
            units.Protoss.Stalker: PROTOSS_BASIC_ACTION_SMAC,
        },
      },
    }
    ...

you can add more sub agent in your configuration if needed.

Customize SubAgent's UnitTeams

You can design agent's unit team in config.
In the example above, we showed a sub agent with 3 teams called 'Zealot-1', 'Zealot-2' and 'Stalker-1'.
You can redefine your agent teams if needed.

A team should consist of:

text
'name': any str
'unit_type': list, of pysc2 unit enum
'game_group': int, -1 to 9, -1 refers to do not add units to any game_group
'select_type': str, 'group' for group recall, 'select' for single select(mouse left click), 'select_all_type' for select screen units of same types(mouse double click)

More examples can be viewed in ./llm_pysc2/agents/configs.

Customize UnitTeam's action space

You can design team's action space in config.
In the example(ConfigSmac_2s3z) above, we showed a action space of CombatGroupSmac.
Where PROTOSS_BASIC_ACTION_SMAC shape as:

[{'name': 'Attack_Unit', 'arg': ['tag'], 'func': [(12, F.Attack_screen, ('queued', 'screen_tag'))]},]

and PROTOSS_BASIC_ACTION_SMAC2 shape as:

text
[{'name': 'Attack_Unit', 'arg': ['tag'], 'func': [(12, F.Attack_screen, ('queued', 'screen_tag'))]},
 {'name': 'Move_Screen', 'arg': ['screen'], 'func': [(331, F.Move_screen, ('queued', 'screen'))]},]

Each action should consist of three parts:

text
'name': any str
'arg': list, either of [] / ['tag'] / ['screen'] / ['minimap'] / ['tag', 'screen'] /  ['tag', 'minimap']
'func': a list of triplet (pysc2_func_id, pysc2_func, args_type)

More examples can be viewed in ./llm_pysc2/agents/configs and ./llm_pysc2/lib/llm_actions.

Customize Observation Wrapper

You can design your own Observation Wrapper by redefining o_translator in ./llm_pysc2/lib/llm_observations.py.

Customize Action Recognizer

You can design your own Text Action Recognizer by redefining a_translator in ./llm_pysc2/lib/llm_actions.py.

Customize LLM Client

You can design your own LLM Client in llm_pysc2/lib/llm_client if needed.

Future Work

We have planed to try our best to add the following features before 2025/2/1:

text
(1) Support of Zerg control
(2) Support of Terran control
(3) Compatibility of classic RL algorithms

And more features before 2025/5/1:

text
(1) Full game experiments series1: in map Simple64, Simple96, Simple128
(2) Full game experiments series2: in map Ancient Cistern LE, Babylon LE, Gresvan LE

Note that this LLM-PySC2 is a preview version, and official version with far more features and better stability
will be open-sourced before 2025/7/1.

Contributors

Zongyuan Li (Main Contributor, Nankai University):
Framework design. Multi-Agent structure. Text observation generation. Text action recognition.
Communication. Experiments design. Prompt. Logger. Documents. Organize.

Runnan Qi, Yanan Ni, Lumin Jiang (National University of Defense Technology):
LLM Client. Multimodal LLM Client. Image observation generation. Documents.

Chang Lu, Xiaojie Xu, Pengfei Li, Yunzheng Guo, Zhe Ma (Nankai University):
Data recorder. Experiments evaluation. Game knowledge.

Kuihua Huang (National University of Defense Technology), Xian Guo, Xuebo Zhang(Nankai University):
Organize.

Thanks

LLM-PYSC2: STARCRAFT II LEARNING ENVIRONMENT FOR
LARGE LANGUAGE MODELS ∗†
Zongyuan Li1
, Yanan Ni2
, Runnan Qi2
, Lumin Jiang2
, Chang Lu1
, Xiaojie Xu1
,
Xiangbei Liu1
, Pengfei Li1
, Yunzheng Guo1
, Zhe Ma1
,
Xian Guo1,∗
, Kuihua Huang2,∗
, Xuebo Zhang1,∗
1 College of Artificial Intelligence, Nankai University
2 Laboratory for Big Data and Decision, National University of Defense Technology
ABSTRACT
This paper introduces a new environment LLM-PySC2 (the Large Language Model StarCraft II
Learning Environment), a platform derived from DeepMind’s StarCraft II Learning Environment
that serves to develop Large Language Models (LLMs) based decision-making methodologies. This
environment is the first to offer the complete StarCraft II action space, multi-modal observation
interfaces, and a structured game knowledge database, which are seamlessly connected with various
LLMs to facilitate the research of LLMs-based decision-making. To further support multi-agent
research, we developed an LLM collaborative framework that supports multi-agent concurrent queries
and multi-agent communication. In our experiments, the LLM-PySC2 environment is adapted to
be compatible with the StarCraft Multi-Agent Challenge (SMAC) task group and provided eight
new scenarios focused on macro-decision abilities. We evaluated nine mainstream LLMs in the
experiments, and results show that sufficient parameters are necessary for LLMs to make decisions,
but improving reasoning ability does not directly lead to better decision-making outcomes. Our
findings further indicate the importance of enabling large models to learn autonomously in the
deployment environment through parameter training or train-free learning techniques. Ultimately,
we expect that the LLM-PySC2 environment can promote research on learning methods for LLMs,
helping LLM-based methods better adapt to task scenarios.
1 Introduction
In 2017, the StarCraft II Learning Environment (SC2LE)[1] was developed by DeepMind and Blizzard Entertainment.
It is the first environment that enables various reinforcement learning (RL) agents to compete with each other in the
StarCraft II game, and promoted the emergence of decision-making methods such as QMix[2], Weighted QMIX[3],
MAPPO[4], and the household name AlphaStar[5]. However, RL-trained agents typically require a substantial amount
of data and prolonged interactions, but still lack generalization capabilities in most scenarios due to the task-relevant
reward function. Consequently, it is urgent to develop new decision-making methods at the present.
Additionally, impressive research efforts such as Stanford Town[6], LLM plays MineCraft[7] and the game of
Diplomacy[8] have demonstrated great potential in LLM-based decision-making in recent years. Considering that
large models exhibit greater interactivity, interpretability, and reasoning capabilities, it is quite natural to apply large
models in complex decision-making environments. However, there is no sufficiently comprehensive platform to support
research on LLM decision-making methods in complex environments. Notably, the mainstream platform SC2LE
environment does not yet support research on decision-making with large models.
In order to leverage the advantages of large models and circumvent the disadvantages of RL, researchers developed the
SC2 module into TextStarCraft II (TSC2)[9], enabling LLMs to interact with the StarCraft II environment for the first
time. However, there are some restrictions in the environment. The LLM based agent can not use micro-operations and
∗Corresponding author.
†Code is available at https://github.com/NKAI-Decision-Team/LLM-PySC2
arXiv:2411.05348v1 [cs.AI] 8 Nov 2024
Zongyuan Li et. al.
unit skills to defeat enemy units due to the scale-cropped discrete action space. While observation only contains unit
counts and upgrade status that are not sufficient for the implementation of complex strategies. What’s more, multi-agent
collaboration is not available because TSC2 is a single-agent framework.
To address these issues, we developed LLM-PySC2, an environment derived from SC2LE, based on PySC2 module.
This environment provides agents with comprehensive observations, including global information and agent-specific
local combat information (in text form or multimodal form) and a structured game knowledge database. We also
expanded the action space to the full StarCraft II action space, enabling agents to perform fine-grained operations and
unit skills. To support multi-agent research, we built a multi-agent framework with a communication system that allows
both point-to-point and domain communication.
In experiments, eight new scenarios were proposed. Unlike the SMAC[10] tasks, these tasks emphasize not only
micro-operations but also task understanding and macro-decision abilities. We tested nine mainstream LLMs in both the
SMAC tasks and the new proposed scenarios. The results indicate that pre-trained LLMs have possess decision-making
ability but lack the ability to make consistently effective decisions. Pre-trained LLMs without task-specific training may
be unable to analyze the key elements for achieving victory. They often fail to identify the important part of the game
knowledge for the most times, making mistakes in analysis or even dealing damage to allies sometimes.
In summary, there remains much to be done to raise the ability of LLMs in the domain of multi-agent decision making.
We hope that the LLM-PySC2 environment will advance research on LLM learning techniques, helping LLM-based
methods better adapt to task scenarios.
2 Related Works
2.1 Starcraft II
Starcraft II is a classic platform for evaluating algorithms. Specially, as a real-time strategy game, StarCraft II features a
high-dimensional partially observable state space and a huge continuous action space. With three species and more than
120 types of units, it is widely regarded as one of the most complex and challenging environments and is commonly
used for evaluating advanced decision-making methods.
To support the research of learning methods, DeepMind and Blizzard Entertainment developed SC2LE, a comprehensive
environment for RL research. This environment is designed to improve research in learning algorithms within complex
strategy games. It provides RL interfaces such as the observation, actions, and reward function, considered as one of the
most significant environments in the field of artificial intelligence.
Consequently, after the introduction of SC2LE, more and more StarCraft II environments have emerged. Among
those environments, SMAC[10] and PyMARL are the most famous. SMAC is a benchmark comprising 23 tasks
specifically designed for multi-agent RL, mostly focusing on distributed multi-agent decision-making. To evaluate
MARL algorithms, the SMAC team also developed PyMARL as their training platform. In the PyMARL framework,
over five algorithms are integrated, and the framework is gradually expanded into a multi-environment available RL
platform.
Overall, their work effectively advanced the research on multi-agent learning methods, made significant contributions
to the field of intelligent decision-making, and motivated us to develop an environment for LLM-based methods.
2.2 LLM Decision-Making and Text StarCraft II
In recent years, the decision-making ability of LLMs has started to attract attention. In 2023, an LLM-based agent
called the Ghost in Minecraft achieved 67.5% success in Minecraft’s diamond challenge. After that, Agent-Pro[11], an
LLM agent capable of using strategies like bluffing in Poker, was developed. Additionally, researchers deployed LLM
agents in Werewolf[12], a game with deception and counter-deception through communication, and developed LLM
agents in the game of Diplomacy, a game of collaboration and competition.
These works inspire researchers to develop LLM-based decision-making methods in games. As one of the most
famous real-time strategy games, StarCraft II was first developed into an LLM-interactable environment called TSC2.
This environment enables LLMs to make macro-decisions in StarCraft II and proves that LLMs can make decisions
and defeat build-in bots at level-4 in StarCraft-II. However, TSC2 does not support micro-operations on units and
multi-agent collaboration and faces limitations in observation and action space.
Under these circumstances, we constructed the LLM-PySC2 environment, aiming to solve these problems and provide
a new StarCraft-II environment. We also make our environment compatible with SMAC tasks, facilitating comparisons
with algorithms developed in the StarCraft environment.
2
Zongyuan Li et. al.
3 LLM-PySC2 environment
3.1 Framework
The LLM-PySC2 environment is built on the PySC2 module’s agent level. In Figure 1, the MainAgent plays the role of
controlling the camera, selecting units, collecting observations, and executing actions, while the LLM agent plays the
role of the actual decision maker that observes game situations, analyzes, and gives actions. Each LLM agent connects
to an LLM, getting a text or multimodal observations from a wrapper, querying the LLM in an independent thread, and
finally getting game analysis and actions.
LLMLLM-PySC2
MainAgent
pysc2.lib.actions.Functions.xx
obs
image
obs
agent info
relevant data
latest communication data
Starcraft2(PySC2)
obs1
obs2
obs3
obs4
analysis: xxcommu: xxactions: xx
systemprompt
example prompt
observed infoCommunication:
<MessageTo(Commander, ‘’‘xxxx’‘’)>
<MessageTo(Channel-1, ‘’‘xxxx’‘’)>
obs5
LLM-PySC2 MainAgent Opponent Agent
Build-inbot
RLAgent
AI Arenacamera
Actions:
Team Stalker-1:
<AttackUnit(0x1000a2001)>
<MoveScreen([54, 32])>
𝑡�� ��
𝑡�� ��
LLMAgent
Action Recognizer
��
�� ��
’ ��
’
𝑖� ��
Obs Wrapper
Game Wiki Data
Communication data
LLM Agent1
LLM Agent2
LLM Agent3
LLMLLMFigure 1: LLM-PySC2 framwork. In LLM-PySC2, the original PySC2 observation will transform into a text-form
observation. LLM-generated text action can be recognized and transformed into PySC2 action functions, enabling
LLMs to interact with the StarCraft II environment.
3.1.1 Interact with environment
An interaction step consists of two phases: auxiliary management and decision-making (and it consists of many game
steps). In the auxiliary management phase, no LLM will be involved. The MainAgent will control the PySC2 camera
and finish works like grouping newly trained units and managing idle workers to avoid excessive involvement of large
models in simple and repetitive labour.
Observations for each agent’s unit teams will be collected in the decision-making phase. After all teams’ observations
are collected, the agents use the Observation Wrapper to translate the structured observations into a text observation.
Then, all agents query remote or local LLMs concurrently, waiting until all the agents get the response.
After all agents get the response, they will use the Action Recognizer to detect valid actions and translate the text actions
into a structured form. Then, the MainAgent moves the camera to the same position when collecting observations and
executes each agent’s stored actions. After executing all the actions, the LLM-PySC2 environment will enter the next
interaction step and repeat the work mentioned above.
3.1.2 Multi-agent communication
Considering that LLMs have inherent advantages in interaction, we designed a communication system for the multiagent framework. In the communication system, agents communicate with each other using ’communication actions’, a
kind of text action similar to unit control actions shown in Figure 1.
3
Zongyuan Li et. al.
In the communication system. An LLM agent can send a message to another agent or send information to a channel.
If the message is sent to an agent, only the designated receiver can get the information. If the message is sent to a
channel, all agents that listen to the channel share the information. Through these communication actions, multi-agent
collaboration frameworks such as centralized decision-making and distributed decision-making can be easily built.
3.2 Observation
Observation is indispensable for decision making. Different types of information are necessary for agents with different
tasks. Roughly, we categorize observational information into two types: local observations for micro-level operations
and global observations for macro-level decision-making. These observations can be divided into text and image
observations according to form.
Valid Communicate Target: Agent XXX: ......
Team XXX: ...... Team XXX: ...... Query Message
System Prompt
Example Input
Example Output
Text Observation
Task Description
Game Time
Team unit info
Game Knowledge
Valid Actions
Valid Commu Target
Communication
Game Time: 00:50
Text Observation
Team XXX Info: ...... Team XXX Info: ......
Team Stalker-1 Info: Minimap Pos: ...... Controlled Units: ...... Nearby Ally Units: ...... Nearby Enemy Units: ...... Relevant Knowledge: Protoss.Stalker: ...... Zerg.Queen: ...... Zerg.Drone: ......
Valid Actions: Team Stalker-1: <Stop()>
<No_Operation()>
<Hold_Position()>
<Move_Minimap(minimap)>
<Move_Screen(screen)>
<Attack_Unit(tag)> Communication: From Agent XXX: ...... Tasks: Team Adept-1' task: Kill as much as enemy workers as possible.
Figure 2: Text observation for micro-operation LLM. Text observation is a part of the query message. It contains
many paragraphs, including team unit info, relevant game Knowledge, and valid actions. Semantic information is added
when the observation wrapper processes the original obs object.
3.2.1 Text Observation
Observation Wrapper for micro-operations This wrapper focuses on local observations. It provides detailed
information of controlled unit, nearby ally and nearby enemy unit for an agent. It extracts unit information from PySC2
obs object and the relevant game knowledge from the knowledge base. As shown in Figure 2 The text observation
generated by the wrapper includes game time, unit information, unit knowledge, valid actions, short-term memory,
communication data, and task descriptions. Agents using the wrapper are designed for micro-operations like fighting
with enemy units or constructing buildings in a specific position.
Observation Wrapper for macro-decisions This wrapper focuses on global observations. It provides deployment
information, unit counts, and upgrade status that are similar to the text observation of the TSC2 environment. For the
agent responsible for military deployment, text observation generated from the wrapper is used for supporting overall
strategy. For the agent responsible for development, the generated global observation will make the agent aware of the
current economy and technology situation, supporting the planning of future works of development.
3.2.2 Image Observation
In the complex environment of StarCraft II, relying solely on textual observations may prevent agents from fully
comprehending the battlefield dynamics. To enhance situational awareness, the LLM-PySC2 environment provides
multimodal observation. This feature enables multimodal large models to integrate visual information, leading to a
more accurate understanding of the situation. Figure 3 highlights two primary types of image observations: game image
observation and feature map observation. These visual inputs provide the agent with critical battlefield information,
facilitating tactical analysis and strategy development.
4
Zongyuan Li et. al.
Figure 3: Image observations. LLM-PySC2 directly extracts PySC2’s image observation, including the game image
and the feature map. It enhances these images by incorporating auxiliary lines, which assist LLMs in accurately
determining the coordinates of various positions. These images, interpretable by multimodal LLMs, provide decisionmakers with complex information, such as terrain layout and unit distribution on the map.
Figure 3 demonstrates the game image and feature map, which are directly extracted from the PySC2 interface. These
images are enhanced with auxiliary lines to provide coordinate information for large models. This approach enables
the agent to accurately perceive crucial battlefield elements, such as unit count and distribution, while also conveying
information that is challenging to express through text, such as terrain features and relative spatial relationships.
3.3 Action
In decision-making environments, the concept of "action" is pivotal to enable interactions between the agent and the
environment. In our framework, LLMs engage with the environment through text-based actions, which must adhere to a
specific format to be recognized and converted into PySC2 action functions. The process of processing text action into
PySC2 functions can be seen in Figure 4.
LLM Response
Analysis:
Team Adept-1 is already attacking a nearby Drone and
should continue this focus, while Team AdeptPhase-1 can
target another Drone nearby for maximum efficiency. Both
teams should remain aggressive to disrupt the enemy's
economy. Actions:
Team Adept-1:
<Attack_Unit(0x101340001)>
<Move_Screen([62, 64])>
Team AdeptPhase-1:
<Attack_Unit(0x1012c0001)>
<Move_Screen([66, 61])>
Team Adept-1:
<Attack_Unit(0x101340001)>
<Move_Screen([62, 64])>
Team AdeptPhase-1:
<Attack_Unit(0x1012c0001)>
<Move_Screen([66, 61])>
Text Action Recognize
pysc2.lib.actions.Attack_screen('now', [40, 47])
pysc2.lib.actions.Move_screen('queued', [62, 64])
pysc2.lib.actions.Attack_screen('now', [48, 51])
pysc2.lib.actions.Move_screen('queued', [66, 61])
0x101340001
0x1012c0001
unit tag unit positionGenerate Action Functions
Figure 4: Text action recognition. The default action recognizer recognizes text actions by searching the "Actions"
part in LLM’s response, extracting action names and arguments, searching for corresponding PySC2 functions, and
generating the callback form of the function.
Text Actions These actions are expressed in a syntax that is intuitive and descriptive, allowing the LLM to comprehend
the intended operation without additional context. A standard text action is encapsulated in angle brackets and several
arguments, shaped as <ActionName()>, <ActionName(arg0)>, or <ActionName(arg0, arg1)>. The arguments can
represent various elements, such as a unit tag, a screen coordinate, or a minimap position, allowing these actions to
encompass the complete continuous action space of PySC2.
5
Zongyuan Li et. al.
In the decision-making phase, LLM will be informed of currently available actions, such as <Attack_Unit(tag)>,
<Move_Screen(screen)> and <Select_Unit_Attack_Unit(tag, tag)>. The LLM can generate actions like <Attack_Unit(0x100030001)> or <Move_Screen([23, 37])> according to observed information and its purpose. If LLM
generated multiple text actions, the first action will be executed immediately, and the remaining actions will be added to
the action sequence waiting for execution.
Action Space All kinds of actions in PySC2 are available in our environment, however, each agent does not have to
face all the actions of its race. In our environment, the action space is agent-specific, allowing each agent to define a
unique set of actions. For the agent that controls units such as Stalkers, the action space consists of text actions like
<Stop()>, <No_Op()>, <Move_Screen(screen)>, <Move_Minimap(minimap)>, <Attack_Unit(tag)>, and do not consist
of actions like training units or research.
4 Experiments
4.1 Experiment Scenarios
To facilitate research in LLM-based decision-making, we have provided two sets of experiments: LLM-SMAC tasks and
LLM-PySC2 tasks. The LLM-SMAC tasks are the same as standard SMAC experiments, which serve as an excellent
bridge for comparing with RL-based methods. LLM-PySC2 tasks are new scenarios, which, compared to the SMAC
tasks designed specifically for micro-operations, place more emphasis on the large model’s ability to understand the
task scenario and make macro-level decisions.
4.1.1 LLM-SMAC tasks
LLM-SMAC tasks share the same settings as the original SMAC tasks. These tasks initialize units for both sides
and automatically raise attacks for enemy units. In these scenarios, the key to victory lies in concentrating firepower,
controlling combat distance, and, sometimes, in interaction frequency. They are good scenarios for comparing the
training data efficiency with RL-based methods but not good scenarios for utilizing the multitasking and macro
decision-making capabilities of LLMs.
4.1.2 LLM-PySC2 tasks
LLM-PySC2 tasks are the newly proposed experiment scenarios, a task group that tests agents’ situation analysis
capabilities, planning abilities, the application of knowledge, communication, and collaboration. Some of the tasks are
shown in Figure 5
(d) Task4: Mid scale combat (e) Task5: Large scale combat (type1)
(a) Task1: 2 Adept harass (b) Task2: 3 Pheonix harass (b) Task3: Intercept enemyairdrops
(f) Task6: Large scale combat (type2)
Figure 5: LLM-PySC task group. The LLM-PySC task group contains eight tasks, with three difficulties for each
task. Compared to SMAC tasks, they place more emphasis on macro decision-making, situation analysis, and skill
use. These scenarios are common in professional competitions. Winning these small tasks is beneficial for winning
complete games in the future.
6
Zongyuan Li et. al.
In these tasks, LLMs need to plan an infiltration route into the enemy base and kill enemy workers, or use unit skills to
implement specific tactics in a battle. In addition, these tasks are more suitable for researching multi-agent collaboration
methods, and implementing centralized or distributed decision-making for LLMs.
There are eight tasks in the LLM-PySC2 task group. Half of the task scenarios are single-agent decision-making
scenarios (from tasks 1 to 4), where one LLM agent controls multiple units, while the other half (from tasks 5 to 8)
tests the cooperation between agents, with multiple agents controlling multiple units with different tactical roles. In
LLM-PySC2 task group, image observation and multi-agent communication are available and can be easily disabled if
needed.
To avoid the situation where methods in SMAC can always reach the 100% winning rate of most tasks, we set three
different difficulty levels for our experiment group. From level 1 to level 3, the forces of the enemy gradually increase.
At a higher level, more units or upgrades will be added to the enemy side, ensuring these tasks can still be effective
even after the LLM-based decision-making methods have been well developed.
4.2 Experiment Results
To facilitate subsequent research, we tested the decision-making ability of various large models. All experiments were
conducted in StarCraft II of Version 5.0.13 (92440), LLM-PySC2 v0.1. We recorded a ratio of resources of the killed
unit over the dead unit (K/D rate) and the winning rate (WR, i.e. task completion rate). The combination of K/D rate
and WR reflects the performance of LLM in decision-making scenarios.
In the LLM-PySC2 environment, we provide series of LLMs, such as GPT-3, GPT-4, GPT-o1, GLM-4, Claude-3,
Llama-3.1. We tested some representative models among them, tested the performance of models with different
reasoning abilities in decision-making tasks (GPT-3.5, GPT-4o-mini, GPT-4o), and tested the performance of models
with different parameters based on the same architecture (Llama3.1-8b, Llama3.1-70b, Llama3.1-405b).
All experiments use the default configuration of the open-sourced codes. As a benchmark, we do not specially design
prompts to promote decision quality or instruct the LLMs to obtain victories, and all the LLMs are not fine-tuned in the
LLM-PySC2 environment. Results show that large models can make decisions and generate text actions in the correct
form. However, when the task is complex enough or requires a lot of micro-operations, large models may not perform
well, suggesting that training or other technical methods are necessary for improving their decision quality.
4.2.1 Experiment Results in LLM-SMAC tasks
In the LLM-SMAC tasks, we conducted 20 repeated experiments for 6 LLMs in each scenario. For scenarios where
decisions were made by GPT-3.5-turbo, we raised the number to 50 due to its good concurrency support and friendly
cost. In these experiments, all large models used textual observations. This setting is completely sufficient for scenarios
other than 2c_vs_64zg, as they basically did not need to utilize terrain information. Results are shown in Table 1.
Table 1: Kill/Death Rates and Winning Rates of LLMs in LLM-SMAC Tasks.
Model Name 2s3z 3s5z 1c3s5z 3s5z_vs_3s6z 2s_vs_1sc 2c_vs_64zg 3s_vs_3z
Gpt-3.5-turbo 0.60 (22%) 0.43 (4%) 0.91 (44%) 0.29 (0%) 0.01 (2%) 0.52 (0%) 0.05 (0%)
Gpt-4o-mini 0.66 (20%) 0.39 (0%) 1.01 (50%) 0.29 (0%) 0.00 (0%) 0.54(0%) 0.09 (0%)
Gpt-4o 0.76 (20%) 0.47 (0%) 0.80 (30%) 0.35 (0%) 0.00 (0%) 0.56 (0%) 0.15 (0%)
Claude3-haiku 0.58 (5%) 0.48 (0%) 0.48 (0%) 0.32 (0%) 0.00 (0%) 0.52 (0%) 0.10 (0%)
Llama3.1-8b 0.19 (0%) 0.23 (0%) 0.18 (0%) 0.14 (0%) 0.00 (0%) 0.49 (0%) 0.00 (0%)
Glm-4-plus 0.81(25%) 0.46 (0%) 0.47 (0%) 0.33 (0%) 0.00 (0%) 0.54 (5%) 0.15 (0%)
We found that, although large models can analyze the observation information and output actions in the correct form,
they performed poorly in SMAC tasks. On the one hand, due to LLM hallucinations and the lack of task-specific
knowledge, they can not deduce the principle that concentrated fire is the key to victory. On the other hand, even if
the observation provided the knowledge that Zealots have a higher attack efficiency than Stalkers, the large models
sometimes still chose to attack the enemy Stalkers first in tasks like 2s3z and 3s5z.
7
Zongyuan Li et. al.
4.2.2 Experiment Results in LLM-PySC2 tasks
The same as the experiments in SMAC, we conducted 20 repeated experiments for each large model in each scenario
and 50 for GPT-3.5-turbo. Considering that all models cannot complete the multi-line attack in Task 8, we only listed
the data from Task 1 to Task 7. Results are shown in Table 2 and 3.
Table 2: Kill/Death Rates and Winning Rates of Gpt-3.5-turbo in LLM-PySC2 Tasks (level-1/2/3).
Task level task1 task2 task3 task4 task5 task6 task7
task-level-1 1.23 (58%) 0.13 (4%) 6.63 (38%) 0.38 (0%) 0.61 (8%) 0.28 (0%) 1.29 (72%)
task-level-2 0.56 (5%) 0.04 (0%) 3.31 (5%) 0.34 (0%) 0.52 (0%) 0.20 (0%) 0.98 (25%)
task-level-3 0.39 (0%) 0.05 (0%) 1.99 (0%) 0.31 (0%) 0.40 (0%) 0.26 (0%) 0.62 (0%)
In table 2, we tested Gpt-3.5-turbo’s performance in all the levels of each task. These data can serve as benchmark values
for future research. These three levels of difficulty not only serve as validation scenarios for developed decision-making
methods in the future but can also be applied to out-of-distribution (OOD) tasks, such as training on level 2 and
validating on level 3.
Table 3: Kill/Death Rates and Winning Rates of LLMs in LLM-PySC2 Tasks (level-1).
Model Name task1 task2 task3 task4 task5 task6 task7
Gpt-3.5-turbo 1.23 (58%) 0.13 (4%) 6.63 (38%) 0.38 (0%) 0.61 (8%) 0.28 (0%) 1.29 (72%)
Gpt-4o-mini 1.67 (70%) 0.16 (0%) 3.46 (0%) 0.39 (0%) 0.62 (20%) 0.30 (0%) 1.02 (40%)
Gpt-4o 2.27 (80%) 0.16 (10%) Inf (100%) 0.46 (0%) TBD TBD TBD
Gpt-o1-mini 1.36 (60%) 0.04 (0%) TBD TBD TBD TBD TBD
Claude3-haiku 2.19 (90%) 0.19 (10%) 5.25 (40%) 0.34 (0%) 0.75 (25%) 0.33 (0%) 0.93 (45%)
Llama3.1-8b 0.28 (5%) 0.12 (5%) 14.9 (75%) 0.18 (0%) 0.48 (5%) 0.14 (0%) 0.71 (25%)
Llama3.1-70b 0.36 (15%) 0.14 (0%) 58.9 (95%) 0.33 (0%) 0.59 (15%) 0.31 (0%) 0.71 (30%)
Llama3.1-405b 0.70 (30%) 0.10 (0%) 3.0k(100%) 0.28 (0%) 0.56 (10%) 0.32 (0%) 0.47 (15%)
Glm-4-plus 0.78 (30%) 0.21 (5%) 153 (100%) 0.38 (0%) 0.60 (10%) 0.30 (0%) 1.03 (55%)
Based on the data presented in Table 3, two conclusions can be extrapolated. First, adequate parameters for the large
model are necessary for decision-making. Llama-3.1-8b, the model with minimum parameters, performs nearly the
worst among all the models we tested, while the 70b and 405b models perform better than the 8b model. Second,
improving reasoning ability does not lead to a linear improvement in decision-making ability. Although GPT-4o
performed the best in most experiments, it still had a zero winning rate in some tasks that can easily be completed,
such as task 4. These results lead to a conclusion: pre-trained large models cannot directly undertake complex
decision-making tasks, and learning in deployment scenarios is almost inevitable.
5 Discussion
In the experiments, we found that there are several deficiencies in LLM-based decision making.
Hallucinations. Hallucination is the first problem that leads to bad decisions. Sometimes, LLMs confuse screen
coordinates with minimap coordinates (as shown in Figure 6), or use unmentioned actions in the Valid Actions part
of the observation. Sometimes, LLMs even damage teammate units. Hallucination has become an urgent problem in
LLM-based decision making.
Poor knowledge utilization. Large models generally exhibit a significant deficiency in leveraging game-related
knowledge. In task 2, game knowledge shows that Phoenix’s GravitonBeam ability will prevent the unit from moving
and attacking. However, this ability is still overused, failing to obtain victories in task 2. In task 5, even the LLM knows
the PurificationNova of Disruptor deals a lot of damage, they use the skill on injured units, causing a large amount of
spillover damage.
8
Zongyuan Li et. al.
Game Info: ...... Team HighTemplar-1 Info:
Team minimap position: [23, 31]
Controlled Team Units:
Unit: HighTemplar Tag: 0x100240001 ScreenPos: [66, 69] ...... Unit: HighTemplar Tag: 0x101dc0001 ScreenPos: [66, 64] ......
......
...... Tasks: Team HighTemplar-1' task: Go to minimap coordinate [32, 32]. Now, start generating your analysis and actions:
Analysis:
Team HighTemplar-1 consists of several High Templars, each with full health andenergy. Our task is to move to the specified minimap coordinate [32, 32]. Beforeproceeding, we should prepare to engage any nearby enemy units with a Psionic
Storm, if they are in range. However, we currently have no information about nearby
enemy units, so we'll focus on reaching the destination while maintaining our ability toengage if necessary. Actions:
Team HighTemplar-1:
<Ability_PsiStorm_Screen([32, 32])>
<Move_Minimap([32, 32])>
<No_Operation()>
screen [32, 32] minimap [32, 32]
<Ability_PsiStorm_Screen([32, 32])>
Query
LLM��
��
Figure 6: LLM hallucination in decision-making. This is an example that LLM confuse the screen coordinates and
minimap coordinates, wasting a PsiStrom skill and 75 energy. This is just an instance of the hallucination of large
models. In fact, there are many other forms of expression, such as attacking teammates and incorrectly choosing priority
targets.
Poor understanding of the world. Lack of world understanding is a kind of lack of knowledge. Pre-trained LLMs are
generally not trained in decision-making tasks. They do not know how to win in each task. In task 4, for example, the
large model should use Stalker’s Blink ability to transfer injured units to the rear. However, this ability is rarely used,
resulting in the unit’s death and a zero winning rate in task 4, even though the LLM is told that Blink is commonly used
to pursue the enemy or retreat injured units.
Low quality collaboration. In the multi-agent tasks like task 5 to 8, LLM agents should collaborate with others and
defeat the enemy together. However, we found it difficult for these agents to reasonably allocate targets, coordinate
attack timing, and coordinate retreat timing, no matter whether they collaborate with or without a leadership/commander.
How to improve the collaboration performance of LLM agents is important in building a high-level multi-agent
decision-making system.
These issues hinder the application of LLMs in decision-making scenarios. Fortunately, there are many ways to improve
the decision-making ability of large models. For example, directly providing knowledge to LLMs may directly improve
their ability. However, providing LLMs knowledge or precisely annotated datasets usually demands quite a lot of
resources. Self-supervised learning is still the most attractive way to enhance decision-making ability, either through
reward-based or reward-free methods (such as LLM reflection), and either through parameter training or training-free
techniques.
6 Conclusion
In this paper, we introduce a new environment for LLM decision-making, the first environment that accommodates
continuous PySC2 actions, and the first LLM StarCraft II environment with a multi-agent framework and communication
system. In experiments, we test mainstream LLMs’ performance in both the LLM-SMAC and LLM-PySC2 task groups,
among which the LLM-PySC2 task group is a brand-new experimental scenario that we designed for large models.
Results of baseline tests show that LLMs can make decisions, generating actions in the correct form. Still, the decision
quality is relatively low and there are several problems like hallucinations, poor utilization of game knowledge, and
lack of world understanding. Results indicate that learning in the deployment environment is necessary for LLM-based
decision-making. We hope the LLM-PySC2 environment can promote research on LLM learning methods, helping
LLM-based decision-making methods better adapt to task scenarios.
9
Zongyuan Li et. al.
References
[1] Oriol Vinyals, Tim Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, and
Demis Hassabis. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782,
2017.
[2] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and Shimon
Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. arXiv
e-prints, page arXiv:1803.11485, March 2018.
[3] Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted QMIX: Expanding Monotonic Value
Function Factorisation for Deep Multi-Agent Reinforcement Learning. arXiv e-prints, page arXiv:2006.10800,
June 2020.
[4] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The Surprising
Effectiveness of PPO in Cooperative, Multi-Agent Games. arXiv e-prints, page arXiv:2103.01955, March 2021.
[5] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung,
David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo
Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets,
Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine,
Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina
McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps,
and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575:350 –
354, 2019.
[6] Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.
Generative Agents: Interactive Simulacra of Human Behavior. arXiv e-prints, page arXiv:2304.03442, April
2023.
[7] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu,
Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open-world environments via large
language models with text-based knowledge and memory. arXiv preprint arXiv:2305.17144, 2023.
[8] Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele
Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the
game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067–1074,
2022.
[9] Anonymous. Large language models play starcraft ii: Benchmarks and a chain of summarization approach. arXiv
preprint arXiv:2312.11865, 2023. Accessed: 2024-10-21.
[10] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ
Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent
challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems
(AAMAS), pages 2186–2188, 2019.
[11] Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting
Zhuang, and Weiming Lu. Agent-pro: Learning to evolve via policy-level reflection and optimization. arXiv
preprint arXiv:2402.17574, 2024.
[12] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large
language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658,
2023.
10
现在请你结合论文内容，理解这个项目

我在这个环境上做了Visual-CoT的工作，请你理解这个工作。
Visual-CoT: A Visual Guided Chain of Thought Framework
for Fine-grained Control in StarCraft II Actions
Yanan Nia∗
, Runnan Qia
, Zongyuan Lib
, Kuihua Huanga
, Lumin Jianga
, Xiaojie Xub
,
Guo Xianb
, and Xuebo Zhangb
aLaboratory for Big Data and Decision, National University of Defense Technology, Changsha,
China
bCollege of Artificial Intelligence, Nankai University, Tianjin, China
ABSTRACT
In recent years, large language models (LLMs) have made significant progress in the field of natural language
processing. However, fine-grained action control in complex environments remains a significant challenge. To
address this issue, we propose a framework called Visual-CoT (Visual guided Chain of Thought), which aims to
enhance the ability of LLMs to perform fine-grained action control in complex strategy games such as StarCraft
II. This framework integrates visual information with the chain of thought, allowing visual data to seamlessly
interact with the reasoning and decision-making processes of LLMs. We implement Visual-CoT on the LLM￾PySC2 environment (the Large Language Model StarCraft II Learning Environment) and validate it on a map
featuring high-ground terrain information, simulating a scenario where one Colossus faces thirty-two Zerglings.
Experimental results show that the Visual-CoT framework improves the LLM’s win rate to 45% (winning 9 out
of 20 games), significantly enhancing fine-grained control and precise action execution capabilities. The code is
open-sourced and available at https://github.com/Flycat-Tom/Visual-COT.
Keywords: large language models, Visual-CoT, fine-grained action control, StarCraft II, strategy games

1. INTRODUCTION
   LLMs have achieved remarkable success in the fields of natural language processing and complex decision-making.
   For instance, models such as the Generative Pre-trained Transformer (GPT) series1 and Large Language Model
   Meta AI(LLaMA)2 have demonstrated outstanding performance in language generation, knowledge reasoning,
   and problem-solving. With the development of multimodal large language models (such as GPT-4),3
   these models
   have begun to integrate textual and visual information, significantly enhancing their perception and decision￾making capabilities.4–6 However, in complex strategy games, particularly in scenarios that require fine-grained
   action control, LLMs still face significant challenges.7 These games demand that agents extract critical features
   from multimodal information and formulate and adjust strategies in real time. Even for multimodal LLMs,
   the integration of visual information is not yet sufficient to meet the demands of real-time decision-making
   in dynamic environments, which limits their ability in spatial understanding and environmental perception.
   Therefore, deeply integrating visual information into the decision-making process of LLMs to enhance their
   performance in complex tasks has become an urgent problem to address.
   In the absence of visual guidance, LLMs exhibit limitations in understanding physical laws and causal relation￾ships.8 Additionally, they struggle to perform fine-grained action control tasks that require visual information
   support. For instance, even advanced multimodal models, such as GPT-4, still face difficulties in fully interpret￾ing visual features in complex scenarios, especially when real-time reasoning is required.9
   In strategic scenarios
   involving complex terrains (e.g., high ground and low ground) and enemy-allied distributions, the lack of effec￾tive visual information integration hampers the model’s ability to accurately interpret key visual elements on
   ∗Corresponding author
   Further author information: (Send correspondence to Yanan Ni)
   Emails: {niyanan, qirunnan13579, jlm_mz, khhuang}@nudt.edu.cn, {2120230524, 2120220490, guoxian, zhangx￾uebo}@mail.nankai.edu.cn
   the map, such as the relative positions of units, advantageous terrain distributions, and threat zones of enemy
   forces. This information deficit directly affects the precision and effectiveness of strategy generation. To address
   this issue, we propose the Visual-CoT framework, which integrates image inputs with Chain of Thought(CoT)
   prompting techniques. This approach seamlessly embeds visual information into the reasoning and decision￾making processes of LLMs, thereby mitigating the inadequacies in their understanding of visual information in
   complex strategic tasks.
   The main contributions of this study are summarized as follows:
   Proposed the Visual-CoT framework: Innovatively combined visual guidance with CoT reasoning to
   enhance the fine-grained action control capabilities of large language models in complex strategy games, providing
   in-depth reasoning and logical support for decision-making.
   Validated effectiveness on the LLM-PySC2 environment: Designed a complex scenario featuring a
   Colossus versus 32 Zerglings on a specialized map, demonstrating the critical role of visual information in strategy
   generation and action control, and thoroughly validated the framework through extensive experiments.
   Achieved significant performance improvements: Experimental results showed that models employing
   the Visual-CoT framework achieved substantial improvements across multiple metrics, including win rate, unit
   operation precision, and tactical planning capabilities, with the win rate increasing from 0% to 45%, highlighting
   the advantages of integrating visual guidance with CoT reasoning.
   The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 outlines the
   Visual-CoT framework. Section 4 describes the methodology. Section 5 presents the experiments and results.
   Finally, Section 6 concludes the paper with future directions.
2. RELATED WORK
   2.1 Recent Advances in LLM Applications for Game AI
   In recent years, LLMs have made significant progress in their application to game AI. In 2023, the ”Ghost in the
   Minecraft” agent, based on LLMs, used a task decomposition approach to achieve a 67.5% success rate in the
   ”Diamond Challenge” of Minecraft.10 This research laid the foundation for subsequent efforts to apply LLMs
   in decision-making across other game scenarios. For example, in the open-world game Red Dead Redemption
   2, an agent called CRADLE,11 which integrates video input, was proposed to construct and update the game’s
   skill library. Other studies have explored using LLMs to master advanced strategies in games with incomplete
   information, such as poker and blackjack.12 LLMs have also been successfully deployed in the strategy game
   Diplomacy to drive decision-making and strategic reasoning.13
   In the realm of real-time strategy (RTS) games, TextStarCraft II14 combined the complexity of StarCraft II
   with LLMs, using a text adapter to interact with the game environment. However, this system faced limitations in
   its action space and single-agent design. To address these challenges, the LLM-PySC2 environment introduced
   RGB images and a richer action space, thereby advancing research in complex strategy decision-making and
   micro-management control.15
   2.2 Visual Guided Chain of Thought Method
   The CoT method enhances the logical reasoning capabilities of LLMs in complex tasks by guiding the model
   through step-by-step reasoning.16 However, traditional CoT methods face limitations in scenarios requiring
   visual information for reasoning. Studies have demonstrated that combining multimodal information with CoT
   can significantly improve the complex reasoning capabilities of LLMs.17
   To improve these limitations, we propose a Visual-CoT method, which embeds visual information into the
   CoT framework. By integrating visual elements into the reasoning chain, our approach empowers the model with
   stronger scene understanding and decision optimization capabilities, achieving deep integration of linguistic and
   visual information.
   2.3 Introduction to the LLM-PySC2 environment
   The LLM-PySC2 environment is a research environment based on StarCraft II, designed to investigate the per￾formance of LLMs in complex decision-making scenarios. The environment provides a diverse set of observation
   interfaces, including global information, localized combat data, and a structured game knowledge database.
   These observations can be presented in textual or multimodal formats.
   The environment supports an expanded and comprehensive action space, enabling LLMs to perform precise
   unit operations and skill executions. It also facilitates multi-agent research. By validating the Visual-CoT
   framework on the LLM-PySC2 environment, our experiments demonstrate the significant role this framework
   plays in enhancing model performance.
3. OVERVIEW OF THE VISUAL-COT FRAMEWORK
   3.1 Framework Design Philosophy
   The core concept of the Visual-CoT framework is to integrate visual information to guide LLMs in generating
   comprehensive chains of thought, enabling fine-grained action control in complex gaming environments. In real￾time strategy games like StarCraft II, visual elements such as terrain features, unit positions, and enemy-allied
   distributions play a crucial role in strategic decision-making.
   Traditional LLMs, when handling such tasks, typically rely on textual or numerical observational data,
   which fails to fully exploit the rich visual cues available. This limitation constrains the quality of decision￾making. The Visual-CoT framework addresses this challenge by incorporating visual guidance, embedding in￾game visual data—such as screenshots and feature maps—into the model’s reasoning process, thereby enhancing
   its situational awareness and comprehension of the environment.
   In our research, we designed a specific map scenario in StarCraft II featuring the Colossus unit, where the
   model needs to interpret visual information to assist in tactical operations. The model is required to analyze
   visual cues to assess terrain characteristics, determine the spatial relationships between enemy and allied units,
   and make tactical choices. This design highlights the importance of visual information in complex decision￾making tasks and provides a robust experimental foundation to validate the effectiveness of the Visual-CoT
   framework.
   3.2 Framework Structure
   We propose the Visual-CoT framework for fine-grained action control in complex scenarios within StarCraft II.
   Visual-CoT enhances the decision-making capabilities of LLMs in multimodal environments by integrating visual
   information with CoT reasoning through optimized prompt design. The core of the framework lies in embedding
   visual descriptions and reasoning logic into prompts, guiding the model to iteratively analyze complex game
   scenarios and generate actions.
   Figure 1 outlines the three key components of the Visual-CoT framework: system prompt, example input
   prompt, and example output prompt. The system prompt consolidates task rules, map information, and strategic
   priorities while incorporating critical features extracted from visual game observations, such as terrain boundaries
   and enemy-allied distributions, to provide a decision-making context. The example input prompt highlights the
   visual cues and related descriptions necessary for reasoning within the CoT framework, demonstrating how to
   leverage visual information for environment understanding and strategy planning. The example output prompt
   showcases the model’s expected reasoning process and generated actions, including position evaluations, strategy
   selection, and detailed execution plans.
   Multimodal large language models (MMLMs) process text observations and RGB screen images as input.
   These inputs, augmented with visual prompts, guide the model in extracting critical visual information. Com￾bined with task knowledge and strategic rules embedded in the System Prompt, the model employs CoT reasoning
   to generate detailed action decisions.
   Figure 1: Visual-CoT Framework: Fine-Grained Action Control in StarCraft II. The framework combines visual
   observations (Screen RGB Image) with text inputs, utilizing supplemental visual information to assist the CoT
   in guiding the LLM’s reasoning and decision-making, ultimately generating fine-grained strategic actions.
4. METHOD
   4.1 Map and Scenario Design
   In this study, we utilized the StarCraft II map editor to design a scenario where one Colossus faces off against
   32 Zerglings,as shown in figure 2. This scenario highlights the importance of strategic planning and fine-grained
   action control while emphasizing the critical role of visual information in model decision-making in complex
   environments. As a high-damage ranged unit, the Colossus requires precise control to avoid being surrounded
   by Zerglings and to maximize its damage output.
   The designed map includes complex terrains such as high ground, low ground, cliffs, and narrow pathways.
   The Colossus starts at the center of the high ground, benefiting from vision and terrain advantages, and can
   leverage its unique ability to traverse cliffs. The Zerglings, on the other hand, must navigate narrow pathways
   to attack the high ground, which limits their movement options and aids the Colossus in defense and offense.
   The edges of the map are marked with impassable black backgrounds, and red lines indicate the boundaries of
   the playable area.
   In this scenario, the model must rely on visual information to perceive the battlefield situation, understand
   terrain characteristics, and grasp the positional relationships between allied and enemy units. Numerical or
   textual information alone cannot fully describe these spatial relationships, making this scenario an ideal exper￾imental condition to evaluate the effectiveness of the Visual-CoT framework in enhancing decision-making in
   complex environments.
   4.2 Visual Information Extraction
   In this study, using the observation wrapper of the LLM-PySC2 environment, a screenshot is captured every
   second (equivalent to 22.4 game steps) to record the current game state. The screen coordinate system ranges
   Figure 2: Map of the 1 Colossus vs 32 Zerglings scenario, featuring high and low terrain with connecting pathway
   from 0 to 256, with the origin located at the top-left corner. To enhance the model’s understanding of map
   boundaries and terrain features, edge masks are generated by detecting the boundaries between black pixel
   regions and non-black regions, with the boundaries marked in red (black backgrounds represent unpassable
   areas). This processing clarifies map boundaries, helping the model to avoid unpassable areas while leveraging
   terrain advantages in decision-making. The specific marking results are shown in figure 3.
   Figure 3: Processed Screen RGB Image with Terrain Boundary Markings
   The processed images are directly embedded into the model’s prompts, providing a reference for the model
   when generating chains of thought. By including this visual information in the prompts, the model can better
   understand the current game state and make more informed action decisions.
   4.3 Prompt Generation
   In the Visual-CoT framework, the prompt generation process is illustrated in algorithm 1. This algorithm
   organizes system prompts, example inputs and outputs, visual information, and current observations into a
   message sequence, which serves as input for the LLMs to perform multimodal reasoning and decision-making.
   By integrating visual information with chains of thought, the model can better comprehend the game environment
   and generate more refined action instructions.The specific generated prompts are provided in Appendix B.
   4.4 Visual-Guided Chain of Thought Design
   4.4.1 Role of the Chain of Thought
   In the Visual-CoT framework, the Chain of Thought is one of the core components of model decision-making.
   By employing step-by-step reasoning and analysis, it enhances the model’s depth of reasoning and transparency
   Algorithm 1 Visual-CoT Prompt Generation
   Input: Text observation ot; User image It (Base64 encoded RGB image data); Example images Iex1 and Iex2
   (Base64 encoded); System prompt pr; Example input pi and output po; Screen RGB prompt prgb; Descriptions
   descex1 and descex2 for Iex1 and Iex2.
   Output: Formatted message sequence mt for multimodal reasoning input to the LLM.
   1: function Generate_Message_Sequence(ot)
   2: Initialize message list mt as an empty list
   3: Append(pr, role=”system”)
   4: Append(descex1 and Iex1, role=”user”)
   5: Append(descex2 and Iex2, role=”user”)
   6: Append(pi
   , role=”user”)
   7: Append(po, role=”assistant”)
   8: Append(prgb and It, role=”user”)
   9: Append(ot, role=”user”)
   10: return mt
   11: end function
   in decision-making. Through iterative analysis, the model organically integrates visual and textual information,
   enabling a more accurate understanding of the current game state (e.g., unit positions, terrain features, and
   enemy dynamics). This facilitates the formulation of precise and effective strategies while avoiding errors caused
   by incomplete information comprehension. Furthermore, the CoT provides the model with a clear reasoning
   logic, not only improving its grasp of complex game states but also offering a solid foundation for optimizing
   and refining the framework.
   4.4.2 Methods for Integrating Visual Information
   To effectively integrate visual information into the Chain of Thought, we embed preprocessed game screenshots
   (Base64 encoded) into the prompts, accompanied by textual descriptions of terrain, unit positions, and map
   coordinates. This guides the model to fully utilize visual information during reasoning. Additionally, by providing
   examples of inputs and outputs that include visual information, the model learns to synthesize visual cues and
   textual information for reasoning within the Chain of Thought.
   4.4.3 Chain of Thought Generation Process
   In the Visual-CoT framework, the CoT facilitates complex decision-making through a step-by-step reasoning
   process, consisting of five logical stages. The first stage is position evaluation, where the model analyzes the
   battlefield situation using both visual and textual information, such as determining whether the Colossus is
   on high ground and if Zerglings are approaching. Based on the results of the position evaluation, the process
   proceeds to action priority assessment, where factors like weapon cooldowns and enemy positions are considered
   to identify the most critical action. The next stage involves strategic recommendation, where global strategies
   are formulated according to the priorities, such as retreating to leverage terrain advantages or prioritizing specific
   targets for attack. Subsequently, during the movement and attack analysis phase, the strategy is broken down
   into concrete operations, with the model evaluating the optimal path, target selection, and attack timing to
   ensure effective and safe execution. Finally, in the action output stage, the model generates specific operational
   commands, such as moving to a designated location or attacking a particular unit. This stepwise reasoning
   structure enhances the accuracy of the model’s decision-making while ensuring interpretability throughout the
   process.
   4.5 Fine-grained Action Control Strategy
   4.5.1 State Assessment for Decision-making
   To achieve precise control of the Colossus, a detailed decision-making process and condition evaluation rules
   are established. Before generating action commands, the model evaluates the current game state and identifies
   key variables, including the positions of the Colossus and Zerglings, weapon cooldown status, and the health
   of the Colossus. By assessing these conditions, the model gains a comprehensive understanding of the current
   battlefield situation, providing a basis for subsequent action decisions.
   For instance, the model uses visual information (such as terrain and unit positions in the image) and textual in￾formation (such as unit attributes and statuses) to determine whether the Colossus is on high ground (IS_COLO￾SSUS_HIGH_GROUND), whether Zerglings are near the Colossus (IS_ZERGLING_NEAR_COLOSSUS), and
   whether the Colossus’s weapon is on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN), among other con￾ditions.
   4.5.2 Action Decision Priorities
   As illustrated in the figure 4, the action decision rules and priorities define the optimal strategies for the model
   under specific circumstances and special conditions. The following rules are outlined to guide the model in
   making reasonable action decisions in complex battle environments.
   For example, when the Colossus has low health, its weapon is on cooldown, and enemies are approaching,
   the model prioritizes moving to retreat to a safe location, leveraging terrain advantages to delay enemy pursuit.
   Figure 4: Action Decision Rules and Priority Judgments for Colossus Combat Scenarios
5. EXPERIMENTS AND RESULTS
   5.1 EXPERIMENTS
   5.1.1 Experimental Setup
   To validate the effectiveness of the proposed Visual-CoT framework in the complex strategy game StarCraft II,
   we performed a series of controlled experiments. The experiments aimed to evaluate the model’s performance in
   fine-grained action control, strategy planning, and decision-making capabilities. We focused on comparing the
   performance of models utilizing the Visual-CoT framework against the baseline method, LLM-PySC2.
   The experiments were conducted on the LLM-PySC2 environment, using GPT-4 and GPT-4O as the mul￾timodal large language models, which served as the primary test models for this study. The experiments were
   executed on version 5.0.13 of StarCraft II, with the selected map featuring a complex battle scenario specifically
   designed for Colossus versus Zerglings.
   Each experimental configuration involved running 20 matches to ensure the statistical significance and relia￾bility of the results.
   5.1.2 Evaluation Metrics
   To evaluate the model’s performance, several metrics were employed to comprehensively assess its performance
   in the competition. The Overall Win Rate (W) quantifies the ratio of the number of victories Nwin to the total
   number of games Ntotal and is defined as:
   W =
   Nwin
   Ntotal
   × 100%, (1)
   Additionally, the Average Enemy Kills (Eavg) is the average number of enemies killed per game, computed
   as the mean of the number of enemies killed in each match, as shown by the formula:
   Eavg =
   ∑Ntotal
   i=1 Ei
   Ntotal
   (2)
   For victorious games, the Average Victory Time (Twin) and Average Health Ratio (Hwin) represent the
   average duration of the winning games and the average remaining health ratio in those games, respectively, and
   are calculated as follows:
   Twin =
   ∑Nwin
   i=1 Ti
   Nwin
   , Hwin =
   ∑Nwin
   i=1 Hi
   Nwin
   For losing games, the Average Loss Time (Tlose) and Average Enemy Kills in Losses (Elose) represent the
   average duration of the losing games and the average number of enemies killed in those games, respectively.
   These are calculated as:
   Tlose =
   ∑Nlose
   i=1 Ti
   Nlose
   , Elose =
   ∑Nlose
   i=1 Ei
   Nlose
   (4)
   Finally, the Kill-to-Loss Ratio (KLR) measures the ratio of total enemy kills to the number of Colossus units
   lost in all games, and is computed using the formula:
   KLR =
   ∑Ntotal
   i=1 Ei
   Ncolossus_lost
   (5)
   These evaluation metrics provide a comprehensive overview of the model’s performance across various di￾mensions, including win rate, kill efficiency, game duration, performance in losses, and survival capability.
   5.2 Experimental Results
   5.2.1 Performance Comparison between Visual-CoT Framework and Baseline Prompt Model
   To evaluate the performance improvements introduced by the Visual-CoT framework in complex strategy game
   scenarios, we conducted comparative experiments on the LLM-PySC2 environment. Each experimental setup,
   including the baseline, was tested in 20 matches to ensure statistical significance and reliability.
   The following setups were compared: (1) LLM-PySC2, utilizing the LLM-PySC2 prompt framework (with
   textual and visual prompts) and GPT-4O as the multimodal model, but without the Visual-CoT framework; (2)
   Visual-CoT Framework, incorporating visual guidance and CoT reasoning techniques, using GPT-4O and GPT-4
   as multimodal models. The comparison metrics included Win Rate (W), Average Enemies Eliminated (Eavg),
   Average Victory Time (Twin), Average Defeat Time (Tlose), Average Enemies Eliminated in Defeats (Elose),
   Average Health Ratio in Victories (Hwin), and Kill-to-Loss Ratio (KLR).
   As shown in Table 1, the Visual-CoT framework significantly outperformed the baseline model across all
   key metrics when using the GPT-4O model. The Win Rate (W) improved from 0.00% to 45.00%, Average
   Enemies Eliminated (Eavg) increased from 14.5 to 24.00, and the Kill-to-Loss Ratio (KLR) rose from 14.5 to
   43.63. These results demonstrate that the deep integration of visual guidance and CoT reasoning is crucial for
   improving decision-making accuracy and battle performance.
   The Visual-CoT model using GPT-4 showed lower performance compared to GPT-4O but still significantly
   outperformed the baseline model. This indicates that the multimodal data handling capability of GPT-4O
   provides an advantage. However, the Visual-CoT framework effectively enhances combat control and strategic
   planning by combining visual information with reasoning capabilities.
   Table 1: Performance Comparison of Visual-CoT Framework
   Metric W (%) Eavg Twin (s) Tlose (s) Elose Hwin (%) KLR
   LLM-PySC2 (GPT-4O) 0.00 14.5 N/A 10.95 14.5 N/A 14.5
   Visual-CoT (GPT-4O) 45.00 24.00 57.44 26.18 17.45 35.08 43.63
   Visual-CoT (GPT-4) 25.00 20.45 72.60 38.40 16.60 24.00 27.27
   5.2.2 Ablation Study
   To evaluate the impact of visual guidance and CoT reasoning on model performance, we conducted two ablation
   experiments, with each setup running 20 matches:
6. ScreenRGB Removed: The model used only textual information and was guided by CoT reasoning but
   received no visual input.
7. COT Removed: The model received real-time game images as input but did not use Chain of Thought
   examples for input-output reasoning. It directly generated action commands without deep reasoning guidance.
   The experimental results are shown in Table 2.
   Table 2: Results of Ablation Study
   Metric W (%) Eavg Twin (s) Tlose (s) Elose Hwin (%) KLR
   Visual-CoT (GPT-4O) 45.00 24.00 57.44 26.18 17.45 35.08 43.63
   ScreenRGB Removed (GPT-4O) 5.00 12.75 55.00 25.36 11.73 35.71 13.42
   COT Removed (GPT-4O) 0.00 14.00 N/A 13.90 14.00 N/A 14.00
   When ScreenRGB Removed, the absence of visual guidance caused the Win Rate (W) to drop significantly
   to 5.00%, and the Average Enemies Eliminated (Eavg) decreased to 12.75. This indicates that visual guidance is
   critical for understanding the game environment and making strategic decisions.
   When CoT Removed, the Win Rate dropped to 0%, and the Kill-to-Loss Ratio (KLR) reduced to 14.00,
   highlighting the essential role of Chain of Thought reasoning in enhancing reasoning ability and decision quality.
   The results demonstrate the unique contributions of both visual guidance and Chain of Thought reasoning
   in improving model performance. Their combination enables the Visual-CoT framework to achieve significant
   performance gains in complex strategic scenarios.
   5.2.3 Conditional Judgment Accuracy Analysis
   To evaluate the effectiveness of the Visual-CoT framework, we analyzed the model’s accuracy in conditional
   judgments, focusing on key conditions. In the best match (where the Colossus had the highest remaining
   health), at 41 seconds of game time, each output from GPT-4O was analyzed alongside visual inputs. The
   results are shown in Table 3.
   As presented in Table 3, the model achieved 85.36% accuracy in determining whether the Colossus was on
   high ground (IS_COLOSSUS_HIGH_GROUND) and 87.80% in recognizing whether Zerglings were near the
   Colossus (IS_ZERGLING_NEAR_COLOSSUS). These results indicate that the model effectively integrates
   Table 3: Accuracy Analysis of Conditional Judgments in the Best Match
   Metric Accuracy (%)
   IS_COLOSSUS_HIGH_GROUND 85.36
   IS_ZERGLING_HIGH_GROUND 87.80
   IS_ZERGLING_NEAR_COLOSSUS 87.80
   IS_COLOSSUS_AT_CLIFF 82.96
   visual inputs with reasoning capabilities to perceive battlefield conditions accurately. The model also achieved
   87.80% accuracy in judging whether Zerglings were on high ground (IS_ZERGLING_HIGH_GROUND) and
   82.96% in determining whether the Colossus was at the edge of a cliff (IS_COLOSSUS_AT_CLIFF). This
   demonstrates the Visual-CoT framework’s ability to understand complex geographical conditions and provide a
   reliable foundation for subsequent decision-making.
   5.3 Result Analysis
   The experimental results demonstrate that the Visual-CoT framework, by integrating visual guidance and Chain
   of Thought reasoning, effectively improves the model’s decision-making accuracy and efficiency in complex combat
   scenarios in StarCraft II. The model is able to accurately understand the battlefield terrain, unit positions, and
   the enemy-friendly situation, formulate reasonable strategies, and achieve tactical objectives through precise
   action control.
   Visual guidance provides the model with critical environmental perception information, enabling the model to
   accurately assess the battlefield situation using visual inputs. The addition of visual information and the Chain
   of Thought reasoning enhance the model’s inference and decision-making capabilities, improving the depth and
   accuracy of strategy planning through step-by-step reasoning. To provide a more intuitive understanding of the
   Visual-CoT framework’s refined control and decision-making capabilities, Appendix A presents a series of visual
   demonstrations. These include annotated screenshots that illustrate the Colossus’s tactical movements, strategic
   positioning, and effective use of terrain to counter enemies.
8. CONCLUSION AND FUTURE WORK
   This paper proposes the Visual-CoT framework, which enhances the action fine-grained control ability of large
   language models in complex strategy games, such as StarCraft II by combining visual guidance with Chain of
   Thought reasoning. Experimental results demonstrate that this framework significantly improves the model’s
   win rate and decision-making quality. In each of the 20 experimental runs, the model’s performance showed
   notable improvements, validating the effectiveness of the framework.
   While this research has made progress in the area of Visual-CoT, there are still limitations. Future research
   directions include: 1) validating the model’s generalization ability in more complex maps and diverse battle
   scenarios; 2) extending to multi-agent decision-making, exploring communication and collaboration mechanisms
   to enhance strategic effectiveness; 3) incorporating advanced computer vision techniques and integrating them
   deeply with large language models to improve visual information processing capabilities; 4) applying the Visual￾CoT framework to other domains to expand its applicability.
   APPENDIX A. VISUAL-COT REFINED CONTROL DEMONSTRATION
   This appendix presents a visual demonstration of the refined control enabled by the Visual-CoT framework.
   Key decision-making processes and corresponding actions are illustrated through annotated screenshots from the
   experimental scenarios.
   (a) When the Colossus is surrounded on the high
   ground, LLM issues a command to move across the
   cliff to the low ground.
   <Move_Screen([135, 200])> moves the Colossus to
   the green marker on the low ground.
   (b) Upon reaching the low ground, the Colossus attacks
   the LLM-locked enemy while enemy Zerglings are mov￾ing toward the low ground.
   <Attack_Unit(0x100440001)> directs the Colossus to
   target the specified enemy tag.
   (c) When the Colossus is approached on the low
   ground, LLM issues a command to move across the
   cliff to the high ground.
   <Move_Screen([135, 64])> moves the Colossus to the
   green marker on the high ground.
   (d) With multiple precise and refined operations, the
   Colossus combines its own abilities and terrain infor￾mation to eliminate an overwhelming number of ene￾mies.
   APPENDIX B. MAIN PROMPTS
   This appendix provides a detailed description of the prompts used in the Visual-CoT framework, including
   system prompt, example input prompt, and example output prompt. Certain content has been summarized and
   streamlined for clarity.
   System Prompt
9. Role Description
   Describes the character’s identity, mission objectives, and primary responsibilities.
10. Map Information
    Provides information about the map’s coordinate system, terrain features, and obstacles.
    • Coordinate System: The 2D grid coordinate system ranges from [0, 256] × [0, 256], with each
    grid cell being 32×32 pixels.
    • Terrain: Includes high ground, low ground, pathways, cliffs, and impassable areas. The Colossus
    can traverse the cliffs between high and low ground, while Zerglings must use the pathway.
11. Key Victory Points
    Strategies to leverage the Colossus’s advantages for victory.
    • Maintain Maximum Attack Range: Stay within the maximum attack range to avoid close
    combat.
    • Focus Fire on Dense Enemy Clusters: Prioritize attacking the densest enemy groups to max￾imize damage.
    • Use Cliff Barrier to Delay Enemy: Use the terrain’s cliff to force enemies to take detours,
    buying time to reposition or counterattack.
12. Decision Process
    Assess the battlefield based on conditions and determine action priorities.
    • Condition Assessment: Evaluate the status and positioning of the Colossus and Zerglings to
    assess the current situation.
    • Action Priority Determination: Based on the assessment, decide whether movement or attack
    should take priority.
13. Integrated Strategy and Decision Framework
    Combine strategic elements to formulate an overall strategy and specific action plan.
    • Strategy Suggestions: Provide strategic advice based on the battle situation, such as utilizing
    terrain advantages or avoiding encirclement.
    • Action Decisions: Based on the current situation, decide on the movement and attack priorities,
    as well as the best timing and routes for actions.
14. Analysis
    In your response, please follow the format outlined in the example output, including section titles, num￾bering, and the terminology used. Ensure that you use the same variable names and terms specified in
    the decision process and chain of thought
    • Condition Assessment: Assess the status and positioning of the Colossus and Zerglings.
    • Action Priority Determination: Based on the situation, decide whether to prioritize movement
    or attack.
    • Strategy Suggestions: Offer timely strategic adjustments based on the battlefield conditions.
    • Movement Analysis: Determine the best movement paths and timing for the Colossus to ensure
    safety.
    • Attack Analysis: Evaluate the best attack timing and targets to maximize damage.
    • Action Output: Based on the analysis, generate specific operational steps and instructions.
15. Action Output:
    Team Colossus-1:
    • Action steps
    Example Input Prompt
    Game Info
    Time: 0:05
    Team Colossus-1 Info:
    Team minimap position: [31, 27]
    Controlled Team Units:
    • Unit: Colossus, Tag: 0x100000001, ScreenPos: [150, 123], Health: 350 (100
    Nearby Enemy Units:
    • Unit: Zergling, Tag: 0x1002c0001, ScreenPos: [52, 164], Distance: 10, Health: 6 (17%)
    • Unit: Zergling, Tag: 0x100180001, ScreenPos: [79, 159], Distance: 7, Health: 35 (100%)
    • Unit: Zergling, Tag: 0x100440001, ScreenPos: [112, 140], Distance: 4, Health: 35 (100%)
    • ...
    Relevant Knowledge:
    • Protoss.Colossus: A massive unit with splash laser attacks, ideal for defeating swarms of weak
    units. It can traverse terrain height due to long legs.
    Weapon info: Attack Range 7, target: [’ground’], DPS 13, anti-light DPS 20.
    • Zerg.Zergling: A fast, weak melee unit suited for swarm attacks.
    Weapon info: Attack Range 1, target: [’ground’], DPS 10.
    • ...
    Valid Actions:
    • Team Colossus-1 Valid Actions:
    – <Attack_Unit(tag)>
    – <Move_Screen(screen)>
    Action Args:
16. tag: A hexadecimal number representing a unit, e.g., 0x000000000.
17. screen: A coordinate in the screen range [0, 256].
    Last Step Actions:
    • Team Colossus-1:
    – <Move_Screen([160, 130])>
    – <Attack_Unit(0x1007c0001)>
    – <Attack_Unit(0x100740001)>
    You need to confirm whether the previous action finished executing, and based on this, determine whether
    to continue the old strategy or immediately take other actions.
    Give each team no more than 3 actions, among which activity release should usually be before move and
    attack.
    Now, start generating your analysis and actions:
    Example Output Prompt
    Analysis:
18. Decision Process
    1.1 Condition Assessment
    • IS_COLOSSUS_HIGH_GROUND = True: The Colossus is currently on the high ground.
    • IS_ZERGLING_HIGH_GROUND = True: Zerglings are also concentrated on the high
    ground.
    • IS_ZERGLING_NEAR_COLOSSUS = True: Multiple Zerglings are approaching the
    Colossus, posing an immediate threat.
    • IS_COLOSSUS_AT_CLIFF = False: The Colossus is not near the cliff.
    • IS_COLOSSUS_WEAPON_COOLDOWN = True: The weapon is currently on cooldown
    and cannot attack immediately.
    • IS_COLOSSUS_HEALTH_LOW = False: The Colossus has full health, no low-health
    threat.
    1.2 Action Decision Rules
    • Decision Basis:
    – Special Rules: Prioritize movement over attack when the weapon is on cooldown.
    – Current Scenario: Zerglings are approaching the Colossus, and the weapon is on cooldown.
    – Terrain Condition: The Colossus is near the red boundary of the map, limiting maneuver￾ability.
    • Decision Result:
    – Prioritize movement: Retreat to low ground to avoid encirclement and give time for weapon
    cooldown.
19. Integrated Strategy and Decision Framework
    2.1 Strategy Suggestions
    • Current Strategy: Emergency retreat to the low ground, using the cliff to delay Zerglings’ pursuit.
    • The retreat will provide time for weapon cooldown and create safe distance to regain battlefield
    control.
    2.2 Movement Decision Analysis
    • Enemy Approach Path Prediction: Zerglings are advancing from the left pathway on the high
    ground, attempting to encircle the Colossus.
    • Movement Target Suggestions: Retreat directly to screen coordinates [160, 200], a safe area
    near the cliff on the low ground.
    • Reason: Retreating to the low ground avoids encirclement and uses the cliff to delay enemy pursuit,
    ensuring the Colossus’s safety.
    2.3 Attack Decision Analysis
    • Attack Target Priority: Once the weapon cooldown ends, prioritize attacking the nearest Zer￾glings.
    • Target Selection: Choose the closest Zerglings (e.g., Zergling, Tag: 0x1004c0001 and
    0x100740001) as primary targets.
20. Actions:
    • Team Colossus-1:
    – <Move_Screen([160, 200])> # Retreat directly from high ground to low ground, avoiding
    Zergling encirclement and using the cliff to delay pursuit.
    – <Attack_Unit(0x1004c0001)> # After weapon cooldown, prioritize attacking the nearest Zer￾gling to reduce threat.
    – <Attack_Unit(0x100740001)> # Continue attacking other nearby Zerglings to weaken the
    enemy forces using splash damage.
    REFERENCES
    [1] Team, O. C., “Chatgpt.” Online (2022). Accessed: 26-Nov-2024.
    [2] Touvron, H., Lavril, T., Izacard, G., and et al., X. M., “Llama: Open and efficient foundation language
    models,” (2023).
    [3] Achiam, J., Adler, S., Agarwal, S., and et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774
    (2023).
    [4] et al., R. G., “A survey on advancements in image–text multimodal models: From general techniques to
    biomedical implementations,” Computers in Biology and Medicine 178, 108709 (2024).
    [5] Huang, W., Wu, A., Yang, Y., Luo, X., Yang, Y., Hu, L., Dai, Q., Dai, X., Chen, D., Luo, C., et al.,
    “Llm2clip: Powerful language model unlock richer visual representation,” arXiv preprint arXiv:2411.04997
    (2024).
    [6] Kuckreja, K., Danish, M. S., Naseer, M., Das, A., Khan, S., and Khan, F. S., “Geochat: Grounded large
    vision-language model for remote sensing,” in [Proceedings of the IEEE/CVF Conference on Computer
    Vision and Pattern Recognition], 27831–27840 (2024).
    [7] Xu, X., Wang, Y., and Xu, C. e. a., “A survey on game playing agents and large models: Methods,
    applications, and challenges,” (2024).
    [8] Buschoff, L. M. S., Akata, E., Bethge, M., and Schulz, E., “Visual cognition in multimodal large language
    models,” (2024).
    [9] Yan, H., Hu, X., and Wan, X. e. a., “Inherent limitations of llms regarding spatial information,” (2023).
    [10] Zhu X., Chen Y., T. H. e. a., “Ghost in the minecraft: Hierarchical agents for minecraft via large language
    models with text-based knowledge and memory,” (2024).
    [11] Tan, W., Zhang, W., Xu, X., Xia, H., and Ding, G. e. a., “Cradle: Empowering foundation agents towards
    general computer control,” in [NeurIPS 2024 Workshop on Open-World Agents], (2024).
    [12] Zhang, W., Tang, K., Wu, H., and Wang, M. e. a., “Agent-pro: Learning to evolve via policy-level reflection
    and optimization,” arXiv preprint arXiv:2402.17574 (2024).
    [13] Meta FAIR Diplomacy Team, Bakhtin, A. B. N. D. E. e. a., “Human-level play in the game of diplomacy
    by combining language models with strategic reasoning,” Science 378(6624), 1067–1074 (2022).
    [14] Ma, W., Mi, Q., Zeng, Y., and Yan, X. e. a., “Large language models play starcraft II:benchmarks and a chain
    of summarization approach,” in [The Thirty-eighth Annual Conference on Neural Information Processing
    Systems], (2024).
    [15] Li, Z., Ni, Y., Qi, R., and Jiang, L. e. a., “Llm-pysc2: Starcraft ii learning environment for large language
    models,” (2024).
    [16] Wei, J., Wang, X., Schuurmans, D., and Bosma, M. e. a., “Chain-of-thought prompting elicits reasoning in
    large language models,” Advances in Neural Information Processing Systems 35, 24824–24837 (2022).
    [17] Zhang, Z., Zhang, A., Li, M., and Zhao, H. e. a., “Multimodal chain-of-thought reasoning in language
    models,” (2024).

我给出你这个思维链具体的内容：# Copyright 2024, LLM-PySC2 Contributors. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS-IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

class BasePrompt:

text
def __init__(self):
    self.sp = ''
    self.eip = ''
    self.eop = ''
    self.screen_img_rgb_prompt = ''
    self.screen_img_fea_prompt = ''
    self.minimap_img_rgb_prompt = ''
    self.minimap_img_fea_prompt = ''

class CombatGroupPrompt(BasePrompt):

text
def __init__(self, name, log_id, config):
    super(CombatGroupPrompt, self).__init__()
    self.name = name
    self.config = config
    self.log_id = log_id

    # Part 1
    self.sp = f"""
1. Role Description
 You are a {self.config.AGENTS[self.name]['describe']}.
 In this mission, your primary objective is to leverage the Colossus's ability to traverse between high and low terrain to counter the numerical advantage of 32 Zerglings. Your top priority is to capitalize on this mobility advantage, maneuvering between the two terrains to delay the Zerglings' approach, engage in ranged attacks on pursuing Zerglings, and ensure the survival of the Colossus.

2.Map Information

  2.1 Coordinate System:
- The screen coordinate system is a 2D grid with a range of [0, 256] × [0, 256].
- Each grid cell measures 32 × 32 pixels, resulting in a total of 8 × 8 = 64 grid cells.
- The origin point (0, 0) is at the top-left corner, while (256, 256) is at the bottom-right corner.

  2.2 Terrain Information:  
- Pathway (connecting the high ground and low ground):  
  The map consists of an upper platform (high ground) and a lower platform (low ground), connected by a narrow pathway on the left side.

- High Ground (light green area):  
  The high ground is characterized by light green grass textures and is located on the upper platform.

- Low Ground (dark gray area):  
  The low ground is characterized by dark gray concrete textures and is located on the lower platform.

- Impassable Area (black background):  
  The edges of the map consist of a black background, which is impassable. The Colossus must avoid entering these areas.

- Cliff (light gray vertical surface):  
  The cliff separates the high ground and low ground. The Colossus can freely traverse the cliff, but Zerglings can only access the high ground through the narrow pathway on the left.  
  Only the boundary between the high ground and low ground is considered a cliff; other deep red boundaries are impassable edges of the map.

- Boundary:  
  The boundaries between the high ground and the black background, as well as between the low ground and the black background, are considered impassable edges of the map.  
  The boundaries are represented by deep red lines on the map.

3. Key Victory Points:
  3.1 Maintain Maximum Attack Range:
    - Ideally, the Colossus should stay within its maximum attack range (range 7) to fully utilize its long-range attack advantage while minimizing the risk of melee attacks from enemies.
  3.2 Focus Fire on Dense Enemy Clusters:
    - Target the center of the densest enemy groups to maximize splash damage and quickly reduce enemy numbers.
  3.3 Cliff Barrier Delay Effect：

    The Cliff Barrier Delay Effect refers to leveraging the terrain characteristics of cliffs, where enemy units cannot directly cross the cliff, while allied units can freely traverse it. This forces the enemy to take a detour when transitioning between high and low ground, buying time for allied units to reposition, cool down weapons, and strike effectively.

    - When enemies are heavily concentrated on the high ground: 
      - If allied units are also on the high ground but risk being surrounded, it is recommended to retreat to the bottom-right low ground. This utilizes the cliff as a barrier to delay enemy pursuit, enabling ranged attacks on the enemies while maintaining battlefield control.

    - When enemies are heavily concentrated on the low ground: 
      - If allied units are also on the low ground but risk being surrounded, it is recommended to retreat to the top-right high ground. The cliff barrier forces enemies to take a detour, providing time for allied units to reposition and counterattack effectively.

4. Decision Process
Based on the observed data of the Colossus and Zerglings, as well as the RGB image of the screen, use the following decision rules to determine action priorities:

text
4.1 Condition Assessment Rules:  

- IS_HIGH_GROUND:  
  - Criteria: If the Colossus is on the green high ground (above the cliff, with a clear boundary from the low ground), set to True; otherwise, set to False.  
  - Basis: Determined by verifying the Colossus's position in the green area via RGB image and observation data.  

- IS_ZERG_HIGH_GROUND:  
  - Criteria: If the Zerglings are on the green high ground (above the cliff, with a clear boundary from the low ground), set to True; otherwise, set to False.  
  - Basis: Determined by analyzing the Zerglings' positions via RGB image.  

- IS_ZERG_NEAR:  
  - Criteria: If at least 5 Zerglings are within the attack range (≤7) or attempting to surround the Colossus, set to True; otherwise, set to False.  
  - Basis: Determined through observation data (distance and position) and RGB image analysis.  

- IS_AT_CLIFF:  
  - Condition: If the Colossus is near the edge of a cliff, the condition is True; otherwise, it is False. It is important to distinguish between a cliff and a boundary. When the Colossus is near the red boundary line, it is not considered a cliff. A cliff is the transitional area between the highland and the lowland.  
  - Determination Basis: This is analyzed through the RGB image and the positional information of the Colossus. 

- IS_COLOSSUS_WEAPON_COOLDOWN:
    - Condition: If the Colossus's weapon cooldown time is greater than 0, the condition is True; otherwise, it is False.  
    - Determination Basis: This is determined by analyzing the Colossus's weapon cooldown time.

- IS_COLOSSUS_HEALTH_LOW:  
    - Condition: If the Colossus's health is below 60%, this condition is True; otherwise, it is False.  
    - Determination Basis: Determined by observing the Colossus's health.  

4.2 Action Decision Rules

- Special Scenarios：
 rule1 ：As long as the weapon is in cooldown, the priority of movement actions is higher than attack actions. Movement > Attack.
 rule2：When the Colossus is about to be surrounded by enemies, regardless of the weapon's status, it should prioritize moving to escape the encirclement.
 rule3：Early Game Phase (<6s), at the start of the game, the Colossus should prioritize moving to the right to increase the distance from the narrow pathway on the left and wait for all Zerglings to reach the high ground. During this phase, movement takes priority over attacking.

- Situation 1: The Colossus and Zerglings are in different areas

  - Weapon not on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = False):
    - Priority: Attack. The Colossus should immediately attack enemies, leveraging its ranged advantage to maximize damage.
  - Weapon on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = True):
    - Priority: Move. The Colossus should create distance from the enemies to avoid being approached and attack once the weapon cooldown ends.

- Situation 2: The Colossus and Zerglings are in the same area

  - Enemies are near and about to attack (IS_ZERG_NEAR = True, Zerglings approaching):
    - Priority: Move. Regardless of whether the weapon is on cooldown, the Colossus should immediately retreat, quickly creating distance from the enemies to avoid being surrounded. This ensures it can fully leverage its ranged advantage and avoid the disadvantage of close combat. Once out of danger, it can then look for opportunities to counterattack.

  - Enemies are not near (IS_ZERG_NEAR = False):
    - Weapon not on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = False):
      - Priority: Attack. The Colossus should take the opportunity to attack enemies and weaken their forces.
    - Weapon on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = True):
      - Priority: Move. The Colossus should reposition to maintain a safe distance while waiting for the weapon cooldown to end.

- Situation 3 ：When IS_COLOSSUS_HEALTH_LOW = True:
    -Priority Strategy:
        - Avoid direct engagement with Zerglings and maintain maximum attack range.
        - Utilize terrain advantages and adopt hit-and-run tactics, moving flexibly between high and low ground.
        - Priority: Movement > Attack

5 Integrated Strategy and Decision Framework
1. Strategy Suggestions

text
    1.1 Common Game Strategies
        - At the start of the game, the Colossus should first move to the right, keeping its Y-coordinate stable while gradually increasing its X-coordinate. This allows the Colossus to maintain distance from the Zerglings climbing up to the high ground and remain positioned on the high ground to wait for most Zerglings to pass through the narrow pathway on the left. This maneuver prevents the Colossus from descending to the low ground too early, avoiding the risk of being surrounded by Zerglings that have not yet reached the high ground.
        - Once most enemy units have entered the high ground, the Colossus should increase its Y-coordinate to retreat to the low ground via the cliff, avoiding encirclement by Zerglings on the high ground.

    1.2 Emergency Response Strategies:
        - If encircled on the high ground, the Colossus should drastically increase the Y-coordinate to quickly escape the high ground and retreat to the low ground.
        - If encircled on the low ground, the Colossus should drastically decrease the Y-coordinate to quickly escape the low ground and retreat to the high ground.

2. Movement and Attack Decision

    2.1 When Movement Is Required:

    - Phase Judgment:  
      - Determine the current phase based on game time and the positions of both allied and enemy units.  

    - Evaluate Priorities:  
      - Decide if movement takes priority over attacking (e.g., prioritize movement when the weapon is on cooldown or there is high risk).  

     - Analyze Enemy Movements:  
      - Observe the Zerglings’ movements to predict their attack paths.  
      - Use the RGB image and the Cliff Barrier Delay Effect to choose the safest movement path (e.g., move down-right or up-right).  

    - Analyze Movement Along X and Y Axes:  
      - X-Axis Movement:  
        - Increase X value (move right): Create distance from the left-side pathway and maintain horizontal spacing from Zerglings.  
        - Decrease X value (move left): Rarely recommended unless repositioning to a strategic location.  
      - Y-Axis Movement:  
        - Decrease Y value (move upward): Retreat from the low ground to the high ground, leveraging the cliff to block enemies.  
        - Increase Y value (move downward): Retreat from the high ground to the low ground to delay enemy advances.  


    2.2 When Attacking Is Required:

    - Target Selection:  
      - Prioritize the densest groups of enemies to maximize splash damage.  
      - Focus fire on high-health enemy units that pose the greatest threat to the Colossus.  

    - Determine the Best Attack Point:  
      - Aim for the densest clusters of Zerglings to reduce their numbers and weaken their combat strength.  

6.Analysis:
In your response, please follow the format outlined in the example output, including section titles, numbering, and the terminology used. Ensure that you use the same variable names and terms specified in the decision process and chain of thought.

text
The output should include:

  1. Decision Process

     1.1 Condition Assessment:

       - `IS_COLOSSUS_HIGH_GROUND = True/False`
       - `IS_ZERGLING_HIGH_GROUND = True/False`
       - `IS_ZERGLING_NEAR_COLOSSUS = True/False`
       - `IS_COLOSSUS_AT_CLIFF = True/False`
       - `IS_COLOSSUS_WEAPON_COOLDOWN = True/False`
       - `IS_COLOSSUS_HEALTH_LOW = True/False'

     1.2 Action Decision Rules Brief Recap
         - Special Scenario： 
           - When the weapon is on cooldown: Move > Attack  
           - When encirclement is imminent, prioritize movement to escape, regardless of weapon cooldown status.
           - Early game (<6s), prioritize moving right to distance from the left pathway and wait for Zerglings on the high ground; movement > attack.            

        - Scenario 1: The Colossus and Zerglings are in different areas  
           - Weapon not on cooldown: Prioritize attacking, leveraging range advantage to maximize damage.  
           - Weapon on cooldown: Prioritize moving to create distance, then attack after cooldown ends.

        - Scenario 2: The Colossus and Zerglings are in the same area  
           - Enemies are close: Prioritize moving to avoid melee combat, then counterattack.  
           - Enemies are not close:  
             - Weapon not on cooldown: Prioritize attacking to weaken enemy forces.  
             - Weapon on cooldown: Prioritize moving to maintain a safe distance.  

        - Scenario 3: The Colossus is low on health (IS_COLOSSUS_HEALTH_LOW = True)  
           - Enemies are close:  
             - Prioritize moving to avoid direct engagement, leveraging terrain to create distance and delay pursuit.  
             - Once a safe position is reached, counterattack cautiously while maintaining maximum attack range.  
           - Enemies are not close:  
             - Weapon not on cooldown: Focus on targeting the densest enemy clusters from a safe position to reduce their numbers.  
             - Weapon on cooldown: Continue repositioning to maintain distance and avoid being flanked.  

     1.3 Judgment Based on 4.2 Action Decision Rules:

        -Provide movement and attack priorities, e.g., attack > move or move > attack.
        -Note:
            -Special rules, Scenario 1, Scenario 2, or Scenario 3 should align with only one scenario that best fits the current situation.
            -Always prioritize special rules (such as weapon cooldown or imminent encirclement). If none apply, proceed to select an action rule based on Scenario 1, Scenario 2, or Scenario 3.

  2 Integrated Strategy and Decision Framework

    2.1 Strategy Suggestions

   - Current Phase Determination:
       Dynamically determine the strategy based on time, the Colossus’s position, and Zergling distribution. Key factors include time range, the Colossus’s X/Y coordinates, and enemy movement trends.            

   - Consider Common Action Recommendations::

    - Early Game Observation Phase: Initially, keep the X-coordinate steady while gradually increasing the Y-coordinate to monitor enemy movements. Utilize the Colossus’s range advantage to weaken enemy forces.
    - Emergency Evacuation Phase:
        - Encircled on High Ground: The Colossus should drastically increase the Y-coordinate to quickly retreat to the low ground and escape the encirclement.
        - Encircled on Low Ground: The Colossus should drastically decrease the Y-coordinate to swiftly return to the high ground and avoid threats.
    - Late-Game Kite-and-Pull Phase: In the late game, the Colossus should maneuver flexibly near the cliff, leveraging the terrain to force enemy repositioning. Focus on ranged attacks to weaken enemy forces and maintain control of the battlefield.

    2.2 Movement Decision Analysis

    - Enemy Approach Path Prediction:  
      Analyze the primary approach paths of enemy units and identify potential threats.  

    - Movement Target Suggestions:  
      List multiple candidate movement points, including defensive positions on the high ground, safe retreat points on the low ground, and avoidance points.  
      Specify the most suitable movement target (e.g., a safe spot on the high ground or near the cliff on the low ground).  
      Explain the reasoning behind the selection (e.g., reducing the risk of being surrounded or gaining a better position for attacks).  

    2.3 Attack Decision Analysis

    - Target Prioritization:  
      Prioritize attacking dense enemy clusters to maximize splash damage.  
      If threatening units are approaching, prioritize targeting those closest to the Colossus to ensure its safety.  

    - Attack Target Selection:  
      Identify multiple candidate attack points (e.g., the center of a dense enemy group or the closest group to the Colossus).  
      Specify the optimal attack point and provide reasoning (e.g., to maximize damage or avoid being overwhelmed).


7.Action Output：
    Team Colossus-1:
      Action steps.
"""

    self.eip = \
        """
    Game Info
      Time: 0:05

    Team Colossus-1 Info:
      Team minimap position: [31, 27]
      Controlled Team Units:
            Unit: Colossus, Tag: 0x100000001, ScreenPos: [150, 123], Health: 350 (100%), Weapon Cooldown Time: 0.91s 
      Nearby Enemy Units:
            Unit: Zergling, Tag: 0x1002c0001, ScreenPos: [52, 164], Distance: 10, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100180001, ScreenPos: [79, 159], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100440001, ScreenPos: [112, 140], Distance: 4, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100640001, ScreenPos: [38, 176], Distance: 12, Health: 21 (60%)
            Unit: Zergling, Tag: 0x100700001, ScreenPos: [52, 178], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100780001, ScreenPos: [42, 169], Distance: 11, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100540001, ScreenPos: [49, 171], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100500001, ScreenPos: [91, 146], Distance: 6, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100800001, ScreenPos: [83, 147], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100280001, ScreenPos: [96, 151], Distance: 6, Health: 21 (60%)
            Unit: Zergling, Tag: 0x1000c0001, ScreenPos: [59, 162], Distance: 9, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100040001, ScreenPos: [93, 157], Distance: 6, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100340001, ScreenPos: [67, 172], Distance: 9, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1001c0001, ScreenPos: [72, 161], Distance: 8, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100600001, ScreenPos: [76, 152], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100140001, ScreenPos: [86, 162], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1004c0001, ScreenPos: [117, 145], Distance: 3, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100400001, ScreenPos: [66, 164], Distance: 9, Health: 21 (60%)
            Unit: Zergling, Tag: 0x100200001, ScreenPos: [78, 166], Distance: 8, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100240001, ScreenPos: [104, 144], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100100001, ScreenPos: [59, 172], Distance: 10, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1003c0001, ScreenPos: [86, 154], Distance: 7, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100480001, ScreenPos: [46, 159], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1006c0001, ScreenPos: [69, 155], Distance: 8, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100580001, ScreenPos: [59, 154], Distance: 9, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100740001, ScreenPos: [130, 134], Distance: 2, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1005c0001, ScreenPos: [97, 142], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100380001, ScreenPos: [39, 183], Distance: 12, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100080001, ScreenPos: [106, 151], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100300001, ScreenPos: [124, 137], Distance: 2, Health: 35 (100%)

        Relevant Knowledge:
            Protoss.Colossus
                The large quad-legged vehicle fires lasers in a splash pattern well-suited to destroying swarms of weaker units. This unit can also traverse differences in terrain height due to its long legs, and will appear to step over ledges and other obstacles due to the inverse kinematics system.
                Unit properties: ['ground', 'air', 'armored', 'massive', 'mechanical']
                Weapon info: Attack Range 7, target: ['ground'], anti: ['light'], DPS(damage per second) 13, DPS-anti 20
                Unit abilities:

            Zerg.Zergling
                Fast but weak melee attacker ideal for swarming attacks in large numbers.
                Unit properties: ['ground', 'light', 'biological']
                Weapon info: Attack Range 1, target: ['ground'], DPS(damage per second) 10


        Valid Actions:
            Team Colossus-1 Valid Actions:
                <Attack_Unit(tag)>
                <Move_Screen(screen)>

        Action Args:
            (1) tag: tag refers to a hexadecimal number, shape as 0x000000000.
            (2) screen: screen refers to a screen coordinate, shape as [x, y], where x and y range from 0 to 256.
            (3) minimap: minimap refers to a minimap coordinate, shape as [x, y], where x and y range from 0 to 64.

        For example, when you want to use an action like <Action_Name(tag, screen)>, you should output like <Action_Name(0x100580001, [37, 55])>; when you want to use an action like <Action_Name(screen)>, you should output like <Action_Name([66, 78])>. Make sure to see whether an action uses screen coordinates or minimap coordinates. If an action name is XXXX_Screen, it uses screen coordinates; if an action name is XXXX_Minimap, it uses minimap coordinates.

        Last Step Actions:
            Team Colossus-1:
                <Move_Screen([160, 130])>
                <Attack_Unit(0x1007c0001)>
                <Attack_Unit(0x100740001)>

        You need to confirm whether the previous action finished executing, and based on this, determine whether to continue the old strategy or immediately take other actions.

        Give each team no more than 2 actions, among which activity release should usually before move and attack.
        Now, start generating your analysis and actions:
        """

    self.eop = \
        """

Analysis:

1. Decision Process

   1.1 Condition Assessment

   text
    - IS_COLOSSUS_HIGH_GROUND = True
       The Colossus is currently on the high ground, confirmed via the image and coordinates.
    - IS_ZERGLING_HIGH_GROUND = True
       Zerglings are primarily concentrated on the high ground, in the same area as the Colossus.
    - IS_ZERGLING_NEAR_COLOSSUS = True
       Multiple Zerglings are close to the Colossus and within its attack range (≤7), posing an immediate threat.
    - IS_COLOSSUS_AT_CLIFF = False
       The Colossus is near the red boundary of the map, not near the cliff (the border between green and gray areas).
    - IS_COLOSSUS_WEAPON_COOLDOWN = True
       The weapon is on cooldown and cannot immediately attack enemy units.
    - IS_COLOSSUS_HEALTH_LOW = False  
       The Colossus's health is at 100%, with no low-health threat.

   1.2 Action Decision Rules

   Decision basis:
   1. Special Rules:
   - When the weapon is on cooldown, movement takes precedence over attack.
   - When surrounded or under close threat, prioritize movement to escape danger.

   text
    2. Scenario Analysis:
        - Current scenario: The Colossus and Zerglings are in the same area, with Zerglings approaching the Colossus:
            - The weapon is currently on cooldown, and the large number of fast-approaching Zerglings necessitates movement to avoid being surrounded and to create safe distance.
        - Terrain condition:
            - Being close to the red boundary of the map limits the Colossus’s maneuverability, further highlighting the importance of movement.

   1.3 Judgment Based on Action Decision Rules

   • Prioritize matching the special rule: Weapon cooldown (Move > Attack). The current scenario matches the special rule, prioritizing movement to retreat, avoid being surrounded, and gain time for the weapon cooldown.
   • Move > Attack
     The high ground space is insufficient, and with enemies advancing and the weapon on cooldown, the Colossus must immediately retreat to the low ground, using the cliff to delay Zergling pursuit and buy time for the cooldown.

2. Integrated Strategy and Decision Framework

   2.1 Strategy Suggestions

   text
   Current Strategy: Emergency Retreat to Low Ground
    - The Colossus and Zerglings are both on the high ground. Given the large number of enemies and their proximity, the situation is critical.
    - The Colossus must quickly retreat to the low ground (gray area), using the cliff barrier to delay Zerglings' pursuit. This retreat will buy time for the weapon to cool down and create a safe distance to regain battlefield control.                

   2.2 Movement Decision Analysis

   -Enemy Approach Path Prediction
   - Zerglings are advancing from the left pathway on the high ground, attempting to encircle the Colossus.
   - If the Colossus does not retreat immediately, it risks being cornered near the red boundary, losing space to maneuver.

   • Movement Target Suggestions

     • Retreat directly to screen coordinates [160, 200], a safe area near the cliff on the low ground.
     • Crossing the cliff to the low ground will delay Zerglings' pursuit, providing more survival space and reducing the risk of melee attacks.

   • Reason:

     • Directly retreating to the low ground avoids encirclement by Zerglings. The cliff barrier delays enemy pursuit, ensuring the Colossus's safety.

   2.3 Attack Decision Analysis

   text
    - Target Prioritization:
        - Once the weapon cooldown ends, prioritize attacking the nearest Zerglings to reduce immediate threats and maximize splash damage.
   
    - Attack Target Selection:
        - The closest Zerglings (e.g., Zergling, Tag: 0x1004c0001 and 0x100740001) pose a direct threat to the Colossus.
        - After repositioning on the low ground, prioritize attacking dense clusters of pursuing Zerglings.

3 .Actions:
Team Colossus-1:
<Move_Screen([160, 200])> # Retreat directly from the high ground to the low ground, avoiding Zergling encirclement and using the cliff to delay pursuit.
<Attack_Unit(0x1004c0001)> # After the weapon cooldown ends, prioritize attacking the nearest Zergling to reduce the threat.
<Attack_Unit(0x100740001)> # Continue attacking other nearby Zerglings, using splash damage to weaken the enemy forces.
"""

text
    # Part 2
    if self.config.ENABLE_COMMUNICATION:
        self.sp += \
            """
            4.Communication Output
              If there is Available Communicate Target, you should keep communicating with them by Communication functions. For example, if 'Commander' and 'CombatGroup4' in Available Communicate Target, you can output as:

              Communications:
                <MessageTo(Commander, '''xxxxxxxxxx''')>
                <MessageTo(CombatGroup4, '''xxxxxxxxxx''')>
            """
        self.eip += \
            """
            Communication:
              From Commander: 
                Your task is to attack the enemy workers of an enemy base near minimap [48,32]. Intelligence shows that two enemy Queens are located on the minimap [44,32]. Try to avoid being detected by enemy Queens before arriving.

            Available Communication Tragets:
              Commander: Protoss military supreme commander. Responsible for making macro decision through communication, and controls nexus for massrecall for tactical objectives.
            Available Communication Functions:
              <MessageTo(AgentName, message)>
              <MessageTo(ChannelName, message)>
              <ListenTo(ChannelName)>
            Args explanation:
              (1)AgentName: refers to a name mentioned in Available Communication Tragets.
              (2)ChannelName: shape as Channel-i, i refers to an integer.
              (2)message: any text wrapped between ''' and '''.
            """
        self.eop += \
            """
            Communications:
                <MessageTo(Commander, '''Copy that, we have arrived enemy base, and started attack enemy workers''')>
            """

    # Part 3
    self.eip += \
        f"""

Give each team no more than {self.config.MAX_NUM_ACTIONS} actions.
Now, start generating your analysis and actions:
"""

class CommanderPrompt(BasePrompt): # TODO: Design a prompt specifically for the supreme military commander
def init(self, name, log_id, config):
super(CombatGroupPrompt, self).init()
self.name = name
self.config = config
self.log_id = log_id
# self.sp = ''
# self.eip = ''
# self.eop = ''

class DeveloperPrompt(BasePrompt): # TODO: Design a prompt specifically for the supreme logistics commander
def init(self, name, log_id, config):
super(CombatGroupPrompt, self).init()
self.name = name
self.config = config
self.log_id = log_id
# self.sp = ''
# self.eip = ''
# self.eop = ''

PROTOSS_FACTORY = {
'default': CombatGroupPrompt,
'commander': CommanderPrompt,
'developer': DeveloperPrompt,
}
TERRAN_FACTORY = {}
ZERG_FACTORY = {}

FACTORY = {
'protoss': PROTOSS_FACTORY,
'terran': TERRAN_FACTORY,
'zerg': ZERG_FACTORY,
}

if name == "main":
from llm_pysc2.agents.configs.config import ProtossAgentConfig

text
config = ProtossAgentConfig()
prompt = CombatGroupPrompt('CombatGroup1', log_id=0, config=config)

print("--" * 25 + "System Prompt" + "--" * 25)
print(prompt.sp)
print("--" * 25 + "Example Input Prompt" + "--" * 25)
print(prompt.eip)
print("--" * 25 + "Example Output Prompt" + "--" * 25)
print(prompt.eop)

这个提示词是visual-cot的提示词，请你理解
D:\Anaconda\envs\starcraftii0828\python.exe "E:\code\llm-pysc2-develop 09242320\llm-pysc2-develop 10092340\llm-pysc2-develop\llm_pysc2\lib\llm_prompt.py"
pygame 2.6.0 (SDL 2.28.4, Python 3.9.19)
Hello from the pygame community. https://www.pygame.org/contribute.html
--------------------------------------------------System Prompt--------------------------------------------------

text
1. Role Description
 You are a Protoss frontline commander, controls several Stalkers. Responsible for providing cover for the main force and restraining enemy forces..
 In this mission, your primary objective is to leverage the Colossus's ability to traverse between high and low terrain to counter the numerical advantage of 32 Zerglings. Your top priority is to capitalize on this mobility advantage, maneuvering between the two terrains to delay the Zerglings' approach, engage in ranged attacks on pursuing Zerglings, and ensure the survival of the Colossus.

2.Map Information

  2.1 Coordinate System:
- The screen coordinate system is a 2D grid with a range of [0, 256] × [0, 256].
- Each grid cell measures 32 × 32 pixels, resulting in a total of 8 × 8 = 64 grid cells.
- The origin point (0, 0) is at the top-left corner, while (256, 256) is at the bottom-right corner.

  2.2 Terrain Information:  
- Pathway (connecting the high ground and low ground):  
  The map consists of an upper platform (high ground) and a lower platform (low ground), connected by a narrow pathway on the left side.

- High Ground (light green area):  
  The high ground is characterized by light green grass textures and is located on the upper platform.

- Low Ground (dark gray area):  
  The low ground is characterized by dark gray concrete textures and is located on the lower platform.

- Impassable Area (black background):  
  The edges of the map consist of a black background, which is impassable. The Colossus must avoid entering these areas.

- Cliff (light gray vertical surface):  
  The cliff separates the high ground and low ground. The Colossus can freely traverse the cliff, but Zerglings can only access the high ground through the narrow pathway on the left.  
  Only the boundary between the high ground and low ground is considered a cliff; other deep red boundaries are impassable edges of the map.

- Boundary:  
  The boundaries between the high ground and the black background, as well as between the low ground and the black background, are considered impassable edges of the map.  
  The boundaries are represented by deep red lines on the map.

3. Key Victory Points:
  3.1 Maintain Maximum Attack Range:
    - Ideally, the Colossus should stay within its maximum attack range (range 7) to fully utilize its long-range attack advantage while minimizing the risk of melee attacks from enemies.
  3.2 Focus Fire on Dense Enemy Clusters:
    - Target the center of the densest enemy groups to maximize splash damage and quickly reduce enemy numbers.
  3.3 Cliff Barrier Delay Effect：

    The Cliff Barrier Delay Effect refers to leveraging the terrain characteristics of cliffs, where enemy units cannot directly cross the cliff, while allied units can freely traverse it. This forces the enemy to take a detour when transitioning between high and low ground, buying time for allied units to reposition, cool down weapons, and strike effectively.

    - When enemies are heavily concentrated on the high ground: 
      - If allied units are also on the high ground but risk being surrounded, it is recommended to retreat to the bottom-right low ground. This utilizes the cliff as a barrier to delay enemy pursuit, enabling ranged attacks on the enemies while maintaining battlefield control.

    - When enemies are heavily concentrated on the low ground: 
      - If allied units are also on the low ground but risk being surrounded, it is recommended to retreat to the top-right high ground. The cliff barrier forces enemies to take a detour, providing time for allied units to reposition and counterattack effectively.

4. Decision Process
Based on the observed data of the Colossus and Zerglings, as well as the RGB image of the screen, use the following decision rules to determine action priorities:

text
4.1 Condition Assessment Rules:  

- IS_HIGH_GROUND:  
  - Criteria: If the Colossus is on the green high ground (above the cliff, with a clear boundary from the low ground), set to True; otherwise, set to False.  
  - Basis: Determined by verifying the Colossus's position in the green area via RGB image and observation data.  

- IS_ZERG_HIGH_GROUND:  
  - Criteria: If the Zerglings are on the green high ground (above the cliff, with a clear boundary from the low ground), set to True; otherwise, set to False.  
  - Basis: Determined by analyzing the Zerglings' positions via RGB image.  

- IS_ZERG_NEAR:  
  - Criteria: If at least 5 Zerglings are within the attack range (≤7) or attempting to surround the Colossus, set to True; otherwise, set to False.  
  - Basis: Determined through observation data (distance and position) and RGB image analysis.  

- IS_AT_CLIFF:  
  - Condition: If the Colossus is near the edge of a cliff, the condition is True; otherwise, it is False. It is important to distinguish between a cliff and a boundary. When the Colossus is near the red boundary line, it is not considered a cliff. A cliff is the transitional area between the highland and the lowland.  
  - Determination Basis: This is analyzed through the RGB image and the positional information of the Colossus. 

- IS_COLOSSUS_WEAPON_COOLDOWN:
    - Condition: If the Colossus's weapon cooldown time is greater than 0, the condition is True; otherwise, it is False.  
    - Determination Basis: This is determined by analyzing the Colossus's weapon cooldown time.

- IS_COLOSSUS_HEALTH_LOW:  
    - Condition: If the Colossus's health is below 60%, this condition is True; otherwise, it is False.  
    - Determination Basis: Determined by observing the Colossus's health.  

4.2 Action Decision Rules

- Special Scenarios：
 rule1 ：As long as the weapon is in cooldown, the priority of movement actions is higher than attack actions. Movement > Attack.
 rule2：When the Colossus is about to be surrounded by enemies, regardless of the weapon's status, it should prioritize moving to escape the encirclement.
 rule3：Early Game Phase (<6s), at the start of the game, the Colossus should prioritize moving to the right to increase the distance from the narrow pathway on the left and wait for all Zerglings to reach the high ground. During this phase, movement takes priority over attacking.

- Situation 1: The Colossus and Zerglings are in different areas

  - Weapon not on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = False):
    - Priority: Attack. The Colossus should immediately attack enemies, leveraging its ranged advantage to maximize damage.
  - Weapon on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = True):
    - Priority: Move. The Colossus should create distance from the enemies to avoid being approached and attack once the weapon cooldown ends.

- Situation 2: The Colossus and Zerglings are in the same area

  - Enemies are near and about to attack (IS_ZERG_NEAR = True, Zerglings approaching):
    - Priority: Move. Regardless of whether the weapon is on cooldown, the Colossus should immediately retreat, quickly creating distance from the enemies to avoid being surrounded. This ensures it can fully leverage its ranged advantage and avoid the disadvantage of close combat. Once out of danger, it can then look for opportunities to counterattack.

  - Enemies are not near (IS_ZERG_NEAR = False):
    - Weapon not on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = False):
      - Priority: Attack. The Colossus should take the opportunity to attack enemies and weaken their forces.
    - Weapon on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = True):
      - Priority: Move. The Colossus should reposition to maintain a safe distance while waiting for the weapon cooldown to end.

- Situation 3 ：When IS_COLOSSUS_HEALTH_LOW = True:
    -Priority Strategy:
        - Avoid direct engagement with Zerglings and maintain maximum attack range.
        - Utilize terrain advantages and adopt hit-and-run tactics, moving flexibly between high and low ground.
        - Priority: Movement > Attack

5 Integrated Strategy and Decision Framework
1. Strategy Suggestions

text
    1.1 Common Game Strategies
        - At the start of the game, the Colossus should first move to the right, keeping its Y-coordinate stable while gradually increasing its X-coordinate. This allows the Colossus to maintain distance from the Zerglings climbing up to the high ground and remain positioned on the high ground to wait for most Zerglings to pass through the narrow pathway on the left. This maneuver prevents the Colossus from descending to the low ground too early, avoiding the risk of being surrounded by Zerglings that have not yet reached the high ground.
        - Once most enemy units have entered the high ground, the Colossus should increase its Y-coordinate to retreat to the low ground via the cliff, avoiding encirclement by Zerglings on the high ground.

    1.2 Emergency Response Strategies:
        - If encircled on the high ground, the Colossus should drastically increase the Y-coordinate to quickly escape the high ground and retreat to the low ground.
        - If encircled on the low ground, the Colossus should drastically decrease the Y-coordinate to quickly escape the low ground and retreat to the high ground.

2. Movement and Attack Decision

    2.1 When Movement Is Required:

    - Phase Judgment:  
      - Determine the current phase based on game time and the positions of both allied and enemy units.  

    - Evaluate Priorities:  
      - Decide if movement takes priority over attacking (e.g., prioritize movement when the weapon is on cooldown or there is high risk).  

     - Analyze Enemy Movements:  
      - Observe the Zerglings’ movements to predict their attack paths.  
      - Use the RGB image and the Cliff Barrier Delay Effect to choose the safest movement path (e.g., move down-right or up-right).  

    - Analyze Movement Along X and Y Axes:  
      - X-Axis Movement:  
        - Increase X value (move right): Create distance from the left-side pathway and maintain horizontal spacing from Zerglings.  
        - Decrease X value (move left): Rarely recommended unless repositioning to a strategic location.  
      - Y-Axis Movement:  
        - Decrease Y value (move upward): Retreat from the low ground to the high ground, leveraging the cliff to block enemies.  
        - Increase Y value (move downward): Retreat from the high ground to the low ground to delay enemy advances.  


    2.2 When Attacking Is Required:

    - Target Selection:  
      - Prioritize the densest groups of enemies to maximize splash damage.  
      - Focus fire on high-health enemy units that pose the greatest threat to the Colossus.  

    - Determine the Best Attack Point:  
      - Aim for the densest clusters of Zerglings to reduce their numbers and weaken their combat strength.  

6.Analysis:
In your response, please follow the format outlined in the example output, including section titles, numbering, and the terminology used. Ensure that you use the same variable names and terms specified in the decision process and chain of thought.

text
The output should include:

  1. Decision Process

     1.1 Condition Assessment:

       - `IS_COLOSSUS_HIGH_GROUND = True/False`
       - `IS_ZERGLING_HIGH_GROUND = True/False`
       - `IS_ZERGLING_NEAR_COLOSSUS = True/False`
       - `IS_COLOSSUS_AT_CLIFF = True/False`
       - `IS_COLOSSUS_WEAPON_COOLDOWN = True/False`
       - `IS_COLOSSUS_HEALTH_LOW = True/False'

     1.2 Action Decision Rules Brief Recap
         - Special Scenario： 
           - When the weapon is on cooldown: Move > Attack  
           - When encirclement is imminent, prioritize movement to escape, regardless of weapon cooldown status.
           - Early game (<6s), prioritize moving right to distance from the left pathway and wait for Zerglings on the high ground; movement > attack.            

        - Scenario 1: The Colossus and Zerglings are in different areas  
           - Weapon not on cooldown: Prioritize attacking, leveraging range advantage to maximize damage.  
           - Weapon on cooldown: Prioritize moving to create distance, then attack after cooldown ends.

        - Scenario 2: The Colossus and Zerglings are in the same area  
           - Enemies are close: Prioritize moving to avoid melee combat, then counterattack.  
           - Enemies are not close:  
             - Weapon not on cooldown: Prioritize attacking to weaken enemy forces.  
             - Weapon on cooldown: Prioritize moving to maintain a safe distance.  

        - Scenario 3: The Colossus is low on health (IS_COLOSSUS_HEALTH_LOW = True)  
           - Enemies are close:  
             - Prioritize moving to avoid direct engagement, leveraging terrain to create distance and delay pursuit.  
             - Once a safe position is reached, counterattack cautiously while maintaining maximum attack range.  
           - Enemies are not close:  
             - Weapon not on cooldown: Focus on targeting the densest enemy clusters from a safe position to reduce their numbers.  
             - Weapon on cooldown: Continue repositioning to maintain distance and avoid being flanked.  

     1.3 Judgment Based on 4.2 Action Decision Rules:

        -Provide movement and attack priorities, e.g., attack > move or move > attack.
        -Note:
            -Special rules, Scenario 1, Scenario 2, or Scenario 3 should align with only one scenario that best fits the current situation.
            -Always prioritize special rules (such as weapon cooldown or imminent encirclement). If none apply, proceed to select an action rule based on Scenario 1, Scenario 2, or Scenario 3.

  2 Integrated Strategy and Decision Framework

    2.1 Strategy Suggestions

   - Current Phase Determination:
       Dynamically determine the strategy based on time, the Colossus’s position, and Zergling distribution. Key factors include time range, the Colossus’s X/Y coordinates, and enemy movement trends.            

   - Consider Common Action Recommendations::

    - Early Game Observation Phase: Initially, keep the X-coordinate steady while gradually increasing the Y-coordinate to monitor enemy movements. Utilize the Colossus’s range advantage to weaken enemy forces.
    - Emergency Evacuation Phase:
        - Encircled on High Ground: The Colossus should drastically increase the Y-coordinate to quickly retreat to the low ground and escape the encirclement.
        - Encircled on Low Ground: The Colossus should drastically decrease the Y-coordinate to swiftly return to the high ground and avoid threats.
    - Late-Game Kite-and-Pull Phase: In the late game, the Colossus should maneuver flexibly near the cliff, leveraging the terrain to force enemy repositioning. Focus on ranged attacks to weaken enemy forces and maintain control of the battlefield.

    2.2 Movement Decision Analysis

    - Enemy Approach Path Prediction:  
      Analyze the primary approach paths of enemy units and identify potential threats.  

    - Movement Target Suggestions:  
      List multiple candidate movement points, including defensive positions on the high ground, safe retreat points on the low ground, and avoidance points.  
      Specify the most suitable movement target (e.g., a safe spot on the high ground or near the cliff on the low ground).  
      Explain the reasoning behind the selection (e.g., reducing the risk of being surrounded or gaining a better position for attacks).  

    2.3 Attack Decision Analysis

    - Target Prioritization:  
      Prioritize attacking dense enemy clusters to maximize splash damage.  
      If threatening units are approaching, prioritize targeting those closest to the Colossus to ensure its safety.  

    - Attack Target Selection:  
      Identify multiple candidate attack points (e.g., the center of a dense enemy group or the closest group to the Colossus).  
      Specify the optimal attack point and provide reasoning (e.g., to maximize damage or avoid being overwhelmed).


7.Action Output：
    Team Colossus-1:
      Action steps.

--------------------------------------------------Example Input Prompt--------------------------------------------------

text
    Game Info
      Time: 0:05

    Team Colossus-1 Info:
      Team minimap position: [31, 27]
      Controlled Team Units:
            Unit: Colossus, Tag: 0x100000001, ScreenPos: [150, 123], Health: 350 (100%), Weapon Cooldown Time: 0.91s 
      Nearby Enemy Units:
            Unit: Zergling, Tag: 0x1002c0001, ScreenPos: [52, 164], Distance: 10, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100180001, ScreenPos: [79, 159], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100440001, ScreenPos: [112, 140], Distance: 4, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100640001, ScreenPos: [38, 176], Distance: 12, Health: 21 (60%)
            Unit: Zergling, Tag: 0x100700001, ScreenPos: [52, 178], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100780001, ScreenPos: [42, 169], Distance: 11, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100540001, ScreenPos: [49, 171], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100500001, ScreenPos: [91, 146], Distance: 6, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100800001, ScreenPos: [83, 147], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100280001, ScreenPos: [96, 151], Distance: 6, Health: 21 (60%)
            Unit: Zergling, Tag: 0x1000c0001, ScreenPos: [59, 162], Distance: 9, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100040001, ScreenPos: [93, 157], Distance: 6, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100340001, ScreenPos: [67, 172], Distance: 9, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1001c0001, ScreenPos: [72, 161], Distance: 8, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100600001, ScreenPos: [76, 152], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100140001, ScreenPos: [86, 162], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1004c0001, ScreenPos: [117, 145], Distance: 3, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100400001, ScreenPos: [66, 164], Distance: 9, Health: 21 (60%)
            Unit: Zergling, Tag: 0x100200001, ScreenPos: [78, 166], Distance: 8, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100240001, ScreenPos: [104, 144], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100100001, ScreenPos: [59, 172], Distance: 10, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1003c0001, ScreenPos: [86, 154], Distance: 7, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100480001, ScreenPos: [46, 159], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1006c0001, ScreenPos: [69, 155], Distance: 8, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100580001, ScreenPos: [59, 154], Distance: 9, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100740001, ScreenPos: [130, 134], Distance: 2, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1005c0001, ScreenPos: [97, 142], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100380001, ScreenPos: [39, 183], Distance: 12, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100080001, ScreenPos: [106, 151], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100300001, ScreenPos: [124, 137], Distance: 2, Health: 35 (100%)

        Relevant Knowledge:
            Protoss.Colossus
                The large quad-legged vehicle fires lasers in a splash pattern well-suited to destroying swarms of weaker units. This unit can also traverse differences in terrain height due to its long legs, and will appear to step over ledges and other obstacles due to the inverse kinematics system.
                Unit properties: ['ground', 'air', 'armored', 'massive', 'mechanical']
                Weapon info: Attack Range 7, target: ['ground'], anti: ['light'], DPS(damage per second) 13, DPS-anti 20
                Unit abilities:

            Zerg.Zergling
                Fast but weak melee attacker ideal for swarming attacks in large numbers.
                Unit properties: ['ground', 'light', 'biological']
                Weapon info: Attack Range 1, target: ['ground'], DPS(damage per second) 10


        Valid Actions:
            Team Colossus-1 Valid Actions:
                <Attack_Unit(tag)>
                <Move_Screen(screen)>

        Action Args:
            (1) tag: tag refers to a hexadecimal number, shape as 0x000000000.
            (2) screen: screen refers to a screen coordinate, shape as [x, y], where x and y range from 0 to 256.
            (3) minimap: minimap refers to a minimap coordinate, shape as [x, y], where x and y range from 0 to 64.

        For example, when you want to use an action like <Action_Name(tag, screen)>, you should output like <Action_Name(0x100580001, [37, 55])>; when you want to use an action like <Action_Name(screen)>, you should output like <Action_Name([66, 78])>. Make sure to see whether an action uses screen coordinates or minimap coordinates. If an action name is XXXX_Screen, it uses screen coordinates; if an action name is XXXX_Minimap, it uses minimap coordinates.

        Last Step Actions:
            Team Colossus-1:
                <Move_Screen([160, 130])>
                <Attack_Unit(0x1007c0001)>
                <Attack_Unit(0x100740001)>

        You need to confirm whether the previous action finished executing, and based on this, determine whether to continue the old strategy or immediately take other actions.

        Give each team no more than 2 actions, among which activity release should usually before move and attack.
        Now, start generating your analysis and actions:
        

Give each team no more than 3 actions.
Now, start generating your analysis and actions:

--------------------------------------------------Example Output Prompt--------------------------------------------------

Analysis:

1. Decision Process

   1.1 Condition Assessment

   text
    - IS_COLOSSUS_HIGH_GROUND = True
       The Colossus is currently on the high ground, confirmed via the image and coordinates.
    - IS_ZERGLING_HIGH_GROUND = True
       Zerglings are primarily concentrated on the high ground, in the same area as the Colossus.
    - IS_ZERGLING_NEAR_COLOSSUS = True
       Multiple Zerglings are close to the Colossus and within its attack range (≤7), posing an immediate threat.
    - IS_COLOSSUS_AT_CLIFF = False
       The Colossus is near the red boundary of the map, not near the cliff (the border between green and gray areas).
    - IS_COLOSSUS_WEAPON_COOLDOWN = True
       The weapon is on cooldown and cannot immediately attack enemy units.
    - IS_COLOSSUS_HEALTH_LOW = False  
       The Colossus's health is at 100%, with no low-health threat.

   1.2 Action Decision Rules

   Decision basis:
   1. Special Rules:
   - When the weapon is on cooldown, movement takes precedence over attack.
   - When surrounded or under close threat, prioritize movement to escape danger.

   text
    2. Scenario Analysis:
        - Current scenario: The Colossus and Zerglings are in the same area, with Zerglings approaching the Colossus:
            - The weapon is currently on cooldown, and the large number of fast-approaching Zerglings necessitates movement to avoid being surrounded and to create safe distance.
        - Terrain condition:
            - Being close to the red boundary of the map limits the Colossus’s maneuverability, further highlighting the importance of movement.

   1.3 Judgment Based on Action Decision Rules

   • Prioritize matching the special rule: Weapon cooldown (Move > Attack). The current scenario matches the special rule, prioritizing movement to retreat, avoid being surrounded, and gain time for the weapon cooldown.
   • Move > Attack
     The high ground space is insufficient, and with enemies advancing and the weapon on cooldown, the Colossus must immediately retreat to the low ground, using the cliff to delay Zergling pursuit and buy time for the cooldown.

2. Integrated Strategy and Decision Framework

   2.1 Strategy Suggestions

   text
   Current Strategy: Emergency Retreat to Low Ground
    - The Colossus and Zerglings are both on the high ground. Given the large number of enemies and their proximity, the situation is critical.
    - The Colossus must quickly retreat to the low ground (gray area), using the cliff barrier to delay Zerglings' pursuit. This retreat will buy time for the weapon to cool down and create a safe distance to regain battlefield control.                

   2.2 Movement Decision Analysis

   -Enemy Approach Path Prediction
   - Zerglings are advancing from the left pathway on the high ground, attempting to encircle the Colossus.
   - If the Colossus does not retreat immediately, it risks being cornered near the red boundary, losing space to maneuver.

   • Movement Target Suggestions

     • Retreat directly to screen coordinates [160, 200], a safe area near the cliff on the low ground.
     • Crossing the cliff to the low ground will delay Zerglings' pursuit, providing more survival space and reducing the risk of melee attacks.

   • Reason:

     • Directly retreating to the low ground avoids encirclement by Zerglings. The cliff barrier delays enemy pursuit, ensuring the Colossus's safety.

   2.3 Attack Decision Analysis

   text
    - Target Prioritization:
        - Once the weapon cooldown ends, prioritize attacking the nearest Zerglings to reduce immediate threats and maximize splash damage.
   
    - Attack Target Selection:
        - The closest Zerglings (e.g., Zergling, Tag: 0x1004c0001 and 0x100740001) pose a direct threat to the Colossus.
        - After repositioning on the low ground, prioritize attacking dense clusters of pursuing Zerglings.

3 .Actions:
Team Colossus-1:
<Move_Screen([160, 200])> # Retreat directly from the high ground to the low ground, avoiding Zergling encirclement and using the cliff to delay pursuit.
<Attack_Unit(0x1004c0001)> # After the weapon cooldown ends, prioritize attacking the nearest Zergling to reduce the threat.
<Attack_Unit(0x100740001)> # Continue attacking other nearby Zerglings, using splash damage to weaken the enemy forces.

进程已结束,退出代码0

请你帮助我分析，有什么提高精简的空间，有什么合适的思路呢？

第一：现在是1巨像对抗32小狗的情况，请结合星际争霸2的这方面知识来思考。
第二：我希望能够在此基础上改进再发一篇论文，方法或者说提示词内容有什么合适的改进思路，可以结合思维链方面思考。

第一：我认为强化学习不可行
第二：思维链是不是可以调整，更加适合这个任务
请你给出你的想法。

在提示词中能不能有多思维链，灵活切换，提高推理结果。

我希望的输出是，输出的攻击和移动指令能够有优先级排序；如果移动距离较远的话，可以中间插入空动作，<No_Operation()>来使用。

就是需要实现这样的效果， 同时还要有多思维链在其中，请你判断可行性以及给出具体思路。
4.2 Action Decision Rules

text
- Special Scenarios：
 rule1 ：As long as the weapon is in cooldown, the priority of movement actions is higher than attack actions. Movement > Attack.
 rule2：When the Colossus is about to be surrounded by enemies, regardless of the weapon's status, it should prioritize moving to escape the encirclement.
 rule3：Early Game Phase (<6s), at the start of the game, the Colossus should prioritize moving to the right to increase the distance from the narrow pathway on the left and wait for all Zerglings to reach the high ground. During this phase, movement takes priority over attacking.

- Situation 1: The Colossus and Zerglings are in different areas

  - Weapon not on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = False):
    - Priority: Attack. The Colossus should immediately attack enemies, leveraging its ranged advantage to maximize damage.
  - Weapon on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = True):
    - Priority: Move. The Colossus should create distance from the enemies to avoid being approached and attack once the weapon cooldown ends.

- Situation 2: The Colossus and Zerglings are in the same area

  - Enemies are near and about to attack (IS_ZERG_NEAR = True, Zerglings approaching):
    - Priority: Move. Regardless of whether the weapon is on cooldown, the Colossus should immediately retreat, quickly creating distance from the enemies to avoid being surrounded. This ensures it can fully leverage its ranged advantage and avoid the disadvantage of close combat. Once out of danger, it can then look for opportunities to counterattack.

  - Enemies are not near (IS_ZERG_NEAR = False):
    - Weapon not on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = False):
      - Priority: Attack. The Colossus should take the opportunity to attack enemies and weaken their forces.
    - Weapon on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = True):
      - Priority: Move. The Colossus should reposition to maintain a safe distance while waiting for the weapon cooldown to end.

- Situation 3 ：When IS_COLOSSUS_HEALTH_LOW = True:
    -Priority Strategy:
        - Avoid direct engagement with Zerglings and maintain maximum attack range.
        - Utilize terrain advantages and adopt hit-and-run tactics, moving flexibly between high and low ground.
        - Priority: Movement > Attack

下面给出一个以“思维链投票机制 + 事件提醒 + 提示词优化”为核心的改进方案示例，省去对“短期记忆”机制的大规模改动，重点在图像信息和Prompt层面完善。可供下一步研发和论文撰写时参考：

一、整体思路概览

1. 思维链投票机制

   • 目标：在同一帧（或同一个决策时刻），让大模型通过多条并行推理路径生成多个可能方案，再通过投票或打分的形式选出最优行动。
   • 价值：减少单次推理的偶发性错误，提高决策稳健性。

2. 事件提醒（Event Reminder）

   • 目标：为大模型提供特殊的高优先级信息触发点，比如“敌人已近身”、“血量告急”等，从而在思维链中优先处理。
   • 价值：让模型的决策及时关注关键事件，避免无关或冗余信息干扰。

3. 提示词优化（Prompt Engineering）

   • 目标：强化对图像信息的解释和使用；引导模型在思维链中对图像中的关键信息（地形、敌军分布）进行多步分析和投票，输出更准确的决策动作。
   • 价值：通过更精细的提示、示例和格式约束，让大模型深度利用视觉特征，不必大规模改动其他系统模块。

二、具体技术路径

以下给出一个从数据流到最终决策输出的技术流程示例，结合以上三大改进点逐步展开。

1. 数据流与基础结构

1. 环境交互

   • 依旧使用 LLM-PySC2 接口，从 StarCraft II 获取文本和图像观测（如单位位置、血量信息，以及处理过的RGB截图）。
   • 每隔一定时间步或每次需要动作决策时，收集最新观测并传入大模型做推理。

2. 图像预处理

   • 与之前类似，标注地图上的不可通行区域、悬崖位置、单位坐标等，让大模型能在 Prompt 中“一眼”看出关键信息。
   • 可使用 base64 格式或其他方案嵌入图片到大模型（若使用多模态 GPT-4 之类）。

2. 思维链投票机制

1. 多条推理路径

   • 当需要做决策时（如“是否撤退”或“是否攻击”），可向大模型发送同样的输入多次，或一次性提示大模型生成 N 个候选策略（譬如通过一个“请给出多个备选方案”的提示语）。
   • 每个候选方案都是一条思维链（Chain of Thought），从条件判断到最终行动输出都各自独立。

2. 投票/打分

   • 在同一个 Prompt 中，请大模型在最后阶段对这些候选方案进行 Self-consistency 投票（或者由一个“评估子步骤”来计算成功概率）。
   • 也可让环境或一个小模块“模拟”或“对比评估”后再选出最高分方案执行。
   • 示例：
     • 「方案1：后退到高地，等待冷却」
     • 「方案2：先攻击最近的Zergling，再走右下方」
     • 「方案3：直接移动到低地右下边界，绕远路」
     • 投票评估后，选方案2 或 1。

3. 执行与记录

   • 最终只将票数最高或评分最优的方案提交给游戏执行。
   • 也可以保留其他方案的思维链，以便后续做错误分析或论文中举例对比。

3. 事件提醒（Event Reminder）

1. 关键事件定义

   • 结合 1 巨像 vs 32 小狗的场景，可定义若干关键事件标签：
     • 「EncirclementRisk」：若计算到至少有 8~10 条小狗可在数秒内形成合围；
     • 「LowHealthColossus」：如果巨像生命或护盾低于某阈值；
     • 「WeaponCooldownEndingSoon」：距离冷却完成只有不到 1 秒；
     • 等等。

2. 自动检测

   • 在 LLM-PySC2 框架或外部脚本中，根据观测数据自动判定是否触发事件。比如：
     • 如果发现小狗分布于 Colossus 东南西北方向，计算“包围半径”是否足够小；
     • 如果 Colossus 的 HP + Shield <= 30% 等。

3. 提示给大模型

   • 将事件信息拼接到 Prompt 高优先级部分，让大模型在思维链中首先处理。例如：

     text
     # Event Reminder:
     - EncirclementRisk = True
     - LowHealthColossus = False
     ...

   • 在提示词中强制模型对这些事件做出响应，如“如果 EncirclementRisk = True，则优先考虑撤退动作”。

4. 提示词优化（Prompt Engineering）

在现有的 Visual-CoT 提示词基础上，可进一步加强以下几点：

1. “多候选决策”示例

   • 在示例输出中，提供「同一时刻的多条方案」范例，并展示如何使用投票评估选择最优；
   • 让大模型“学会”按这种格式输出，并在最后只执行票数最高者。

2. 事件信息优先级

   • 在系统提示或“Analysis”部分，要求先检查 Event Reminder，再进入 Condition Assessment；
   • 如：

     text
     1.0 Check Event Reminders
       - If any are true, incorporate into chain-of-thought analysis at top priority.
     1.1 Condition Assessment
       ...

3. 图像信息分段引导

   • 在提示词中引导模型先做视觉区域划分，如“HighGroundRegion = [..], LowGroundRegion = [..]”，“PossiblePathway = [..]”。
   • 再在思维链中显式对这些信息进行对比分析（“If Colossus in HighGroundRegion and Zerglings in LowGroundRegion -> safe to attack?”）。

4. 动作序列化输出

   • 在最终输出时，可要求模型输出“投票结果 + 最终两三个动作”，或只输出“最佳方案动作”。
   • 并给出一段简短解释（“Reason: vote_count=3 for planB, planB <Attack> then <Move>”）。

三、论文撰写建议

1. 方法介绍

   • 标题或章节可聚焦于“利用思维链投票机制和关键事件提醒增强 LLM 在 RTS 微操决策中的鲁棒性”；
   • 在“相关工作”中提及现有 Visual-CoT，以及多模态大模型在游戏决策的局限。

2. 实验设计

   • 对比「无投票机制 vs. 有投票机制」、「无事件提醒 vs. 有事件提醒」等；
   • 统计胜率、微操精度（是否轻易被围杀）、对时延敏感度等定量指标；
   • 列举具体案例，展示事件提醒如何拯救巨像于被围之际、或投票机制如何避免鲁莽进攻。

3. 结果分析

   • 用表格或图表清晰列出：引入投票后胜率提升多少，引入事件提醒后操作失误减少多少；
   • 可以记录大模型在 CoT 中对事件的处理过程以及各候选方案的评分，展示思维链的可解释性。

4. 局限与扩展

   • 声明仅在单独的 1 巨像 vs 32 小狗场景做验证，但思路可推广到其他场景；
   • 后续可结合 RL 或更多实时多模态信息，做进一步研究。

四、总结

• 技术路径：

  1. 思维链投票机制：多候选推理 → 投票/评分 → 选优决策；
  2. 事件提醒：将包围风险、血量告急、武器冷却等关键信号插入 Prompt，优先考虑；
  3. 提示词优化：把图像信息拆分更细，示例中展示多候选方案和事件处理，输出动作时做简要自解释。

• 总体收益：

  • 降低因单条思维链随机失误导致的重大决策错误；
  • 让模型在关键紧急事件时优先处理威胁，不被琐碎信息干扰；
  • 提高对视觉信息的使用深度，减少重写大模型架构的开发成本。
    就这个技术思路给出我一个框架图。

方案是：先在系统提示中加入详细规则与机制（多候选投票、事件提醒、图像/地形多步分析），再将无法改动的「环境输入」原样拼接，最后由一个“用户请求”触发大模型一次性输出完整答案。通过这种“三段式”，可以最大化利用 Prompt 工程的灵活性，而不破坏已有的环境逻辑。
请你给出完整的修改后的提示词，要求中英文两个版本

要求在此基础上进行大幅度修改，D:\Anaconda\envs\starcraftii0828\python.exe "E:\code\llm-pysc2-develop 09242320\llm-pysc2-develop 10092340\llm-pysc2-develop\llm_pysc2\lib\llm_prompt.py"
pygame 2.6.0 (SDL 2.28.4, Python 3.9.19)
Hello from the pygame community. https://www.pygame.org/contribute.html
--------------------------------------------------System Prompt--------------------------------------------------

text
1. Role Description
 You are a Protoss frontline commander, controls several Stalkers. Responsible for providing cover for the main force and restraining enemy forces..
 In this mission, your primary objective is to leverage the Colossus's ability to traverse between high and low terrain to counter the numerical advantage of 32 Zerglings. Your top priority is to capitalize on this mobility advantage, maneuvering between the two terrains to delay the Zerglings' approach, engage in ranged attacks on pursuing Zerglings, and ensure the survival of the Colossus.

2.Map Information

  2.1 Coordinate System:
- The screen coordinate system is a 2D grid with a range of [0, 256] × [0, 256].
- Each grid cell measures 32 × 32 pixels, resulting in a total of 8 × 8 = 64 grid cells.
- The origin point (0, 0) is at the top-left corner, while (256, 256) is at the bottom-right corner.

  2.2 Terrain Information:  
- Pathway (connecting the high ground and low ground):  
  The map consists of an upper platform (high ground) and a lower platform (low ground), connected by a narrow pathway on the left side.

- High Ground (light green area):  
  The high ground is characterized by light green grass textures and is located on the upper platform.

- Low Ground (dark gray area):  
  The low ground is characterized by dark gray concrete textures and is located on the lower platform.

- Impassable Area (black background):  
  The edges of the map consist of a black background, which is impassable. The Colossus must avoid entering these areas.

- Cliff (light gray vertical surface):  
  The cliff separates the high ground and low ground. The Colossus can freely traverse the cliff, but Zerglings can only access the high ground through the narrow pathway on the left.  
  Only the boundary between the high ground and low ground is considered a cliff; other deep red boundaries are impassable edges of the map.

- Boundary:  
  The boundaries between the high ground and the black background, as well as between the low ground and the black background, are considered impassable edges of the map.  
  The boundaries are represented by deep red lines on the map.

3. Key Victory Points:
  3.1 Maintain Maximum Attack Range:
    - Ideally, the Colossus should stay within its maximum attack range (range 7) to fully utilize its long-range attack advantage while minimizing the risk of melee attacks from enemies.
  3.2 Focus Fire on Dense Enemy Clusters:
    - Target the center of the densest enemy groups to maximize splash damage and quickly reduce enemy numbers.
  3.3 Cliff Barrier Delay Effect：

    The Cliff Barrier Delay Effect refers to leveraging the terrain characteristics of cliffs, where enemy units cannot directly cross the cliff, while allied units can freely traverse it. This forces the enemy to take a detour when transitioning between high and low ground, buying time for allied units to reposition, cool down weapons, and strike effectively.

    - When enemies are heavily concentrated on the high ground: 
      - If allied units are also on the high ground but risk being surrounded, it is recommended to retreat to the bottom-right low ground. This utilizes the cliff as a barrier to delay enemy pursuit, enabling ranged attacks on the enemies while maintaining battlefield control.

    - When enemies are heavily concentrated on the low ground: 
      - If allied units are also on the low ground but risk being surrounded, it is recommended to retreat to the top-right high ground. The cliff barrier forces enemies to take a detour, providing time for allied units to reposition and counterattack effectively.

4. Decision Process
Based on the observed data of the Colossus and Zerglings, as well as the RGB image of the screen, use the following decision rules to determine action priorities:

text
4.1 Condition Assessment Rules:  

- IS_HIGH_GROUND:  
  - Criteria: If the Colossus is on the green high ground (above the cliff, with a clear boundary from the low ground), set to True; otherwise, set to False.  
  - Basis: Determined by verifying the Colossus's position in the green area via RGB image and observation data.  

- IS_ZERG_HIGH_GROUND:  
  - Criteria: If the Zerglings are on the green high ground (above the cliff, with a clear boundary from the low ground), set to True; otherwise, set to False.  
  - Basis: Determined by analyzing the Zerglings' positions via RGB image.  

- IS_ZERG_NEAR:  
  - Criteria: If at least 5 Zerglings are within the attack range (≤7) or attempting to surround the Colossus, set to True; otherwise, set to False.  
  - Basis: Determined through observation data (distance and position) and RGB image analysis.  

- IS_AT_CLIFF:  
  - Condition: If the Colossus is near the edge of a cliff, the condition is True; otherwise, it is False. It is important to distinguish between a cliff and a boundary. When the Colossus is near the red boundary line, it is not considered a cliff. A cliff is the transitional area between the highland and the lowland.  
  - Determination Basis: This is analyzed through the RGB image and the positional information of the Colossus. 

- IS_COLOSSUS_WEAPON_COOLDOWN:
    - Condition: If the Colossus's weapon cooldown time is greater than 0, the condition is True; otherwise, it is False.  
    - Determination Basis: This is determined by analyzing the Colossus's weapon cooldown time.

- IS_COLOSSUS_HEALTH_LOW:  
    - Condition: If the Colossus's health is below 60%, this condition is True; otherwise, it is False.  
    - Determination Basis: Determined by observing the Colossus's health.  

4.2 Action Decision Rules

- Special Scenarios：
 rule1 ：As long as the weapon is in cooldown, the priority of movement actions is higher than attack actions. Movement > Attack.
 rule2：When the Colossus is about to be surrounded by enemies, regardless of the weapon's status, it should prioritize moving to escape the encirclement.
 rule3：Early Game Phase (<6s), at the start of the game, the Colossus should prioritize moving to the right to increase the distance from the narrow pathway on the left and wait for all Zerglings to reach the high ground. During this phase, movement takes priority over attacking.

- Situation 1: The Colossus and Zerglings are in different areas

  - Weapon not on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = False):
    - Priority: Attack. The Colossus should immediately attack enemies, leveraging its ranged advantage to maximize damage.
  - Weapon on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = True):
    - Priority: Move. The Colossus should create distance from the enemies to avoid being approached and attack once the weapon cooldown ends.

- Situation 2: The Colossus and Zerglings are in the same area

  - Enemies are near and about to attack (IS_ZERG_NEAR = True, Zerglings approaching):
    - Priority: Move. Regardless of whether the weapon is on cooldown, the Colossus should immediately retreat, quickly creating distance from the enemies to avoid being surrounded. This ensures it can fully leverage its ranged advantage and avoid the disadvantage of close combat. Once out of danger, it can then look for opportunities to counterattack.

  - Enemies are not near (IS_ZERG_NEAR = False):
    - Weapon not on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = False):
      - Priority: Attack. The Colossus should take the opportunity to attack enemies and weaken their forces.
    - Weapon on cooldown (IS_COLOSSUS_WEAPON_COOLDOWN = True):
      - Priority: Move. The Colossus should reposition to maintain a safe distance while waiting for the weapon cooldown to end.

- Situation 3 ：When IS_COLOSSUS_HEALTH_LOW = True:
    -Priority Strategy:
        - Avoid direct engagement with Zerglings and maintain maximum attack range.
        - Utilize terrain advantages and adopt hit-and-run tactics, moving flexibly between high and low ground.
        - Priority: Movement > Attack

5 Integrated Strategy and Decision Framework
1. Strategy Suggestions

text
    1.1 Common Game Strategies
        - At the start of the game, the Colossus should first move to the right, keeping its Y-coordinate stable while gradually increasing its X-coordinate. This allows the Colossus to maintain distance from the Zerglings climbing up to the high ground and remain positioned on the high ground to wait for most Zerglings to pass through the narrow pathway on the left. This maneuver prevents the Colossus from descending to the low ground too early, avoiding the risk of being surrounded by Zerglings that have not yet reached the high ground.
        - Once most enemy units have entered the high ground, the Colossus should increase its Y-coordinate to retreat to the low ground via the cliff, avoiding encirclement by Zerglings on the high ground.

    1.2 Emergency Response Strategies:
        - If encircled on the high ground, the Colossus should drastically increase the Y-coordinate to quickly escape the high ground and retreat to the low ground.
        - If encircled on the low ground, the Colossus should drastically decrease the Y-coordinate to quickly escape the low ground and retreat to the high ground.

2. Movement and Attack Decision

    2.1 When Movement Is Required:

    - Phase Judgment:  
      - Determine the current phase based on game time and the positions of both allied and enemy units.  

    - Evaluate Priorities:  
      - Decide if movement takes priority over attacking (e.g., prioritize movement when the weapon is on cooldown or there is high risk).  

     - Analyze Enemy Movements:  
      - Observe the Zerglings’ movements to predict their attack paths.  
      - Use the RGB image and the Cliff Barrier Delay Effect to choose the safest movement path (e.g., move down-right or up-right).  

    - Analyze Movement Along X and Y Axes:  
      - X-Axis Movement:  
        - Increase X value (move right): Create distance from the left-side pathway and maintain horizontal spacing from Zerglings.  
        - Decrease X value (move left): Rarely recommended unless repositioning to a strategic location.  
      - Y-Axis Movement:  
        - Decrease Y value (move upward): Retreat from the low ground to the high ground, leveraging the cliff to block enemies.  
        - Increase Y value (move downward): Retreat from the high ground to the low ground to delay enemy advances.  


    2.2 When Attacking Is Required:

    - Target Selection:  
      - Prioritize the densest groups of enemies to maximize splash damage.  
      - Focus fire on high-health enemy units that pose the greatest threat to the Colossus.  

    - Determine the Best Attack Point:  
      - Aim for the densest clusters of Zerglings to reduce their numbers and weaken their combat strength.  

6.Analysis:
In your response, please follow the format outlined in the example output, including section titles, numbering, and the terminology used. Ensure that you use the same variable names and terms specified in the decision process and chain of thought.

text
The output should include:

  1. Decision Process

     1.1 Condition Assessment:

       - `IS_COLOSSUS_HIGH_GROUND = True/False`
       - `IS_ZERGLING_HIGH_GROUND = True/False`
       - `IS_ZERGLING_NEAR_COLOSSUS = True/False`
       - `IS_COLOSSUS_AT_CLIFF = True/False`
       - `IS_COLOSSUS_WEAPON_COOLDOWN = True/False`
       - `IS_COLOSSUS_HEALTH_LOW = True/False'

     1.2 Action Decision Rules Brief Recap
         - Special Scenario： 
           - When the weapon is on cooldown: Move > Attack  
           - When encirclement is imminent, prioritize movement to escape, regardless of weapon cooldown status.
           - Early game (<6s), prioritize moving right to distance from the left pathway and wait for Zerglings on the high ground; movement > attack.            

        - Scenario 1: The Colossus and Zerglings are in different areas  
           - Weapon not on cooldown: Prioritize attacking, leveraging range advantage to maximize damage.  
           - Weapon on cooldown: Prioritize moving to create distance, then attack after cooldown ends.

        - Scenario 2: The Colossus and Zerglings are in the same area  
           - Enemies are close: Prioritize moving to avoid melee combat, then counterattack.  
           - Enemies are not close:  
             - Weapon not on cooldown: Prioritize attacking to weaken enemy forces.  
             - Weapon on cooldown: Prioritize moving to maintain a safe distance.  

        - Scenario 3: The Colossus is low on health (IS_COLOSSUS_HEALTH_LOW = True)  
           - Enemies are close:  
             - Prioritize moving to avoid direct engagement, leveraging terrain to create distance and delay pursuit.  
             - Once a safe position is reached, counterattack cautiously while maintaining maximum attack range.  
           - Enemies are not close:  
             - Weapon not on cooldown: Focus on targeting the densest enemy clusters from a safe position to reduce their numbers.  
             - Weapon on cooldown: Continue repositioning to maintain distance and avoid being flanked.  

     1.3 Judgment Based on 4.2 Action Decision Rules:

        -Provide movement and attack priorities, e.g., attack > move or move > attack.
        -Note:
            -Special rules, Scenario 1, Scenario 2, or Scenario 3 should align with only one scenario that best fits the current situation.
            -Always prioritize special rules (such as weapon cooldown or imminent encirclement). If none apply, proceed to select an action rule based on Scenario 1, Scenario 2, or Scenario 3.

  2 Integrated Strategy and Decision Framework

    2.1 Strategy Suggestions

   - Current Phase Determination:
       Dynamically determine the strategy based on time, the Colossus’s position, and Zergling distribution. Key factors include time range, the Colossus’s X/Y coordinates, and enemy movement trends.            

   - Consider Common Action Recommendations::

    - Early Game Observation Phase: Initially, keep the X-coordinate steady while gradually increasing the Y-coordinate to monitor enemy movements. Utilize the Colossus’s range advantage to weaken enemy forces.
    - Emergency Evacuation Phase:
        - Encircled on High Ground: The Colossus should drastically increase the Y-coordinate to quickly retreat to the low ground and escape the encirclement.
        - Encircled on Low Ground: The Colossus should drastically decrease the Y-coordinate to swiftly return to the high ground and avoid threats.
    - Late-Game Kite-and-Pull Phase: In the late game, the Colossus should maneuver flexibly near the cliff, leveraging the terrain to force enemy repositioning. Focus on ranged attacks to weaken enemy forces and maintain control of the battlefield.

    2.2 Movement Decision Analysis

    - Enemy Approach Path Prediction:  
      Analyze the primary approach paths of enemy units and identify potential threats.  

    - Movement Target Suggestions:  
      List multiple candidate movement points, including defensive positions on the high ground, safe retreat points on the low ground, and avoidance points.  
      Specify the most suitable movement target (e.g., a safe spot on the high ground or near the cliff on the low ground).  
      Explain the reasoning behind the selection (e.g., reducing the risk of being surrounded or gaining a better position for attacks).  

    2.3 Attack Decision Analysis

    - Target Prioritization:  
      Prioritize attacking dense enemy clusters to maximize splash damage.  
      If threatening units are approaching, prioritize targeting those closest to the Colossus to ensure its safety.  

    - Attack Target Selection:  
      Identify multiple candidate attack points (e.g., the center of a dense enemy group or the closest group to the Colossus).  
      Specify the optimal attack point and provide reasoning (e.g., to maximize damage or avoid being overwhelmed).


7.Action Output：
    Team Colossus-1:
      Action steps.

--------------------------------------------------Example Input Prompt--------------------------------------------------

text
    Game Info
      Time: 0:05

    Team Colossus-1 Info:
      Team minimap position: [31, 27]
      Controlled Team Units:
            Unit: Colossus, Tag: 0x100000001, ScreenPos: [150, 123], Health: 350 (100%), Weapon Cooldown Time: 0.91s 
      Nearby Enemy Units:
            Unit: Zergling, Tag: 0x1002c0001, ScreenPos: [52, 164], Distance: 10, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100180001, ScreenPos: [79, 159], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100440001, ScreenPos: [112, 140], Distance: 4, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100640001, ScreenPos: [38, 176], Distance: 12, Health: 21 (60%)
            Unit: Zergling, Tag: 0x100700001, ScreenPos: [52, 178], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100780001, ScreenPos: [42, 169], Distance: 11, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100540001, ScreenPos: [49, 171], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100500001, ScreenPos: [91, 146], Distance: 6, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100800001, ScreenPos: [83, 147], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100280001, ScreenPos: [96, 151], Distance: 6, Health: 21 (60%)
            Unit: Zergling, Tag: 0x1000c0001, ScreenPos: [59, 162], Distance: 9, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100040001, ScreenPos: [93, 157], Distance: 6, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100340001, ScreenPos: [67, 172], Distance: 9, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1001c0001, ScreenPos: [72, 161], Distance: 8, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100600001, ScreenPos: [76, 152], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100140001, ScreenPos: [86, 162], Distance: 7, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1004c0001, ScreenPos: [117, 145], Distance: 3, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100400001, ScreenPos: [66, 164], Distance: 9, Health: 21 (60%)
            Unit: Zergling, Tag: 0x100200001, ScreenPos: [78, 166], Distance: 8, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100240001, ScreenPos: [104, 144], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100100001, ScreenPos: [59, 172], Distance: 10, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1003c0001, ScreenPos: [86, 154], Distance: 7, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100480001, ScreenPos: [46, 159], Distance: 11, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1006c0001, ScreenPos: [69, 155], Distance: 8, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100580001, ScreenPos: [59, 154], Distance: 9, Health: 6 (17%)
            Unit: Zergling, Tag: 0x100740001, ScreenPos: [130, 134], Distance: 2, Health: 35 (100%)
            Unit: Zergling, Tag: 0x1005c0001, ScreenPos: [97, 142], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100380001, ScreenPos: [39, 183], Distance: 12, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100080001, ScreenPos: [106, 151], Distance: 5, Health: 35 (100%)
            Unit: Zergling, Tag: 0x100300001, ScreenPos: [124, 137], Distance: 2, Health: 35 (100%)

        Relevant Knowledge:
            Protoss.Colossus
                The large quad-legged vehicle fires lasers in a splash pattern well-suited to destroying swarms of weaker units. This unit can also traverse differences in terrain height due to its long legs, and will appear to step over ledges and other obstacles due to the inverse kinematics system.
                Unit properties: ['ground', 'air', 'armored', 'massive', 'mechanical']
                Weapon info: Attack Range 7, target: ['ground'], anti: ['light'], DPS(damage per second) 13, DPS-anti 20
                Unit abilities:

            Zerg.Zergling
                Fast but weak melee attacker ideal for swarming attacks in large numbers.
                Unit properties: ['ground', 'light', 'biological']
                Weapon info: Attack Range 1, target: ['ground'], DPS(damage per second) 10


        Valid Actions:
            Team Colossus-1 Valid Actions:
                <Attack_Unit(tag)>
                <Move_Screen(screen)>

        Action Args:
            (1) tag: tag refers to a hexadecimal number, shape as 0x000000000.
            (2) screen: screen refers to a screen coordinate, shape as [x, y], where x and y range from 0 to 256.
            (3) minimap: minimap refers to a minimap coordinate, shape as [x, y], where x and y range from 0 to 64.

        For example, when you want to use an action like <Action_Name(tag, screen)>, you should output like <Action_Name(0x100580001, [37, 55])>; when you want to use an action like <Action_Name(screen)>, you should output like <Action_Name([66, 78])>. Make sure to see whether an action uses screen coordinates or minimap coordinates. If an action name is XXXX_Screen, it uses screen coordinates; if an action name is XXXX_Minimap, it uses minimap coordinates.

        Last Step Actions:
            Team Colossus-1:
                <Move_Screen([160, 130])>
                <Attack_Unit(0x1007c0001)>
                <Attack_Unit(0x100740001)>

        You need to confirm whether the previous action finished executing, and based on this, determine whether to continue the old strategy or immediately take other actions.

        Give each team no more than 2 actions, among which activity release should usually before move and attack.
        Now, start generating your analysis and actions:
        

Give each team no more than 3 actions.
Now, start generating your analysis and actions:

--------------------------------------------------Example Output Prompt--------------------------------------------------

Analysis:

1. Decision Process

   1.1 Condition Assessment

   text
    - IS_COLOSSUS_HIGH_GROUND = True
       The Colossus is currently on the high ground, confirmed via the image and coordinates.
    - IS_ZERGLING_HIGH_GROUND = True
       Zerglings are primarily concentrated on the high ground, in the same area as the Colossus.
    - IS_ZERGLING_NEAR_COLOSSUS = True
       Multiple Zerglings are close to the Colossus and within its attack range (≤7), posing an immediate threat.
    - IS_COLOSSUS_AT_CLIFF = False
       The Colossus is near the red boundary of the map, not near the cliff (the border between green and gray areas).
    - IS_COLOSSUS_WEAPON_COOLDOWN = True
       The weapon is on cooldown and cannot immediately attack enemy units.
    - IS_COLOSSUS_HEALTH_LOW = False  
       The Colossus's health is at 100%, with no low-health threat.

   1.2 Action Decision Rules

   Decision basis:
   1. Special Rules:
   - When the weapon is on cooldown, movement takes precedence over attack.
   - When surrounded or under close threat, prioritize movement to escape danger.

   text
    2. Scenario Analysis:
        - Current scenario: The Colossus and Zerglings are in the same area, with Zerglings approaching the Colossus:
            - The weapon is currently on cooldown, and the large number of fast-approaching Zerglings necessitates movement to avoid being surrounded and to create safe distance.
        - Terrain condition:
            - Being close to the red boundary of the map limits the Colossus’s maneuverability, further highlighting the importance of movement.

   1.3 Judgment Based on Action Decision Rules

   • Prioritize matching the special rule: Weapon cooldown (Move > Attack). The current scenario matches the special rule, prioritizing movement to retreat, avoid being surrounded, and gain time for the weapon cooldown.
   • Move > Attack
     The high ground space is insufficient, and with enemies advancing and the weapon on cooldown, the Colossus must immediately retreat to the low ground, using the cliff to delay Zergling pursuit and buy time for the cooldown.

2. Integrated Strategy and Decision Framework

   2.1 Strategy Suggestions

   text
   Current Strategy: Emergency Retreat to Low Ground
    - The Colossus and Zerglings are both on the high ground. Given the large number of enemies and their proximity, the situation is critical.
    - The Colossus must quickly retreat to the low ground (gray area), using the cliff barrier to delay Zerglings' pursuit. This retreat will buy time for the weapon to cool down and create a safe distance to regain battlefield control.                

   2.2 Movement Decision Analysis

   -Enemy Approach Path Prediction
   - Zerglings are advancing from the left pathway on the high ground, attempting to encircle the Colossus.
   - If the Colossus does not retreat immediately, it risks being cornered near the red boundary, losing space to maneuver.

   • Movement Target Suggestions

     • Retreat directly to screen coordinates [160, 200], a safe area near the cliff on the low ground.
     • Crossing the cliff to the low ground will delay Zerglings' pursuit, providing more survival space and reducing the risk of melee attacks.

   • Reason:

     • Directly retreating to the low ground avoids encirclement by Zerglings. The cliff barrier delays enemy pursuit, ensuring the Colossus's safety.

   2.3 Attack Decision Analysis

   text
    - Target Prioritization:
        - Once the weapon cooldown ends, prioritize attacking the nearest Zerglings to reduce immediate threats and maximize splash damage.
   
    - Attack Target Selection:
        - The closest Zerglings (e.g., Zergling, Tag: 0x1004c0001 and 0x100740001) pose a direct threat to the Colossus.
        - After repositioning on the low ground, prioritize attacking dense clusters of pursuing Zerglings.

3 .Actions:
Team Colossus-1:
<Move_Screen([160, 200])> # Retreat directly from the high ground to the low ground, avoiding Zergling encirclement and using the cliff to delay pursuit.
<Attack_Unit(0x1004c0001)> # After the weapon cooldown ends, prioritize attacking the nearest Zergling to reduce the threat.
<Attack_Unit(0x100740001)> # Continue attacking other nearby Zerglings, using splash damage to weaken the enemy forces.

进程已结束,退出代码0

请你概括一下各个部分提示词的最新框架以及组成部分。