Is mobile_use/agents/agent_qwen.py essentially a screenshot-only model (no A11y grounding) for qwen family?

[Jeffjeno]

  Hi maintainers,

I’m trying to understand the design of mobile_use/agents/agent_qwen.py.

Could you confirm whether UI semantics (A11y / uiautomator dumps) are used elsewhere (e.g., Environment.get_state() or prompts), or is this agent intentionally screenshot-grounded only?

[NingLi670]

Due to the large amount of noise in the accessibility tree and its incompleteness in some cases, our design principle is that the agent uses only screenshots for all operations.

Our agent also supports a11y_tree input (for example,

mobile-use/mobile_use/agents/sub_agent.py

Lines 337 to 341 in 1460899

	if self.include_a11y_tree and current_step.curr_env_state.a11y_tree is not None:
	a11y_tree_prompt = self.prompt.a11y_tree_prompt.format(
	a11y_tree = current_step.curr_env_state.a11y_tree,
	)
	prompt_list.append(a11y_tree_prompt)

), but you need to implement the code to add the a11y_tree content to the environment state. If you use the AndroidWorld environment, you can refer to

mobile-use/benchmark/android_world/run.py

Lines 166 to 176 in 1460899

	if hasattr(agent.config, 'operator') and agent.config.operator and agent.config.operator.include_a11y_tree:
	print("Lode MobileUseEnvironment to get a11y tree.")
	import mobile_use_environment
	new_env = mobile_use_environment.MobileUseEnvironment(
	serial_no=agent.env.serial_no,
	host=agent.env.host,
	port=agent.env.port,
	wait_after_action_seconds=agent.env.wait_after_action_seconds,
	aw_env=env,
	)
	agent.env = new_env

. It's important to note that we are still focusing on purely visual solutions and have not made significant optimizations to the accessibility tree. Our experimental results were obtained without utilizing the accessibility tree.

[Jeffjeno]

@NingLi670
Thank u a lot for the explanation! By the way , I'm curious about the reason why you annotated
# adb_utils.clear_app_data( # chrome_activity, # env.controller, # )
in the android world task_evals/single/browser.py.

:)

[NingLi670]

@Jeffjeno If Chrome is accidentally set as the default application for opening HTML files, clearing Chrome's data can prevent HTML files from opening with a single click, affecting the agent's ability to perform its tasks.