Here, the agent is able to generate a useful API to manage the process of checking out all items in a cart.

async def express_checkout(page):
  """
  Perform an express checkout for the items currently in the cart.

  Args:
      page: The Playwright page object to perform actions on.

  Usage Log:
  - Successfully completed express checkout, resulting in an order confirmation page with order number 000000191.
  - Initial attempts failed due to a timeout error when clicking 'Proceed to Checkout'. Resolved by ensuring items were in the cart.

  Note:
  - Ensure that the cart is pre-filled with the desired items before calling this function.
  - The function assumes that the 'Proceed to Checkout' button is visible and clickable from the cart page.
  - The function navigates through the checkout process by clicking 'Next' on the Shipping page and 'Place Order' on the Review & Payments page.
  - If the 'Place Order' button is not immediately visible, a delay is included to allow dynamic elements to load.
  """
  import asyncio

  await page.goto("/")
  await page.get_by_role("link", name="My Cart").click()
  await asyncio.sleep(5)
  await page.get_by_role("button", name="Proceed to Checkout").click()
  await asyncio.sleep(5)
  await page.get_by_role("button", name="Next").click()
  await asyncio.sleep(5)
  await page.get_by_role("button", name="Place Order").click()
  await asyncio.sleep(5)

Images before (left) and after (right) API execution.

Here, we demonstrate that the agent is capable of fixing DOM resolution errors, by making specific tweaks to problematic sections of code. After the following trajectory, the agent makes a patch to the knowledge base.

Error Detection

Here is the trajectory in which the agent attempted to execute the API, encountered a runtime error, and then fixed the API.

Agent's Proposed Fix: To fix the runtime error in the generated API, the agent correctly identifies the source of the error and then makes the code more robust.

async def delete_merged_branches(page, project_path):
    """
    Automates the deletion of merged branches in a specified GitLab project.
    This function navigates to the branches page of the specified project and
    deletes all merged branches
    by interacting with the confirmation dialog that appears.
    Args:
    page: The Playwright page object.
    project_path: The path of the project in GitLab (e.g., 'byteblaze/
    accessible-html-content-patterns').
    Usage log:
    - Successfully used to delete merged branches in the 'a11y-syntax-
    highlighting' project by navigating to the branches page,
    clicking the 'Delete merged branches' button, and confirming the action
    by typing 'delete'.
    - Ensure the project path is correctly specified to match the URL structure
    in GitLab.
    - The function assumes the user is logged in and has access to the
    specified project.
    - Resolved an issue with strict mode violation by specifying the correct
    textbox within the dialog.
    """
    await page.goto(f"/{project_path}/-/branches")
    await page.get_by_role("button", name="Delete merged branches").click()
--  await page.get_by_role("textbox")
++  await page.get_by_role("dialog", name="Delete all merged branches?").get_by_role(
++    "textbox"
++  ).fill("delete")
    await page.get_by_role("dialog", name="Delete all merged branches?").
    get_by_role(
      "button", name="Delete merged branches"
    ).click()
    await asyncio.sleep(2)
                

Failing to Use APIs

Sometimes, the agent fails to utilize an API that it has generated. For example, on the website cookpad.com, the agent fails to call search_recipes_by_cuisine_type('hamburger') to gain recipe details.

Using APIs Incorrectly

Sometimes, website features (like search bars) have certain mechanics that the agent fails to grasp, but which could potentially be encoded in the documentation of the API. For example, here the agent calls search_recipes_by_ingredients('chocolate chip, -nuts') instead of search_recipes_by_ingredients('chocolate chip without nuts') . This causes search results containing nuts to be displayed, because this website doesn't use the "-" symbol in a search query to negatively filter by a keyword.

While it is very promising to have Python APIs to navigate a website, and especially appealing to have an agent improve its performance simply by gaining more experience, there are some limitations. First of all, our method (like many others) uses the accessibility tree DOM to obtain a compressed or semantically meaningful text representation of the page. However, some websites fail to parse into a natural accessibility tree representation, which may hinder performance.