internal repo sync

Author: gasse

README.md

@@ -1,42 +1,99 @@
-<img src="./banner.png" />
+# 🤖💻 WorkArena - How Capable are Web Agents at Solving Common Knowledge Work Tasks?
 
-`WorkArena` is a suite of browser-based tasks designed for ServiceNow products, acting as a benchmark for automating commonly conducted activities within the product environment.
+[[Paper]](https://arxiv.org/abs/2403.07718) ♦ [[Benchmark Contents]](#benchmark-contents) ♦ [[Getting Started]](#getting-started) ♦ [[BrowserGym]](https://github.com/ServiceNow/BrowserGym) ♦ [[Citing This Work]](#citing-this-work)
 
-## Setup
+`WorkArena` is a suite of browser-based tasks tailored to gauge web agents' effectiveness in supporting routine tasks for knowledge workers. 
+By harnessing the ubiquitous [ServiceNow](https://www.servicenow.com/what-is-servicenow.html) platform, this benchmark will be instrumental in assessing the widespread state of such automations in modern knowledge work environments.
 
-### ServiceNow Instance
+WorkArena is included in [BrowserGym](https://github.com/ServiceNow/BrowserGym), a conversational gym environment for the evaluation of web agents.
 
-1. Go to https://developer.servicenow.com/ and create an account
-2. Request a Utah developer instance (initializing it might take a while)
-3. Log into your ServiceNow instance via the browser and change the admin password if instructed to do so. If you're already registered in the instance, you can find the instance information (Username, Password, instance URL) in the `My Instances` section of your developer account.
-4. Set the following environment variables:
+
+## Benchmark Contents
+
+At the moment, WorkArena includes `23,150` task instances drawn from `29` tasks that cover the main components of the ServiceNow user interface. The following videos show an agent built on `GPT-4-vision` interacting with every such component. As emphasized by our results, this benchmark is not solved and thus, the performance of the agent is not always on point.
+
+### Knowledge Bases
+
+**Goal:** The agent must search for specific information in the company knowledge base.
+
+_The agent interacts with the user via BrowserGym's conversational interface._
+
+https://github.com/ServiceNow/ui-copilot/assets/2374980/a778fbfd-6f9c-41b2-9c20-1d97cc348866
+
+### Forms
+
+**Goal:** The agent must fill a complex form with specific values for each field.
+
+https://github.com/ServiceNow/ui-copilot/assets/2374980/1f3fa96d-d76e-4f04-a75f-bcf758c5aa42
+
+### Service Catalogs
+
+**Goal:** The agent must order items with specific configurations from the company's service catalog.
+
+https://github.com/ServiceNow/ui-copilot/assets/2374980/8451faa8-3776-4e52-bb90-a560ea23a709
+
+### Lists
+
+**Goal:** The agent must filter a list according to some specifications.
+
+_In this example, the agent struggles to manipulate the UI and fails to create the filter._
+
+https://github.com/ServiceNow/ui-copilot/assets/2374980/042f058b-a966-4f5e-a38f-146464132c49
+
+### Menus
+
+**Goal:** The agent must navigate to a specific application using the main menu.
+
+https://github.com/ServiceNow/ui-copilot/assets/2374980/d5f89fd0-ed72-49b8-81ce-8a493a2c8f5f
+
+
+## Getting Started
+
+To setup WorkArena, you will need to get your own ServiceNow instance, install our Python package, and upload some data to your instance. Follow the steps below to achieve this.
+
+### a) Create a ServiceNow Developer Instance
+
+1. Go to https://developer.servicenow.com/ and create an account.
+2. Click on `Request an instance` and select the `Vancouver` release (initializing the instance will take a few minutes)
+3. Once the instance is ready, you will see a popup showing its URL and credentials. You will also receive a copy by email. Based on this information, set the following environment variables:
     * `SNOW_INSTANCE_URL`: URL of your ServiceNow developer instance
-    * `SNOW_INSTANCE_UNAME`: username for your instance (usually `admin`)
-    * `SNOW_INSTANCE_PWD`: password for your instance (you'll receive this by email and you can get it from your ServiceNow developer account)
+    * `SNOW_INSTANCE_UNAME`: Just use "admin"
+    * `SNOW_INSTANCE_PWD`: The password for your instance. Make sure you place the value in quotes "" since it might contain special characters.
+4. Log into your instance via a browser using the admin credentials. Close any popup that appears on the main screen (e.g., agreeing to analytics).
 
-To set environment variables in Bash, you can use the `export` command. Here's an example:
+**Warning:** Feel free to look around the platform, but please make sure you revert any changes (e.g., changes to list views, pinning some menus, etc.) as these changes will be persistent and affect the benchmarking process.
 
+### b) Install WorkArena and Initialize your Instance
+
+Run the following command to install WorkArena in the [BrowswerGym](https://github.com/servicenow/browsergym) environment:
 ```
-export SNOW_INSTANCE_URL="https://your-instance-url.service-now.com"
-export SNOW_INSTANCE_UNAME="your-username"
-export SNOW_INSTANCE_PWD="your-password"
+pip install browsergym-workarena
 ```
 
-Another option is to add the environment variables to your conda environment. To do this, you can execute the following command :
-
+Then, run this command in a terminal to upload the benchmark data to your ServiceNow instance:
 ```
-conda env config vars set ENV_VAR=VALUE
+workarena-install
 ```
 
-### Install Data
-
-Run the following code to install all the data shipped with the benchmark:
+### c) Validate Your Installation
 
+The are a lot of moving parts (authentication credentials, benchmark data, etc.) so we highly recommend that you sanity-check your installation using our provided unit tests. Do this by running (might take a few minutes):
 ```
-from browsergym.workarena.install import setup
-setup()
+pytest -v .
 ```
 
-### Finally
+Your installation is now complete! 🎉
+
 
-1. Run `pytest -v .` to make sure that your setup works.
+## Citing This Work
+
+```
+@misc{workarena2024,
+      title={WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?}, 
+      author={Alexandre Drouin and Maxime Gasse and Massimo Caccia and Issam H. Laradji and Manuel Del Verme and Tom Marty and Léo Boisvert and Megh Thakkar and Quentin Cappart and David Vazquez and Nicolas Chapados and Alexandre Lacoste},
+      year={2024},
+      eprint={2403.07718},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}
+```

pyproject.toml

@@ -28,6 +28,9 @@ dynamic = ["dependencies", "version"]
 [project.urls]
 homepage = "https://github.com/ServiceNow/WorkArena"
 
+[project.scripts]
+workarena-install = "browsergym.workarena.install:main"
+
 [tool.hatch.version]
 path = "src/browsergym/workarena/__init__.py"
 

src/browsergym/workarena/install.py

@@ -285,3 +285,12 @@ def setup():
     # XXX: Install workflows first because they may automate some downstream installations
     setup_workflows()
     setup_knowledge_base()
+
+
+def main():
+    """
+    Entrypoint for package CLI installation command
+
+    """
+    logging.basicConfig(level=logging.INFO)
+    setup()

src/browsergym/workarena/tasks/form.py

@@ -322,7 +322,7 @@ def _run_init_scripts(self, page: Page) -> None:
 
     def _generate_random_config(self, seed: int, page: Page) -> None:
         """Generate a random configuration for the task."""
-        super().setup(seed, page)
+        self.pre_setup(seed, page)
         self._run_init_scripts(page)
         # Determine task fields
         logging.debug("Determining task fields")

src/browsergym/workarena/tasks/list.py

@@ -222,7 +222,7 @@ def get_goal(self) -> str:
         return self.goal
 
     def _generate_random_config(self, seed: int, page: Page):
-        super().setup(seed, page)
+        self.pre_setup(seed, page)
         self._wait_for_ready(page)
         self.list_info = self._extract_list_info(page)
         # Get available fields
@@ -427,7 +427,7 @@ def setup(self, seed: int, page: Page) -> tuple[str, dict]:
         self.filter_len = len(self.filter_columns)
 
     def _generate_random_config(self, seed: int, page: Page):
-        super().setup(seed, page)
+        self.pre_setup(seed, page)
         self._wait_for_ready(page)
 
         # Extract the list from the page

src/browsergym/workarena/tasks/service_catalog.py

@@ -196,11 +196,23 @@ def _wait_for_ready(self, page: Page, wait_for_form_api=False) -> None:
     def form_js_selector(self):
         return self.js_prefix + "." + self.js_api_forms
 
-    def setup(self, page: playwright.sync_api.Page, seed: int = None) -> tuple[str, dict]:
-        self.pre_setup(seed=seed, page=page)
-
-        self._add_init_scripts_to_context_and_reload(page, ["registerGsftMainLoaded();"])
+    def setup(self, page: playwright.sync_api.Page, seed: int = None) -> None:
+        self.pre_setup(page=page, seed=seed)
+        # the cart is shared for all agents running in parallel. The "Order Now" button
+        # is not affected so we'll make sure the agent can only use that one
+        disable_add_to_cart = """
+        window.addEventListener('DOMContentLoaded', (event) => {
+            const button = document.querySelector('button[aria-label="Add to Cart"]');
+            if (button) {
+                button.disabled = true;
+            }
+        });
+        """
+        self._add_init_scripts_to_context_and_reload(
+            page, ["registerGsftMainLoaded()", disable_add_to_cart]
+        )
         self._wait_for_ready(page)
+        self._remove_top_items_panel(page)
         assert self.all_configs is not None, "No configuration available for the task."
         config = self.fixed_config if self.fixed_config else self.random.choice(self.all_configs)
         # use fixed config if any
@@ -219,6 +231,23 @@ def setup(self, page: playwright.sync_api.Page, seed: int = None) -> tuple[str,
 
         return goal, info
 
+    def _remove_top_items_panel(self, page: Page):
+        """Removes the 'top items' panel that sometimes on the landing page"""
+        frame = page.wait_for_selector("iframe#gsft_main").content_frame()
+
+        # Use evaluate to find and remove divs containing an element with role="heading" and the text "Top Requests"
+        frame.evaluate(
+            """() => {
+                const headings = Array.from(document.querySelectorAll('[role="heading"]'));
+                headings.forEach((heading) => {
+                    if (heading.textContent.includes("Top Requests")) {
+                        let parentDiv = heading.closest('div.drag_section');
+                        if (parentDiv) parentDiv.remove();
+                    }
+                });
+            }"""
+        )
+
     def cheat(self, page: Page, chat_messages: list[str]) -> None:
         super().cheat(page, chat_messages)
         self._wait_for_ready(page=page)
@@ -302,9 +331,20 @@ def cheat(self, page: Page, chat_messages: list[str]) -> None:
             order_now_button.click()
 
     def _generate_random_config(self, seed: int, page: Page):
-        super().setup(seed=seed, page=page)
-
-        self._add_init_scripts_to_context_and_reload(page, ["registerGsftMainLoaded()"])
+        self.pre_setup(page=page, seed=seed)
+        # the cart is shared for all agents running in parallel. The "Order Now" button
+        # is not affected so we'll make sure the agent can only use that one
+        disable_add_to_cart = """
+        window.addEventListener('DOMContentLoaded', (event) => {
+            const button = document.querySelector('button[aria-label="Add to Cart"]');
+            if (button) {
+                button.disabled = true;
+            }
+        });
+        """
+        self._add_init_scripts_to_context_and_reload(
+            page, ["registerGsftMainLoaded()", disable_add_to_cart]
+        )
         self._wait_for_ready(page)
         if self.fixed_request_item:
             self.requested_item = self.fixed_request_item

tests/test_task_setup.py

@@ -0,0 +1,79 @@
+"""
+Tests that are not specific to any particular kind of task.
+
+"""
+
+import pytest
+import json
+import logging
+import random
+
+# bugfix: use same playwright instance in browsergym and pytest
+from utils import setup_playwright
+from playwright.sync_api import Page, TimeoutError
+from tenacity import retry, stop_after_attempt, retry_if_exception_type
+from browsergym.workarena.config import ORDER_APPLE_WATCH_TASK_CONFIG_PATH
+
+from browsergym.workarena.tasks.service_catalog import OrderAppleWatchTask
+
+
+@retry(
+    stop=stop_after_attempt(5),
+    retry=retry_if_exception_type(TimeoutError),
+    reraise=True,
+    before_sleep=lambda _: logging.info("Retrying due to a TimeoutError..."),
+)
+@pytest.mark.slow
+def test_add_to_cart_disabled(page: Page):
+    task_config = json.load(open(ORDER_APPLE_WATCH_TASK_CONFIG_PATH, "r"))[0]
+    task = OrderAppleWatchTask(fixed_config=task_config)
+    # setup the task and try clicking on the "Add to cart button"
+    task.setup(page=page)
+    order_apple_watch_page = (
+        task.instance.snow_url
+        + "/now/nav/ui/classic/params/target/com.glideapp.servicecatalog_cat_item_view.do%3Fv%3D1%26sysparm_id%3D774906834fbb4200086eeed18110c737%26sysparm_link_parent%3Dd258b953c611227a0146101fb1be7c31%26sysparm_catalog%3De0d08b13c3330100c8b837659bba8fb4%26sysparm_catalog_view%3Dcatalog_default%26sysparm_view%3Dcatalog_default"
+    )
+    task.page.goto(order_apple_watch_page)
+    page.wait_for_timeout(1000)
+    iframe_element = task.page.wait_for_selector("#gsft_main")
+    iframe = iframe_element.content_frame()
+
+    # verify that Add to cart is disabled and order now is enabled
+    assert iframe.locator('button[aria-label="Add to Cart"]').is_disabled()
+    assert iframe.locator('button[aria-label="Order Now"]').is_enabled()
+
+
+def test_top_items_panel_removed(page: Page):
+    def check_top_items_panel(page: Page) -> bool:
+        """Checks if the 'top items' panel exists on the landing page"""
+        frame = page.wait_for_selector("iframe#gsft_main").content_frame()
+
+        # Use evaluate to find divs containing an element with role="heading" and the text "Top Requests"
+        panel_exists = frame.evaluate(
+            """() => {
+                const headings = Array.from(document.querySelectorAll('[role="heading"]'));
+                let panelExists = false;
+                headings.forEach((heading) => {
+                    if (heading.textContent.includes("Top Requests")) {
+                        panelExists = true;
+                    }
+                });
+                return panelExists;
+            }"""
+        )
+
+        return panel_exists
+
+    task_config = json.load(open(ORDER_APPLE_WATCH_TASK_CONFIG_PATH, "r"))[0]
+    task = OrderAppleWatchTask(fixed_config=task_config)
+
+    # Setup the task and check if the Top Items panel exists
+    task.setup(page=page)
+    panel_exists = check_top_items_panel(page=page)
+    page.wait_for_timeout(2000)
+    assert panel_exists is False
+    # Reload the page and check if the Top Items panel exists
+    page.goto(task.start_url)
+    page.wait_for_timeout(2000)
+    panel_exists = check_top_items_panel(page=page)
+    assert panel_exists is True