Update E2E tests (#2302)

* Update E2E tests

* Add to leaderboard only for submissions expected to work

* Update filenames

* Fix typos in filenames

* Add py312 to images to pull

* Relax regex for Failed

* Update documentation

* Update Running-tests.md

Author: Didayolo

.circleci/config.yml

@@ -46,6 +46,7 @@ jobs:
           name: "Docker: Pull required images"
           # not available without "not e2e" tests as they pull ahead of time
           command: |
+            docker pull codalab/codalab-legacy:py312
             docker pull codalab/codalab-legacy:py37
             docker pull codalab/codalab-legacy:py3
             docker pull vergilgxw/autotable:v2

documentation/docs/Developers_and_Administrators/Robot-submissions.md

@@ -127,7 +127,7 @@ On the Leaderboard, you can see the score details of each of your tasks.
 
 * Create a competition with robot submissions enabled
 
-  [Example competition bundle](https://github.com/codalab/competitions-v2/blob/develop/src/tests/functional/test_files/competition.zip)
+  [Example competition bundle](https://github.com/codalab/competitions-v2/blob/develop/src/tests/functional/test_files/competition_v2_wheat_code.zip)
 
 
 ![Edit Competition Page Allow Bots Checkbox](../_attachments/87486437-3037af00-c5f0-11ea-8edf-e758c969ab84_1752851308355029.jpeg)

documentation/docs/Developers_and_Administrators/Running-tests.md

@@ -4,6 +4,7 @@ $ docker compose exec django py.test -m "not e2e"
 
 # Playwright tests (make sure to install uv first: https://docs.astral.sh/uv/getting-started/installation/) 
 uv sync --frozen
+uv add playwright
 uv run playwright install
 docker compose exec -e DJANGO_SUPERUSER_PASSWORD=codabench django python manage.py createsuperuser --username codabench --email codabench@test.mail --no-input
 uv run pytest test_auth.py test_account_creation.py test_competition.py test_submission.py
@@ -21,16 +22,60 @@ docker compose -f docker-compose.yml exec django py.test src/ -m "not e2e"
 
 The repo comes with a couple examples that are used during tests:
 
-### v2 test data
- ```
- src/tests/functional/test_files/submission.zip
- src/tests/functional/test_files/competition.zip
- ```
-### v1.5 legacy test data
- ```
- src/tests/functional/test_files/submission15.zip
- src/tests/functional/test_files/competition15.zip
- ```
+### Codabench (v2) test data
+
+Competition bundles:
+
+```
+codabench/tests/test_files/competitions/competition_v2_miniautoml.zip
+codabench/tests/test_files/competitions/competition_v2_multi_task.zip
+codabench/tests/test_files/competitions/competition_v2_multi_task_fact_sheet.zip
+codabench/tests/test_files/competitions/competition_v2_wheat_code.zip
+codabench/tests/test_files/competitions/competition_v2_wheat_results.zip
+```
+
+And the corresponding submissions:
+
+```
+codabench/tests/test_files/submissions/submission_v2_miniautoml.zip
+codabench/tests/test_files/submissions/submission_v2_wheat_code.zip
+codabench/tests/test_files/submissions/submission_v2_wheat_results.zip
+codabench/tests/test_files/submissions/submission_v2_wheat_results_failure.zip
+```
+
+### CodaLab (v1.8) test data
+
+Competition bundle:
+
+```
+codabench/tests/test_files/competitions/competition_v18_autowsl.zip
+```
+
+And the corresponding submission:
+
+```
+codabench/tests/test_files/submissions/submission_v18_autowsl.zip
+```
+
+### CodaLab (v1.5) test data
+
+Competition bundles:
+
+```
+codabench/tests/test_files/competitions/competition_v15_iris.zip
+codabench/tests/test_files/competitions/competition_v15_sncf.zip
+```
+
+And the corresponding submissions:
+
+```
+codabench/tests/test_files/submissions/submission_v15_iris_code.zip
+codabench/tests/test_files/submissions/submission_v15_iris_results.zip
+codabench/tests/test_files/submissions/submission_v15_sncf.zip
+```
 
-### Other Codalab Competition examples
-[https://github.com/codalab/competition-examples/tree/master/v2/](https://github.com/codalab/competition-examples/tree/master/v2/)
+### Other competition examples
+
+The following repository contains Codabench and CodaLab competition examples:
+
+[https://github.com/codalab/competition-examples/](https://github.com/codalab/competition-examples/)

tests/test_competition.py

@@ -25,7 +25,7 @@ def test_competition_upload(page: Page):
     with page.expect_file_chooser() as fc_info:
         page.get_by_role("button", name="").click()
     file_chooser = fc_info.value
-    file_chooser.set_files("test_files/competitions/competition.zip")
+    file_chooser.set_files("test_files/competitions/competition_v2_wheat_code.zip")
     expect(page.get_by_text("Competition created!")).to_be_visible()
 
 
@@ -75,7 +75,7 @@ def test_manual_competition_creation(page: Page):
     with page.expect_file_chooser() as fc_info:
         page.get_by_role("button", name="").click()
     file_chooser = fc_info.value
-    file_chooser.set_files("test_files/competition/test_logo.png")
+    file_chooser.set_files("test_files/competitions/test_logo.png")
     page.locator(".CodeMirror-scroll").first.click()
     page.get_by_role("application").get_by_role("textbox").fill("Test Description ")
     page.get_by_role("textbox", name="Example: $1000 for the top").click()