EQA merge & DynaMem CPU by hello-peiqi · Pull Request #254 · hello-robot/stretch_ai

hello-peiqi · 2025-02-04T22:32:53Z

Anyone want to test this system based on this instruction Codes are currently on hello-peiqi/grapheqa_merge branch.

You don't have to worry about discord bot.

I also modified DynaMem and AI Pick up. So please also check:

Some testing steps recommended:

Test installation ./install.sh
Test EQA python -m stretch.app.run_eqa --robot_ip $ROBOT_IP (You will need Gemini token)
Test DynaMem mllm + visual servoing python -m stretch.app.run_dynamem --robot_ip $ROBOT_IP --visual-servo -M (You will need OpenAI token)
Test installation ./install.sh --cpu. Then test DynaMem on CPU python -m stretch.app.run_dynamem --robot_ip $ROBOT_IP --cpu --match-method "class(dynamem)" --vs
(Optional) Obtain discord bot token by this and test EQA with discord bot python -m stretch.app.run_eqa --robot_ip $ROBOT_IP -D

Start with some simple questions such as "is there a monitor on the table?" "what is the color of the chair"

…/stretch_ai into hello-peiqi/grapheqa_merge

hello-cpaxton · 2025-02-06T04:41:22Z

+
+
+class SparseVoxelMap(SparseVoxelMapBase):
+    def __init__(


It would be good to add a docstring here, this is a lot of params (mostly borrowed - can use @OVERRIDES)

hello-cpaxton · 2025-02-06T04:41:30Z

+                DYNAMEM_VISUAL_GROUNDING_PROMPT, model="gpt-4o-2024-05-13"
+            )
+
+    def find_alignment_over_model(self, queries: str):


hello-cpaxton · 2025-02-06T04:42:07Z

+            print("Points close the the point are not similar to the text!")
+        return torch.max(alignments[distances < distance_threshold]) >= similarity_threshold
+
+    def get_2d_map(


Is this different from the base one? If not - just override

if so maybe we could do some refactoring for less duplication?

hello-cpaxton · 2025-02-06T04:42:17Z

+        if debug:
+            import matplotlib.pyplot as plt
+
+            plt.subplot(2, 2, 1)


you can probably delete this debug code

hello-cpaxton · 2025-02-06T04:42:44Z

+            print(f"Error: {e}")
+            return None
+
+    def localize_with_mllm(self, text: str, debug=True, return_debug=False):


this new function could probably use some documentation

hello-cpaxton · 2025-02-06T04:44:14Z

        rr.init("Stretch_robot", spawn=spawn_gui)

+        if output_path is not None:
+            rr.save(output_path / "rerun_log.rrd")


do you find these rerun logs useful?

Not yet, but I guess it is for automatically saving rerun res; since grapheqa does that, I think it is useful especially for some researchers want to collect demo for their websites.

hello-cpaxton · 2025-02-06T04:44:36Z

 # hellorobotinc/stretch-ai_cuda-11.8:$VERSION: the docker image to run
 run_docker_command run \
    -it \
-    --runtime nvidia \


why do you get rid of this? is it no longer necessary?

It gives me bugs

Can you still run with access to the NVIDIA hardware?

hello-cpaxton · 2025-02-06T04:44:57Z

+        self._manipulation_radius = parameters["motion_planner"]["goals"]["manipulation_radius"]
+        self._voxel_size = parameters["voxel_size"]
+
+        # self.image_processor = VoxelMapImageProcessor(


hello-cpaxton · 2025-02-06T04:45:21Z

+
+    def execute_action(
+        self,
+        text: str,


documentation - say what actions can be?

* update? * clean up codes * update docs * update * update * cpu * update * update * change models * fix bug * installation * debug * tune parameters * restore gui * fix bug * update docs and args * fix bug

* update a bit * small update * revert some changes * revert some changes * revert some changes * automatically decide which model to use --------- Co-authored-by: GRAIL <grail@demobox.hydra-bee.ts.net>

* remove moonlight * remove dinobot * remove yolo * remove detic * update installation * remove the remains of Detic * remove paligemma * remove molmo * Hello peiqi/remove online audio (#278) * remove audio * tune audio speed * add piper * add new engine * modify piper * modify piper * fix bugs * update test * update test * update tets * rearrange files * small update * small update * remove cli * remove more files * remove voice chat app * remove useless files * remove useless folder * add files * new gitignore * fix bug * update gemma and docker * add terminate on failure * Hello peiqi/clean app (#279) * remove files * remove emote app * update keyboard teleop * update print states * merge check status and print states * merge check status and print states * remove start rerun * remove more perception modules * remove vlm planning * replace git with blip * remove dinov2siglip * restore scripts * remove mobile sam * deal with sam * update docs * update install.sh * remove map loader app * remove more apps * remove load map util * remove demo * remove ENV and Agent * clean core * clean drivers * update detic * unify agent * clean robot agent * remove get detections * remove unused attr * fix bug * fix ai pick up bug * Hello peiqi/fix simulation (#283) * fix * fix intrisincs * fix not robocasa parts * update * tune readme * Hello peiqi/slam checking (#281) * remove readme * remove unnecessary core * remove files * restore * remove navigation utils * remove navigation app * remove drivers * fix orbslam3 bugs * fix orbslam3 bugs * remove log * remove unnecessary ros2 files * further clean * update slam * checkpoint * remove orbslam docs * update detic * clean more files * clean slam * update * fix bug * fix typo * fix match method * fix bug * installation * debug grasp * slight tunning

hello-robot-shehab

I was able to run the apps in the PR on my Stretch 3.

blakerbuchanan and others added 16 commits November 5, 2024 16:45

Add semnav-specific / hydra-related changes to stretch_ai

63041b0

commit changes used for in-home hardware experiments

3706b02

installation update

a105448

grapheqa

daa6880

grapheqa fix

0e2e7ae

update

665ac75

restore rerun

6d12fbf

restore branch

c9d93f6

dynamem fix

4e31af9

revert unexpected bugs

f30ddc9

revert unexpected bugs

c9eda02

Merge branch 'main' into hello-peiqi/grapheqa

b7218f0

grapheqa push

216ddbb

Merge branch 'main' into hello-peiqi/grapheqa

7b37896

Merge branch 'hello-peiqi/grapheqa' of https://github.com/hello-robot…

c9feeba

…/stretch_ai into hello-peiqi/grapheqa_merge

debug to make scene grpah building work

b58cd69

Base automatically changed from hello-peiqi/grapheqa to main February 5, 2025 15:18

update

4a7ea35

hello-cpaxton reviewed Feb 6, 2025

View reviewed changes

hello-peiqi added 11 commits February 6, 2025 22:03

update

12dbed8

update

15c8cef

checkpoint

8032c56

update

a384867

update save image

a17cc36

update save best image

232da6e

add planner

b55a649

debug

a18fe80

debug planner

051c4b8

write custom question

7511362

add discord support

1850efe

hello-peiqi added 7 commits May 17, 2025 09:53

update

350f43c

update

cb7b38e

update

533c2f9

update

75fd6ed

update

13b9583

udpate

47bc45e

Hello peiqi/dynamem cpu (#270)

ec0dace

* update? * clean up codes * update docs * update * update * cpu * update * update * change models * fix bug * installation * debug * tune parameters * restore gui * fix bug * update docs and args * fix bug

hello-peiqi changed the title ~~EQA merge~~ EQA merge & DynaMem CPU Jun 6, 2025

hello-peiqi and others added 17 commits June 8, 2025 14:33

fix bug

1adc58d

update

1ff78d1

update hyperparameter

542c895

update hyperparameter

87dfda3

update hyperparameter

54ddaf7

update

97db994

update

ee102a0

update gemini

92857a2

update

7023980

Hello peiqi/cvpr update (#274)

a330ded

* update a bit * small update * revert some changes * revert some changes * revert some changes * automatically decide which model to use --------- Co-authored-by: GRAIL <grail@demobox.hydra-bee.ts.net>

add SAMv2 installation

82a59da

comment small update

9fd1972

fix bug

fd9cf9e

fix installation

4ed7def

resolve conflict

b2eda9a

update docs

18512e9

hello-peiqi enabled auto-merge (squash) August 11, 2025 17:57

hello-peiqi disabled auto-merge August 11, 2025 19:20

hello-robot-shehab approved these changes Aug 11, 2025

View reviewed changes

hello-binit merged commit f9f814c into main Sep 30, 2025
1 of 3 checks passed

hello-binit deleted the hello-peiqi/grapheqa_merge branch September 30, 2025 03:24

Conversation

hello-peiqi commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hello-peiqi Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hello-robot-shehab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

hello-peiqi commented Feb 4, 2025 •

edited

Loading

hello-peiqi Feb 6, 2025 •

edited

Loading