Merge pull request #645 from OceanPresentChao/dualsense

feat: Dualsense device support

Author: kevin-thankyou-lin

docs/algorithms/demonstrations.md

@@ -2,11 +2,11 @@
 
 ## Collecting Human Demonstrations
 
-We provide teleoperation utilities that allow users to control the robots with input devices, such as the keyboard, [SpaceMouse](https://www.3dconnexion.com/spacemouse_compact/en/) and mujoco-gui. Such functionality allows us to collect a dataset of human demonstrations for learning. We provide an example script to illustrate how to collect demonstrations. Our [collect_human_demonstrations](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/scripts/collect_human_demonstrations.py) script takes the following arguments:
+We provide teleoperation utilities that allow users to control the robots with input devices, such as the keyboard, [SpaceMouse](https://www.3dconnexion.com/spacemouse_compact/en/), [DualSense](https://www.playstation.com/en-us/accessories/dualsense-wireless-controller/) and mujoco-gui. Such functionality allows us to collect a dataset of human demonstrations for learning. We provide an example script to illustrate how to collect demonstrations. Our [collect_human_demonstrations](https://github.com/ARISE-Initiative/robosuite/blob/master/robosuite/scripts/collect_human_demonstrations.py) script takes the following arguments:
 
 - `directory:` path to a folder for where to store the pickle file of collected demonstrations
 - `environment:` name of the environment you would like to collect the demonstrations for
-- `device:` either "keyboard" or "spacemouse" or "mjgui"
+- `device:` either "keyboard" or "spacemouse" or "dualsense" or "mjgui"
 
 See the [devices page](https://robosuite.ai/docs/modules/devices.html) for details on how to use the devices.
 

docs/conf.py

@@ -185,12 +185,21 @@
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, "robosuite", "robosuite Documentation", author, "robosuite", "ARISE", "Miscellaneous"),
+    (
+        master_doc,
+        "robosuite",
+        "robosuite Documentation",
+        author,
+        "robosuite",
+        "ARISE",
+        "Miscellaneous",
+    ),
 ]
 
 autodoc_mock_imports = [
     "robosuite.devices.mjgui",
     "robosuite.devices.spacemouse",
+    "robosuite.devices.dualsense",
     "robosuite.devices.keyboard",
     "robosuite.devices.device",
 ]

docs/demos.md

@@ -121,6 +121,12 @@ The `demo_device_control.py` scripts shows how to teleoperate robot with [contro
         This current implementation only supports macOS (Linux support can be added).
         Download and install the [driver](https://www.3dconnexion.com/service/drivers.html) before running the script.
 
+* **DualSense**
+    We use the DualSense joystick from [DualSense](https://www.playstation.com/en-us/accessories/dualsense-wireless-controller/) to control the end-effector of the robot. The joystick provides 6-DoF control commands.
+
+    **Note:**
+        Make sure `hidapi` can detect your DualSense in your computer. In Linux, you may add udev rules in `/etc/udev/rules.d` to get access to the device without root privilege. For the rules content you can refer to [game-device-udev](https://codeberg.org/fabiscafe/game-devices-udev).
+
 * **Mujoco GUI**
         The Mujoco GUI provides a graphical user interface for viewing and interacting with a mujoco simulation. We use the GUI and a mouse to drag and drop mocap bodies, whose
         poses are tracked by a controller. More specifically, once the mujoco GUI is loaded from running `python demo_device_control.py`, you first need to hit the <Tab> key to reach the interactive mujoco viewer state. Then, you should double click on

docs/installation.md

@@ -37,7 +37,7 @@ The base installation requires the MuJoCo physics engine (with [mujoco](https://
    ```
    This will also install our library as an editable package, such that local changes will be reflected elsewhere without having to reinstall the package.
 
-3. (Optional) We also provide add-on functionalities, such as [OpenAI Gym](https://github.com/openai/gym) [interfaces](source/robosuite.wrappers), [inverse kinematics controllers](source/robosuite.controllers) powered by [PyBullet](http://bulletphysics.org), and [teleoperation](source/robosuite.devices) with [SpaceMouse](https://www.3dconnexion.com/products/spacemouse.html) devices. To enable these additional features, please install the extra dependencies by running
+3. (Optional) We also provide add-on functionalities, such as [OpenAI Gym](https://github.com/openai/gym) [interfaces](source/robosuite.wrappers), [inverse kinematics controllers](source/robosuite.controllers) powered by [PyBullet](http://bulletphysics.org), and [teleoperation](source/robosuite.devices) with [SpaceMouse](https://www.3dconnexion.com/products/spacemouse.html) and [DualSense](https://www.playstation.com/en-us/accessories/dualsense-wireless-controller/) devices. To enable these additional features, please install the extra dependencies by running
    ```sh
    $ pip3 install -r requirements-extra.txt
    ```

docs/modules/devices.md

@@ -1,6 +1,6 @@
 # I/O Devices
 
-Devices are used to read user input and teleoperate simulated robots in real-time. This is achieved by either using a keyboard or a [SpaceMouse](https://www.3dconnexion.com/spacemouse_compact/en/), and whose teleoperation capabilities can be demonstrated with the [demo_device_control.py](../demos.html#teleoperation) script. More generally, we support any interface that implements the [Device](../simulation/device) abstract base class. In order to support your own custom device, simply subclass this base class and implement the required methods.
+Devices are used to read user input and teleoperate simulated robots in real-time. This is achieved by either using a keyboard, a [SpaceMouse](https://www.3dconnexion.com/spacemouse_compact/en/) or a [DualSense](https://www.playstation.com/en-us/accessories/dualsense-wireless-controller/) joystick, and whose teleoperation capabilities can be demonstrated with the [demo_device_control.py](../demos.html#teleoperation) script. More generally, we support any interface that implements the [Device](../simulation/device) abstract base class. In order to support your own custom device, simply subclass this base class and implement the required methods.
 
 ## Keyboard
 
@@ -19,10 +19,10 @@ Note that the rendering window must be active for these commands to work.
 |        o-p          |                 rotate (yaw)               |
 |        y-h          |                rotate (pitch)              |
 |        e-r          |                 rotate (roll)              |
-|         b           |     toggle arm/base mode (if appli cable)  |
+|         b           |     toggle arm/base mode (if applicable)  |
 |         s           |  switch active arm (if multi-armed robot)  |
 |         =           | switch active robot (if multi-robot env)   |
-|        ESC          |                    quit                    |
+|       Ctrl+C           |                    quit                    |
 
 
 ## 3Dconnexion SpaceMouse
@@ -38,8 +38,29 @@ We support the use of a [SpaceMouse](https://www.3dconnexion.com/spacemouse_comp
 |   Move mouse laterally    |  move arm horizontally in x-y plane   |
 |   Move mouse vertically   |          move arm vertically          |
 | Twist mouse about an axis | rotate arm about a corresponding axis |
-|      ESC (keyboard)       |                 quit                  |
-
+|           b               |  toggle arm/base mode (if applicable) |
+|           s               |  switch active arm (if multi-armed robot)  |
+|           =               |  switch active robot (if multi-robot environment)   |
+|      Ctrl+C (keyboard)    |                 quit                  |
+
+## Sony DualSense
+
+we support the use of a [Sony DualSense](https://www.playstation.com/en-us/accessories/dualsense-wireless-controller/) as well.
+
+**Sony DualSense controls**
+
+|          Control             |                Command                |
+| :--------------------------- | :------------------------------------ |
+|       Square button          |           reset simulation            |
+|    Circle button (hold)      |             close gripper             |
+|      Move LX/LY Stick        |  move arm horizontally in x-y plane   |
+|   Press L2 Trigger with or without L1 button   |          move arm vertically          |
+|   Move RX/RY Stick           |  rotate arm about x/y axis (roll/pitch)   |
+|   Press R2 Trigger with or without R1 button   |          rotate arm about z axis (yaw)          |
+|      Triangle button         |           toggle arm/base mode (if applicable)             |
+|   Left/Right Direction Pad   |   switch active arm (if multi-armed robot)  |
+|    Up/Down Direction Pad     |    switch active robot (if multi-robot environment)  |
+|      Ctrl+C (keyboard)       |                 quit                  |
 
 ## Mujoco GUI Device
 

docs/overview.md

@@ -18,7 +18,7 @@ This framework was originally developed in late 2017 by researchers in [Stanford
 * **standardized tasks**: a set of standardized manipulation tasks of large diversity and varying complexity and RL benchmarking results for reproducible research;
 * **procedural generation**: modular APIs for programmatically creating new environments and new tasks as combinations of robot models, arenas, and parameterized 3D objects. Check out our repo [robosuite_models](https://github.com/ARISE-Initiative/robosuite_models) for extra robot models tailored to robosuite.
 * **robot controllers**: a selection of controller types to command the robots, such as joint-space velocity control, inverse kinematics control, operational space control, and whole body control;
-* **teleoperation devices**: a selection of teleoperation devices including keyboard, spacemouse and MuJoCo viewer drag-drop;
+* **teleoperation devices**: a selection of teleoperation devices including keyboard, spacemouse, dualsense and MuJoCo viewer drag-drop;
 * **multi-modal sensors**: heterogeneous types of sensory signals, including low-level physical states, RGB cameras, depth maps, and proprioception;
 * **human demonstrations**: utilities for collecting human demonstrations, replaying demonstration datasets, and leveraging demonstration data for learning. Check out our sister project [robomimic](https://arise-initiative.github.io/robomimic-web/);
 * **photorealistic rendering**: integration with advanced graphics tools that provide real-time photorealistic renderings of simulated scenes, including support for NVIDIA Isaac Sim rendering.

docs/simulation/device.rst

@@ -1,7 +1,7 @@
 Device
 ======
 
-Devices allow for direct real-time interfacing with the MuJoCo simulation. The currently supported devices are ``Keyboard``. ``SpaceMouse`` and ``MjGUI``.
+Devices allow for direct real-time interfacing with the MuJoCo simulation. The currently supported devices are ``Keyboard``. ``SpaceMouse``. ``DualSense`` and ``MjGUI``.
 
 Base Device
 -----------
@@ -34,6 +34,18 @@ SpaceMouse Device
   .. automethod:: control_gripper
   .. automethod:: _display_controls
 
+DualSense Device
+-----------------
+
+.. autoclass:: robosuite.devices.dualsense.DualSense
+
+  .. automethod:: get_controller_state
+  .. automethod:: run
+  .. automethod:: control
+  .. automethod:: control_gripper
+  .. automethod:: _display_controls
+  .. automethod:: _check_connection_type
+
 MjGUI Device
 ------------
 .. autoclass:: robosuite.devices.mjgui.MJGUI

docs/source/robosuite.devices.rst

@@ -12,6 +12,14 @@ robosuite.devices.device module
    :undoc-members:
    :show-inheritance:
 
+robosuite.devices.dualsense module
+----------------------------------
+
+.. automodule:: robosuite.devices.dualsense
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 robosuite.devices.keyboard module
 ---------------------------------
 

robosuite/demos/demo_device_control.py

@@ -99,31 +99,68 @@
 from robosuite.wrappers import VisualizationWrapper
 
 if __name__ == "__main__":
-
     parser = argparse.ArgumentParser()
     parser.add_argument("--environment", type=str, default="Lift")
-    parser.add_argument("--robots", nargs="+", type=str, default="Panda", help="Which robot(s) to use in the env")
     parser.add_argument(
-        "--config", type=str, default="default", help="Specified environment configuration if necessary"
+        "--robots",
+        nargs="+",
+        type=str,
+        default="Panda",
+        help="Which robot(s) to use in the env",
+    )
+    parser.add_argument(
+        "--config",
+        type=str,
+        default="default",
+        help="Specified environment configuration if necessary",
+    )
+    parser.add_argument(
+        "--arm",
+        type=str,
+        default="right",
+        help="Which arm to control (eg bimanual) 'right' or 'left'",
+    )
+    parser.add_argument(
+        "--switch-on-grasp",
+        action="store_true",
+        help="Switch gripper control on gripper action",
+    )
+    parser.add_argument(
+        "--toggle-camera-on-grasp",
+        action="store_true",
+        help="Switch camera angle on gripper action",
     )
-    parser.add_argument("--arm", type=str, default="right", help="Which arm to control (eg bimanual) 'right' or 'left'")
-    parser.add_argument("--switch-on-grasp", action="store_true", help="Switch gripper control on gripper action")
-    parser.add_argument("--toggle-camera-on-grasp", action="store_true", help="Switch camera angle on gripper action")
     parser.add_argument(
         "--controller",
         type=str,
         default=None,
         help="Choice of controller. Can be generic (eg. 'BASIC' or 'WHOLE_BODY_MINK_IK') or json file (see robosuite/controllers/config for examples) or None to get the robot's default controller if it exists",
     )
     parser.add_argument("--device", type=str, default="keyboard")
-    parser.add_argument("--pos-sensitivity", type=float, default=1.0, help="How much to scale position user inputs")
-    parser.add_argument("--rot-sensitivity", type=float, default=1.0, help="How much to scale rotation user inputs")
+    parser.add_argument(
+        "--pos-sensitivity",
+        type=float,
+        default=1.0,
+        help="How much to scale position user inputs",
+    )
+    parser.add_argument(
+        "--rot-sensitivity",
+        type=float,
+        default=1.0,
+        help="How much to scale rotation user inputs",
+    )
     parser.add_argument(
         "--max_fr",
         default=20,
         type=int,
         help="Sleep when simluation runs faster than specified frame rate; 20 fps is real time.",
     )
+    parser.add_argument(
+        "--reverse_xy",
+        type=bool,
+        default=False,
+        help="(DualSense Only)Reverse the effect of the x and y axes of the joystick.It is used to handle the case that the left/right and front/back sides of the view are opposite to the LX and LY of the joystick(Push LX up but the robot move left in your view)",
+    )
     args = parser.parse_args()
 
     # Get controller config
@@ -168,18 +205,35 @@
     if args.device == "keyboard":
         from robosuite.devices import Keyboard
 
-        device = Keyboard(env=env, pos_sensitivity=args.pos_sensitivity, rot_sensitivity=args.rot_sensitivity)
+        device = Keyboard(
+            env=env,
+            pos_sensitivity=args.pos_sensitivity,
+            rot_sensitivity=args.rot_sensitivity,
+        )
         env.viewer.add_keypress_callback(device.on_press)
     elif args.device == "spacemouse":
         from robosuite.devices import SpaceMouse
 
-        device = SpaceMouse(env=env, pos_sensitivity=args.pos_sensitivity, rot_sensitivity=args.rot_sensitivity)
+        device = SpaceMouse(
+            env=env,
+            pos_sensitivity=args.pos_sensitivity,
+            rot_sensitivity=args.rot_sensitivity,
+        )
+    elif args.device == "dualsense":
+        from robosuite.devices import DualSense
+
+        device = DualSense(
+            env=env,
+            pos_sensitivity=args.pos_sensitivity,
+            rot_sensitivity=args.rot_sensitivity,
+            reverse_xy=args.reverse_xy,
+        )
     elif args.device == "mjgui":
         from robosuite.devices.mjgui import MJGUI
 
         device = MJGUI(env=env)
     else:
-        raise Exception("Invalid device choice: choose either 'keyboard' or 'spacemouse'.")
+        raise Exception("Invalid device choice: choose either 'keyboard', 'dualsene' or 'spacemouse'.")
 
     while True:
         # Reset the environment