Agents not consistent with environment Python

[threexc]

While testing #1835 , I saw this error:

(venv) tgamblin@outpost:~/workspace/git/labgrid-power-up$ labgrid-client -c workbench/wb-client.yaml inst identify --name 'tm-inst-sdg1032x'
Selected role main and place workbench from configuration file
Traceback (most recent call last):
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/remote/client.py", line 2335, in main
    args.func(session)
    ~~~~~~~~~^^^^^^^^^
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/remote/client.py", line 999, in instrument
    res = method()
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/binding.py", line 101, in wrapper
    return func(self, *_args, **_kwargs)
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/driver/tminstrument.py", line 57, in identify
    return self.inst.identify()
           ~~~~~~~~~~~~~~~~~~^^
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/binding.py", line 101, in wrapper
    return func(self, *_args, **_kwargs)
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/driver/pyvisadriver.py", line 95, in identify
    return self.query("*IDN?")
           ~~~~~~~~~~^^^^^^^^^
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/binding.py", line 101, in wrapper
    return func(self, *_args, **_kwargs)
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/driver/pyvisadriver.py", line 63, in query
    return self.proxy.query(self.device_identifier, self.pyvisa_resource.backend, cmd, timeout).rstrip()
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/util/agentwrapper.py", line 29, in __call__
    return self.wrapper.call(self.name, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages/labgrid/util/agentwrapper.py", line 107, in call
    raise AgentException(e)
labgrid.util.agentwrapper.AgentException: ModuleNotFoundError('pyvisa module not found, please install it')

But my venv includes pyvisa (and pyvisa-py):

(venv) tgamblin@outpost:~/workspace/git/labgrid-power-up$ pip show pyvisa
Name: PyVISA
Version: 1.16.2
Summary: Python VISA bindings for GPIB, RS232, TCPIP and USB instruments
Home-page: https://github.com/pyvisa/pyvisa
Author: Gregor Thalhammer
Author-email: Torsten Bronger <bronger@physik.rwth-aachen.de>, "Hernan E. Grecco" <hernan.grecco@gmail.com>
License: The MIT License

Copyright (c) 2005-2024 PyVISA Authors and contributors. See AUTHORS

Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Location: /home/tgamblin/workspace/git/labgrid-power-up/venv/lib/python3.13/site-packages
Requires: typing_extensions
Required-by: PyVISA-py

It seems to be because of how a subprocess is spawned here in agentwrapper.py?

    def __init__(self, host=None):
        self.agent = None
        self.loaded = {}
        self.logger = logging.getLogger(f"ResourceExport({host})")

        agent = os.path.join(
            os.path.abspath(os.path.dirname(__file__)),
            'agent.py')
        agent_prefix = os.environ.get("LG_AGENT_PREFIX", "")
        if host:
            # copy agent.py and run via ssh
            with open(agent, 'rb') as agent_fd:
                agent_data = agent_fd.read()
            agent_hash = hashlib.sha256(agent_data).hexdigest()
            agent_remote = os.path.join(agent_prefix, f'.labgrid_agent_{agent_hash}.py')
            connect_timeout = get_ssh_connect_timeout()
            ssh_opts = f'ssh -x -o ConnectTimeout={connect_timeout} -o PasswordAuthentication=no'.split()
            subprocess.check_call(
                ['rsync', '-e', ' '.join(ssh_opts), '-tq', agent,
                 f'{host}:{agent_remote}'],
            )
            self.agent = subprocess.Popen(
                ssh_opts + [host, '--', 'python3', agent_remote],
                stdin=subprocess.PIPE,
                stdout=subprocess.PIPE,
                start_new_session=True,
            )
        else:
            # run locally
            self.agent = subprocess.Popen(
                ['python3', agent],
                stdin=subprocess.PIPE,
                stdout=subprocess.PIPE,
                start_new_session=True,
            )

Is this intentional, or could it be fixed?

[Emantor]

Currently we require the python modules for the agent to be globally available. This also is a problem on NixOS based systems, since by default there may be no python3 in the PATH at all and it definitely won't have external modules unless explicitly created as an environment with the modules. So we will have to look at making the python interpreter that is used on the agent side configurable, this way you could point this to a virtualenv with the module or use a separate python environment in the NixOS case.

[threexc]

Currently we require the python modules for the agent to be globally available. This also is a problem on NixOS based systems, since by default there may be no python3 in the PATH at all and it definitely won't have external modules unless explicitly created as an environment with the modules. So we will have to look at making the python interpreter that is used on the agent side configurable, this way you could point this to a virtualenv with the module or use a separate python environment in the NixOS case.

Let me know if I can help test!

[jluebbe]

So we will have to look at making the python interpreter that is used on the agent side configurable, this way you could point this to a virtualenv with the module or use a separate python environment in the NixOS case.

The interpreter (and possibly venv) used for the exporter should normally work well for the agent. The issue is that we currently only send Resource information to the coordinator, so nothing that's global per exporter. Replicating the interpreter path in each resource seems a bit excessive and inefficient, but doing it differently would mean adding an Exporter object to the APIs exposed by the coordinator.

[Emantor]

#1790 is the related NixOS issue.

The interpreter (and possibly venv) used for the exporter should normally work well for the agent. The issue is that we currently only send Resource information to the coordinator, so nothing that's global per exporter. Replicating the interpreter path in each resource seems a bit excessive and inefficient, but doing it differently would mean adding an Exporter object to the APIs exposed by the coordinator.

From a functional POV all we have to replace is the hardcoded python3 in the AgentWrapper. For NixOS we could hackily replace this with labgrid-python3 which is the entrypoint to a separate python environment with the necessary modules.

But you are absolutely right, we currently have no way to carry exporter global information in the current remote architecture. There is some precedence with the proxy_required extra parameter, which we simply set for all resources if --isolated is supplied on the exporter CLI.

[Emantor]

#1791 is a similar proposal which uses a check for a _labgrid-agent environment.