Conversation
f771198 to
76622d2
Compare
Trezor plugin is updated for the upcoming `trezorlib` 0.20 release. Tested the following scenarios with Trezor 1 (with FW 1.14.0), T, Safe 3 and Safe 5 (with FW 2.10.0): - create a new wallet: - generate new seed - recover from backup - verify an address - send & RBF flows - set a PIN & a passphrase - open an existing wallet file - locking the device on exit - flow cancellation Safe 7 support will be added in a subsequent PR.
76622d2 to
c4fca7a
Compare
|
Thanks a lot for this! I see you are bumping the minimum supported trezorlib version here and the new minimum is not tagged/released yet. So I guess this should not be merged as-is? Alternatively, should we perhaps have some compatibility code to support both older and newer trezorlib? |
We are planning to release |
|
|
For example, can My notes follow: Added 8 new runtime dependencies, 2 runtime dependencies removed. As for build-time, because of needing hatchling, 5 new build-time dependencies are pulled in, and hatchling itself we cannot really build from source. Runtime: removed: ecdsa (and six) Runtime old depsRuntime new depsbuild-time: How to install old trezorlibHow to install new trezorlibEDIT: it's even worse I made too many shortcuts for my build-time checks... e.g. |
|
|
it's stupid that the build engines are pulled in. neither trezorlib nor construct_classes actually care about what builds them, there is no "build process" beyond the old thing that distutils could do fine. this is all a consequence of locally using uv for managing dependencies, and that requires us to specify a build engine. off the top of your head, do you know of a build engine that is compatible with pyproject.toml and doesn't impose burdens on you?
going forward i think we'll have to split off a |
Right. I don't like it either. Now I realise source-only wheels are somewhat of a weird edge-case. I guess we could theoretically also accept those in addition to sdists, however it does not seem easily possible to instruct pip and the related tools to accept either sdists or source-only wheels. Also, I am not sure if it's possible to reliably tell if a wheel is actually source-only but I guess this is a conceptually separate issue - sdists too could incorrectly include compiled code inside. Still, only allowing sdists allows us to only review just a single distributable (the
That's the core of the issue. If we want to build from sdists in our build scripts, what PEP-517 build backend our dependencies choose we need to pull in. I understand that some of these build backends are very convenient for the developers of a given library (python package) during their normal workflow, but they actually put a burden on downstream projects who don't just want to blindly download binaries from PyPI. Perhaps the problem is our approach, perhaps we are doing it wrong, but to me it just seems like the whole python ecosystem just does not seem to care about reproducing and bootstrapping builds. People seem okay with downloading random binaries from PyPI and not to care being able to build them themselves, much less cross-compile them. Have you ever tried building a hidapi or a cryptography wheel on Linux for a Windows target? It's madness. We gave up on that (for now). see here to get a grasp of how e.g. our AppImage build works: electrum/contrib/build-linux/appimage/make_appimage.sh Lines 125 to 158 in 4f7b6e8 see here for requirements-build-base.txt and its generated counterpart
(Note how some packages such as PyQt6 and cryptography we begrudgingly give a lot more trust and just take their binary wheels. We have a few similar whitelists across the different builds that would be great to make shorter over time.)
setuptools, flit-core, and poetry-core we already added to requirements-build-base, so those are already pulled in and trusted. flit-core and poetry-core are both ~minimal and don't have any transitive dependencies of their own. setuptools (and pip and wheel) is unavoidable and we consider it ultimately trusted due to historical reasons in any case.
Ok, right, that sounds good in theory. :) I will have a look at how to do the blacklisting in practice.
Yes, please. Something similar to the trezor-parts vendored inside HWI, a fully stripped-down bare minimum library to communicate with the hardware but that you guys maintain would be extremely desirable. For example, I would rather not pull in Click either :D Alternatively, perhaps long-term we should run the hardware-device comms stuff in a separate process and just do IPC with the main python electrum process. That way we would be able to afford to care less about what dependencies hardware wallet support pulls in. Currently if a user even tries to scan for connected hardware signers, all the plugins get loaded and get to execute code, and then if the user later opens a software-seeded wallet without restarting the python process, their seed in memory is fair game... |
|
btw I think you can pin construct_classes to 1.x to get rid of uv_build? |
Yes, that looks correct. For now we can pin "construct_classes<0.2". |
|
Besides the new dependencies, this pull request looks simple and could go in. But I don't want to pull in new dependencies. In general I would like to ask you to be very conscious about the dependencies you pull in and avoid doing so whenever possible. Dependencies that are maintained by the same people are a special case, so e.g. construct-classes does not even really count (but uv-build would).
What about hatchling in the build system of trezorlib? Could you use setuptools/wheel/poetry-core/flit-core instead?
This is in my backlog but haven't had time to look into it yet and might not for a while... |
The method works as follows: 1. for every blacklisted dependency, as listed in ghost.txt, create and install an empty ghost package which will satisfy the dependency resolver 2. before hash resolution step, remove those ghosts to make hashin happy This required converting find_restricted_dependencies to use locally installed package metadata instead of looking it up online on pypi. But that seems to be a good idea anyway.
switching to flit in trezor/trezor-firmware#6564
have a look at 797f5c8 (i'm sure @romanz can cherry-pick it if you like it) this is certainly a faster option than waiting for our side to split up trezorlib into multiple packages |
|
general note: my first instinct (as a former distro package maintainer) would be to fetch the sdist and patch the pyproject.toml inside. that's what i wanted to do originally, because i thought your source-only build process is better than that way you could get rid of uv_build wherever you see it :) it's still an option, though probably somewhat more involved because of how you use |
Nice, thank you.
Great, that looks promising! Please cherry-pick it to the branch. I tried to build from that commit - but ofc as it relies on trezor/trezor-firmware#6564, I had to hack some more stuff on top.
Right, it sounds feasible but not sure how much work it would be to maintain the patches. |
| @@ -153,8 +155,8 @@ class TrezorPlugin(HW_PluginBase): | |||
| libraries_URL = 'https://pypi.org/project/trezor/' | |||
| minimum_firmware = (1, 5, 2) | |||
There was a problem hiding this comment.
I have tested this on a T1 with fw 1.12.1 and with a model T with fw 2.6.4. Worked ok.
I expect you tested cutting edge fw, so no point for me to repeat that.
I also have another T1 with fw 1.8.3. That device works with electrum master and trezorlib 0.13.x, but not with this branch and new trezorlib.
This is what I get in the Electrum Wizard when trying to create a wallet with that trezor model one:
10.69 | D | gui.qt.wizard.wallet | resolve_next view is trezor_xpub
10.69 | D | gui.qt.wizard.wallet | wizard stack:
0: 0x7f3e8cd72d00 - {}
1: 0x7f3e8cd88200 - {'wallet_name': 'wallet_88', 'wallet_exists': False, 'wallet_is_open': False, 'password': '<redacted>', 'wallet_needs_hw_unlock': False}
2: 0x7f3e8cd81d00 - {'wallet_name': 'wallet_88', 'wallet_exists': False, 'wallet_is_open': False, 'password': '<redacted>', 'wallet_needs_hw_unlock': False, 'wallet_type': 'standard'}
3: 0x7f3e771a9880 - {'wallet_name': 'wallet_88', 'wallet_exists': False, 'wallet_is_open': False, 'password': '<redacted>', 'wallet_needs_hw_unlock': False, 'wallet_type': 'standard', 'keystore_type': 'hardware'}
4: 0x7f3e77360a00 - {'wallet_name': 'wallet_88', 'wallet_exists': False, 'wallet_is_open': False, 'password': '<redacted>', 'wallet_needs_hw_unlock': False, 'wallet_type': 'standard', 'keystore_type': 'hardware', 'hardware_device': ('trezor', DeviceInfo(device=Device(path='webusb:001:4', interface_number=-1, id_='webusb:001:4', product_key='Trezor', usage_page=0, transport_ui_string='webusb:001:4'), label='white', initialized=True, exception=None, plugin_name='trezor', soft_device_id='466354651092540CD6071750', model_name='Trezor One'))}
c: 0x7f3e77360f80 - {'wallet_name': 'wallet_88', 'wallet_exists': False, 'wallet_is_open': False, 'password': '<redacted>', 'wallet_needs_hw_unlock': False, 'wallet_type': 'standard', 'keystore_type': 'hardware', 'hardware_device': ('trezor', DeviceInfo(device=Device(path='webusb:001:4', interface_number=-1, id_='webusb:001:4', product_key='Trezor', usage_page=0, transport_ui_string='webusb:001:4'), label='white', initialized=True, exception=None, plugin_name='trezor', soft_device_id='466354651092540CD6071750', model_name='Trezor One')), 'script_type': 'p2wpkh', 'derivation_path': 'm/84h/1h/0h'}
10.70 | D | gui.qt.wizard.wallet | load_next_component: <class 'electrum.plugins.trezor.qt.WCTrezorXPub'>
10.70 | D | gui.qt.wizard.wallet | view "trezor_xpub" last: False
10.74 | E | trezorlib.protocol_v1 | Trezor did not return a session ID. Session management is now broken.
/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/trezorlib/protocol_v1.py:210: UserWarning: Your Trezor firmware does not support root fingerprint.
warnings.warn("Your Trezor firmware does not support root fingerprint.")
11.25 | E | plugins.trezor.qt.WCTrezorXPub | AssertionError()
Traceback (most recent call last):
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/electrum/gui/qt/wizard/wallet.py", line 1438, in get_xpub_task
self.xpub = self.get_xpub_from_client(_client, _derivation, _xtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/electrum/plugins/trezor/qt.py", line 815, in get_xpub_from_client
return client.get_xpub(derivation, xtype, True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/electrum/plugin.py", line 993, in wrapper
return run_in_hwd_thread(partial(func, *args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/electrum/plugin.py", line 986, in run_in_hwd_thread
return fut.result()
^^^^^^^^^^^^
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/electrum/plugins/trezor/clientbase.py", line 183, in get_xpub
node = trezorlib.btc.get_public_node(self.session, address_n).node
^^^^^^^^^^^^
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/electrum/plugins/trezor/clientbase.py", line 82, in session
session.ensure_unlocked()
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/trezorlib/tools.py", line 336, in wrapper
return context_func(context, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrdPnlnC/usr/lib/python3.12/site-packages/trezorlib/client.py", line 184, in ensure_unlocked
assert resp.root_fingerprint is not None
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
I did not investigate the root cause. I guess if you think that's the best way to handle, we could just bump minimum_firmware. but you need to investigate at least to the degree that you know which version to bump to. :)
OTOH note that some people with geo-distributed cold multisigs likely won't be happy if we bump the minimum. Up to you.
There was a problem hiding this comment.
Thanks! Will take a look tomorrow.
There was a problem hiding this comment.
trezor/trezor-firmware#6580 should support old FWs.
| keystore_class = TrezorKeyStore | ||
| minimum_library = (0, 13, 0) | ||
| maximum_library = (0, 14) | ||
| minimum_library = (0, 20, 0) |
There was a problem hiding this comment.
please see SomberNight@f8226f9 and cherry-pick that or add something similar manually
I built an appimage and found that the plugin is broken at runtime due to the change to trezorlib.__version__.
ref https://github.com/trezor/trezor-firmware/blame/e754ba60a15c770de4667f072ae4e1f09c58e731/python/src/trezorlib/__init__.py#L21-L28
(btw note how the code says removal in 0.15 -- and you jumped from 0.14 to 0.20, guess that needs updating)
8.16 | W | gui.qt.wizard.wallet.WCChooseHWDevice | error getting device infos for trezor: Library version for 'trezor' is incompatible. // Installed: unknown, Needed: 0.20.0 <= x < 0.21 // Make sure you install it with python3
>>> trezorlib.__version__
Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'trezorlib' is not defined
>>> import trezorlib
>>> trezorlib.__version__
Traceback (most recent call last):
File "/tmp/.mount_electrcmMljC/usr/lib/python3.12/importlib/metadata/__init__.py", line 397, in from_name
return next(cls.discover(name=name))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/.mount_electrcmMljC/usr/lib/python3.12/site-packages/trezorlib/__init__.py", line 27, in __getattr__
return importlib.metadata.version("trezor")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrcmMljC/usr/lib/python3.12/importlib/metadata/__init__.py", line 889, in version
return distribution(distribution_name).version
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrcmMljC/usr/lib/python3.12/importlib/metadata/__init__.py", line 862, in distribution
return Distribution.from_name(distribution_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/.mount_electrcmMljC/usr/lib/python3.12/importlib/metadata/__init__.py", line 399, in from_name
raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for trezor
We atm have a whitelist for which packages get .dist-info/ included and trezor needs to be put on that list now.
That's fine, it just needs to be added.
The issue in general with the .dist-info metadata stuff is that the RECORD file is often not reproducible, making the whole build not deterministic (see c52a29f). Usually it is due to packages building .so files with debug symbols, pip-random-generated-filesystem-paths, etc included. All that non-determinism in actual .so files we strip after-the-fact, see
electrum/contrib/build-linux/appimage/make_appimage.sh
Lines 196 to 207 in 3695e00
but I just built two appimages and trezorlib's RECORD file is reproducible, so it can get whitelisted.
Cherry-picked as 797f5c8. |
This PR updates Trezor plugin to use
trezorlib0.20 (trezor/trezor-firmware@45ff241).Tested the following scenarios with Trezor 1 (with FW 1.14.0), T, Safe 3 and Safe 5 (with FW 2.10.0):
Safe 7 support will be added in a subsequent PR.