1
0
Commit Graph

74 Commits

Author SHA1 Message Date
f321x
e80551192b plugins: structure plugin storage in wallet
store all plugin data by plugin name in a root dictionary `plugin_data`
inside the wallet db so that plugin data can get deleted again.
Prunes the data of plugins from the wallet db on wallet stop if the
plugin is not installed anymore.
2025-05-06 13:16:49 +02:00
ThomasV
ef08fb37a6 json_db: add StoredList.clear() method 2025-01-12 10:32:00 +01:00
SomberNight
b3a908f647 WalletDB: (trivial) add type hint 2024-02-12 18:26:08 +00:00
ThomasV
061c821128 Merge pull request #8843 from SomberNight/202401_jsondb_set
walletdb: change structure of prevouts_by_scripthash, and db upgrade
2024-01-31 12:56:11 +01:00
SomberNight
cbece71894 walletdb: rm some dead code 2024-01-23 02:20:01 +00:00
SomberNight
1734042293 jsondb: raise when trying to store sets in the db
Some generic protection to prevent bugs similar to what was just fixed in prev commits.
related https://github.com/spesmilo/electrum/issues/8842
2024-01-23 01:53:10 +00:00
SomberNight
6467db0b7d json_db: rename "_write" to sth more descriptive 2023-11-27 15:30:26 +00:00
ThomasV
98cecb305e jsondb: raise an exception if a daemon thread attempts to write 2023-09-28 18:33:56 +02:00
ThomasV
c495445f51 storage: remember file length when writing to file
- make append fail if the actual file length differs
 - consolidate when file length > 2 * last consolidated length
2023-09-24 12:24:21 +02:00
ThomasV
64cd816a82 json_db: recover corrupt db from incomplete file appends 2023-09-24 12:24:21 +02:00
ThomasV
7ca89f56ee partial-writes using jsonpatch
- partial writes are append only.

 - StoredDict objects will append partial writes to the wallet
   file when items are added, replaced, removed.

 - Lists in the wallet file that have not been registered
   as StoredObject are converted to StoredList, which
   overloads append() and remove(). Those methods too will
   append partial writes to the wallet file.

 - Unlike the old jsonpatch branch, this branch does not support
   file encryption. Encrypted files always fully rewritten, even
   if the change before encryption is a partial write.
2023-09-24 12:24:09 +02:00
ThomasV
019be008d6 json_db: do not overload load_data.
instead, pass an upgrader function to the constructor.
2023-09-24 12:10:35 +02:00
ThomasV
30038f4ace follow-up previous commit: write storage only if there was an upgrade 2023-09-23 15:15:17 +02:00
ThomasV
517cef8248 jsondb: write storage in __init__, in case there was a db upgrade 2023-09-23 14:05:18 +02:00
ThomasV
b5bc5ff9ed Separate WalletDB from storage upgrades.
Make sure that WalletDB.data is always a StoredDict.
Perform db upgrades in a separate class, since they
operate on a dict object.
2023-09-22 15:02:06 +02:00
ThomasV
b96cc82333 Make storage a field of db
This comes from the jsonpatch_new branch.
I rather have in master now, because it touches a lot of filese.
2023-08-18 08:08:31 +02:00
SomberNight
7012d998f1 json_db: add back profiler to _write
follow-up 411098f293
2023-08-15 17:04:49 +00:00
ThomasV
97768a13b9 follow-up previous commit 2023-06-24 13:00:15 +02:00
ThomasV
b8b36c7c30 follow-up prev: fix flake8 test 2023-06-24 12:56:55 +02:00
ThomasV
411098f293 move methods from wallet_db to json_db
the goal of this commit is to call JsonDB.__init__ with data,
not an empty dict
2023-06-24 12:45:07 +02:00
ThomasV
759eaf1cf5 json_db: register extra types outside of constructor 2023-06-23 12:16:14 +02:00
ThomasV
606c51bc4e follow-up previous commit 2023-06-18 13:49:56 +02:00
ThomasV
295734fc53 storage: encapsulate type conversions of stored objects using
decorators (instead of overloading JsonDB._convert_dict and
 _convert_value)
 - stored_in for elements of a StoreDict
 - stored_as for singletons
 - extra register methods are defined for key conversions

This commit was adapted from the jsonpatch branch
2023-06-18 13:08:57 +02:00
SomberNight
312f2641e7 don't use bare except
use "except Exception", or if really needed explicitly "except BaseException"
2023-04-24 12:58:01 +00:00
ThomasV
015648c4e5 Move get_dict method from wallet_db to json_db.
Define overloaded methods: _convert_dict _convert_values
2022-11-03 12:41:49 +01:00
bitromortac
6915e3cb10 lnpeer+wallet: use channel type for channel open
* channel_type is put into storage, serialized as int and
  deserialized as ChannelType
* check for static_remotekey is done via channel type
2022-01-20 16:47:48 +01:00
ThomasV
eadd1bebb2 Do not convert StoredDict keys to str.
Instead, convert json keys to int when storage is read.
2021-09-20 14:49:59 +02:00
Malcolm Smith
67ae678137 storage/db: use faster JSON encoder settings when wallet is encrypted
The standard json module has an optimized C encoder, but that doesn't
currently support indentation. So if you request indentation, it falls
back on the slower Python encoder.

Readability doesn't matter for encrypted wallets, so this disables
indentation when the wallet is encrypted.

-----

based on b2399b6a3e

For a large encrypted wallet, compare:
before change:
JsonDB.dump 1.3153 sec
zlib.compress 1.281 sec
ECPubkey.encrypt_message 0.1744 sec

after change:
JsonDB.dump 0.5059 sec
zlib.compress 1.3120 sec
ECPubkey.encrypt_message 0.1630 sec

Co-authored-by: SomberNight <somber.night@protonmail.com>
2021-01-06 21:14:56 +01:00
SomberNight
a7199696d3 json_db: exempt keystore from StoredDict conversion
The keystore logic would need to be significantly changed to nicely
interoperate with StoredDict/json_db logic.
(just see KeyStore.__init__() and KeyStore.dump())
For now we exempt the keystore from the recursive StoredDict conversion, as
it is a smaller change that is also easier to review for correctness.

fixes #6066
fixes #6401

also reverts 2d3c2eeea9 (which was an even hackier workaround for #6066)
2020-09-04 16:11:01 +02:00
SomberNight
d5f368c584 LN invoices: support msat precision
fixes #6250
2020-06-22 22:48:13 +02:00
SomberNight
9a70b79eea follow-up prev: try to handle json db int key madness :/ 2020-03-17 20:32:27 +01:00
ThomasV
e8ee4250d9 Do not save new channels before they are added to lnworker 2020-02-24 09:44:35 +01:00
ThomasV
dbceed2647 Restructure wallet storage:
- Perform json deserializations in wallet_db
 - use StoredDict class that keeps tracks of its modifications
2020-02-04 13:35:58 +01:00
ThomasV
7507942b7a (minor) json_db: add file header, fix formatting 2020-02-04 12:43:04 +01:00
ThomasV
63963323be storage: take the DB lock when writing to disk. 2020-02-03 17:08:34 +01:00
ThomasV
149cd9598a Separate JsonDB and WalletDB 2020-02-03 12:36:07 +01:00
ThomasV
aa51df0a1a Use attr.s for Feeupdates and Outpoints
Storage upgrade to version 23
2020-02-01 16:45:19 +01:00
SomberNight
18c6451518 json_db: only deserialize transactions on-demand 2020-01-19 05:49:12 +01:00
SomberNight
d2f132738a wallet: only select mature coins by default
this is a regression from #5721

Removed the `TxInput.is_coinbase` method as I think it is a confusing API,
instead we now have `TxInput.is_coinbase_input` and `TxInput.is_coinbase_output`.

related #5872
2020-01-02 00:43:49 +01:00
SomberNight
72491bdf18 synchronizer: request tx from server if we only have partial local tx
Note that there is a slight distinction between
`not tx.is_complete()` and `isinstance(tx, PartialTransaction)`,
which is that technically you can have a PSBT that is already complete
but was not yet converted to a standard bitcoin tx.
2019-12-16 21:15:20 +01:00
SomberNight
01fc048484 CLI: properly auto-upgrade storage when needed even if storage-encrypted
previously commands would error if user had an encrypted storage that needed upgrading
2019-12-15 20:12:51 +01:00
ThomasV
2c6a1f55fb Merge pull request #5825 from SomberNight/201912_local_tx_can_be_partial
wallet: allow saving partial tx as local (if it has a txid)
2019-12-15 16:40:46 +01:00
SomberNight
869a728317 wallet: use abstract base classes 2019-12-10 19:34:44 +01:00
SomberNight
30dcab0877 wallet: allow saving partial txns as local (but require txid) 2019-12-08 04:32:44 +01:00
SomberNight
8dbbc21aff wallet: better (outgoing) invoice "paid" detection
- no more passing around "invoice" in GUIs, invoice "paid" detection is now handled by wallet logic
- a tx can now pay for multiple invoices
- an invoice can now be paid by multiple txs (through partial payments)
- new data structure in storage: prevouts_by_scripthash
  - type: scripthash -> set of (outpoint, value)
  - also, storage upgrade to build this for existing wallets
2019-11-29 15:06:16 +01:00
ThomasV
61dfcba092 Refactor channel states:
- persisted states are saved
 - state transitions are checked
 - transient states are stored in channel.peer_state
 - new channel states: 'PREOPENING', 'FUNDED' and 'REDEEMED'
 - upgrade storage to version 21
2019-11-22 20:14:54 +01:00
SomberNight
e6c841d05f psbt: put fake xpubs into globals. keystores handle xfp/der_prefix missing 2019-11-04 22:24:59 +01:00
SomberNight
bafe8a2fff integrate PSBT support natively. WIP 2019-11-04 22:24:36 +01:00
ThomasV
7b828359c6 simplify get_tx_fee 2019-09-12 12:26:49 +02:00
SomberNight
482605edbb wallet: organise get_tx_fee. store calculated fees. storage version 19. 2019-09-12 08:59:27 +02:00