Commit Graph

1163 Commits

Author SHA1 Message Date
Ian Jackson 7564fabd68 Replace a match with some as_ref and as
The match was doing a deref coercion.

Found by current nightly's needless_match (whose suggestion doesn't
compile, https://github.com/rust-lang/rust-clippy/issues/8551 )
2022-03-16 14:41:04 +00:00
Nick Mathewson 4ccc3af423 dirmgr: Always send if-modified-since on consensus documents.
We never want a consensus document that's super-old, since we would
reject it immediately for being too old.

Also, never send an if-modified-since that's so old that we'd reject
the response.

Closes #403
2022-03-16 09:48:37 -04:00
Nick Mathewson 11883b9e8d Remove re-export of retry-schedule in tor-dirmgr.
It wasn't necessary.
2022-03-16 08:47:02 -04:00
Nick Mathewson 90c48dc1b0 Fix up documentation on RetryDelay.
It's no longer about downloads; it's about whatever you need to
retry.
2022-03-16 08:42:46 -04:00
Nick Mathewson 67645473d5 Move RetryDelay from dirmgr to basic-utils.
This (almost) a pure code-movement commit: it also makes one public
function private in order to suppress a warning.
2022-03-16 08:31:54 -04:00
Nick Mathewson ef619ba70a RetryDelay: add accessors for state
We'll need these for our unit tests in tor-dirmgr once we move
RetryDelay into tor-basic-utils.
2022-03-16 08:26:14 -04:00
Nick Mathewson e8c63fe3e8 tor-basic-utils/Cargo.toml: reformat.
This is a separate commit to avoid putting format changes in the
next commit.
2022-03-16 08:21:44 -04:00
trinity-1686a 60431ade4f make run_dns_resolver public and fix typo 2022-03-15 14:11:42 +01:00
trinity-1686a f4581ffd04 add simple unit test on UDP 2022-03-14 21:19:20 +01:00
trinity-1686a fa29925689 fix typos and minor issues 2022-03-14 21:16:56 +01:00
trinity-1686a 604362bf80 actually add DNS support 2022-03-14 21:04:38 +01:00
trinity-1686a 9b3be0ad47 add skeleton for DNS handling 2022-03-14 20:59:35 +01:00
trinity-1686a 4a44ef56c0 add udp to runtime 2022-03-14 20:59:33 +01:00
Nick Mathewson 8f430fd58d Merge branch 'main' into 'intern-more'
# Conflicts:
#   doc/semver_status.md
2022-03-14 14:19:44 +00:00
eta c0baf86e52 Merge branch 'missing_shrink_to_fit' into 'main'
dirmgr: Call shrink_to_fit on missing-microdesc hashtable

Closes #388

See merge request tpo/core/arti!399
2022-03-14 13:20:57 +00:00
eta 258e42ad3e Merge branch 'slice-from-ref' into 'main'
Simpify tor-netdoc TokVal

See merge request tpo/core/arti!404
2022-03-14 13:14:05 +00:00
Ian Jackson 8f5d089f53 Revert "humantime_serde_option: New module in tor-basic-utils"
We have upstreamed this code.

This reverts commit 6193c9d974.
2022-03-14 10:33:59 +00:00
Ian Jackson 409c5dd6d2 Use new upstream humantime_serde_option feature
Replace all uses of our copy of this code.
2022-03-14 10:33:59 +00:00
Ian Jackson 8b8c2a426e humantime: Update to humantime-serde 1.1.1
This has the humantime_serde::option module, which we have upstreamed
and are about to switch to.

The remaining dependency with  version = "1"  is going to be removed
in a moment.
2022-03-14 10:33:59 +00:00
Nick Mathewson a98ae2a1f0 Canonicalize and intern protover lines on descriptors
This should save around 1MB per consensus, since every relay has a
'protocols' lines, but there are only a few distinct possibilities
for such a line.

Closes #385.
2022-03-11 16:56:12 -05:00
Nick Mathewson 16f1bda324 Canonicalize and intern family representations to save memory.
This should save 1-3 MB of ram on each running arti client.

Closes #384. See also tor#27359 and proposal 298.
2022-03-11 16:54:33 -05:00
Ian Jackson 927594c3a6 Merge branch 'arti-lib' into 'main'
arti: Move all code from binary crate to library crate

See merge request tpo/core/arti!403
2022-03-11 19:57:53 +00:00
Nick Mathewson b835a48528 Merge branch 'smaller_routerstatus' into 'main'
tor-netdoc: Save allocation space in GenericRouterstatus

Closes #387

See merge request tpo/core/arti!400
2022-03-11 19:55:22 +00:00
Nick Mathewson 718a1ee340 Add a more generic implementation of intern-by-ref 2022-03-11 14:03:29 -05:00
Nick Mathewson 6f42c79654 netdoc: Call shrink_to_fit() when we're done parsing rs sections
(Once we're done parsing these, the vector will never get any
longer.)
2022-03-11 14:03:29 -05:00
Nick Mathewson 1c6c05fac3 netdoc: allocate no extra space for RouterStatus addresses.
This should save around 181K of heap on a current directory.
2022-03-11 14:03:29 -05:00
Nick Mathewson e7c584f1b3 netdoc: Make versions smaller in GenericRouterStatus
When the version is a Tor version, we can just parse it; otherwise,
we can intern it.  This shrinks GenericRouterStatus and avoids a lot
of extra help allocations.
2022-03-11 14:03:29 -05:00
Nick Mathewson 3c9093f294 tor-netdoc: reformat cargo.toml.
This is the default vscode format, and seems popular elsewhere.
This commit does nothing but reformat.
2022-03-11 14:03:29 -05:00
Nick Mathewson da82c76399 tor-netdoc: Remove unused or_port field from RS.
Part of #387
2022-03-11 14:03:29 -05:00
Ian Jackson 888eea47b3 tor-netdoc: Actually, make TokVal be a Vec
This enum was otiose: its set of valid values is precisely those of a
Vec.  (Indeed what would TokVal::Multi(vec![]) have meant?)
2022-03-11 18:42:01 +00:00
Ian Jackson 8226a68f7c arti library crate: Add another note about panicking 2022-03-11 18:28:37 +00:00
Ian Jackson 3b7d37e218 arti library crate: re-add clippy lints
I ought to have c&p these.

As per
  https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/403#note_2786099
2022-03-11 18:28:37 +00:00
Ian Jackson 4a9c660ce2 arti library crate: Add note about panicking 2022-03-11 18:28:37 +00:00
Ian Jackson 32fe5cdd5b arti: Make main module entrypoints pub
This does not constitute any kind of stable API promise.
But it might allow people to use our arti client code in novel ways.
2022-03-11 18:28:15 +00:00
Ian Jackson 4ae5f11c20 arti: Provide some rubric docs. 2022-03-11 18:28:15 +00:00
Ian Jackson e680960619 arti: Move all code from binary crate to library crate
This is just code motion and the minimal fixups.
2022-03-11 18:28:15 +00:00
Ian Jackson 9a7783a230 Remove a pointless slice
std::slice::from_ref exists.  Spotted while reviewing !400
2022-03-11 18:24:03 +00:00
Ian Jackson d320413912 Merge branch 'config-flatten' into 'main'
Make ArtiConfigBuilder contain a TorClientConfigBuilder

See merge request tpo/core/arti!401
2022-03-11 18:22:23 +00:00
Ian Jackson f914eee6bc rustfmt 2022-03-11 16:41:59 +00:00
Nick Mathewson 837c7d717b Follow-up from arti!318
Make update_config only conditionally exported; add semver-status update.
2022-03-11 11:34:51 -05:00
Nick Mathewson d95dce7c70 Merge branch 'add-flatfile-dirmgr' into 'main'
Add flatfile DirMgr

See merge request tpo/core/arti!318
2022-03-11 16:34:26 +00:00
Ian Jackson 1d281abaf8 Make ArtiConfigBuilder contain a TorClientConfigBuilder
This is an API break: now one must use `.tor()` to access the Tor
configuration parts.

But it is not a config file format break, because `#[serde(flatten)]`.
2022-03-11 16:18:27 +00:00
Ian Jackson 0bbee1e477 Make TorClientConfigBuilder Deserialize
As per #371
2022-03-11 16:18:22 +00:00
Christian Grigis 65486b185a
Expose APIs for external DirProvider 2022-03-11 13:44:48 +01:00
Nick Mathewson d9c1555229 dirmgr: Call shrink_to_fit on missing-microdesc hashtable
This hashtable starts out pretty large, but it can spend most of our
runtime (when we aren't downloading) being small.  To avoid doing
too much work, I've made it so we only call shrink_to_fit twice per
consensus: once when we're no longer pending, and once when we're
complete.

Closes #388.
2022-03-10 11:48:58 -05:00
Ian Jackson 6761cf2b69 Merge branch 'config-no-retcon' into 'main'
Drop config back-conversion from FooConfig to FooConfigBuilder

See merge request tpo/core/arti!391
2022-03-10 14:47:01 +00:00
Nick Mathewson 98c5c5b824 Merge branch 'arti-testing-part2' into 'main'
arti-testing: support for simulated TCP breakage

See merge request tpo/core/arti!392
2022-03-10 14:08:44 +00:00
Nick Mathewson d1b151436e arti-testing: Make Action implement Copy 2022-03-10 08:45:56 -05:00
Christian Grigis 335af4ac30
Add methods to `UnverifiedConsensus` 2022-03-10 11:29:58 +01:00
eta 4f34f7cea8 Merge branch 'ticket_391' into 'main'
arti-bench: don't allocate a separate receive-buffer for each job

Closes #391

See merge request tpo/core/arti!395
2022-03-09 13:26:44 +00:00
Ian Jackson 3425a6cbcd Merge branch 'ticket_383' into 'main'
tor-chanmgr: Do not allocate an Internal error unless we mean it.

Closes #383

See merge request tpo/core/arti!394
2022-03-09 11:30:08 +00:00
Nick Mathewson 26bdbc4b61 arti-bench: don't allocate a separate receive-buffer for each job
This makes heap profiling more viable.

Closes #391.
2022-03-08 16:41:06 -05:00
Nick Mathewson a110755193 tor-chanmgr: Do not allocate an Internal error unless we mean it.
Previously we'd allocate an error as a place-holder here, but it's
not a great idea to do that with a `Bug`: each `Bug` stores a whole
stack trace, which uses a whole pile of allocations to construct.

Now we keep an `Option<Error>` instead.

Found while heap profiling.

Closes #383.
2022-03-08 14:04:25 -05:00
Nick Mathewson a460de6e75 arti-testing: reverse two more attrs in attempt to fix min-versions 2022-03-07 15:29:44 -05:00
Nick Mathewson 84536a9c1e arti-testing: reverse two attrs in attempt to fix min-versions 2022-03-07 14:34:02 -05:00
Nick Mathewson de0cf90a32 arti-testing: todo comment cleanup 2022-03-07 11:47:25 -05:00
Nick Mathewson a86f00c222 arti-testing: support for conditional TCP failure. 2022-03-07 11:44:43 -05:00
Ian Jackson 56c2153b9e Merge branch 'fix-expand-tilde-on-windows' into 'main'
Fix expand tilde and invalid path chars on windows

See merge request tpo/core/arti!274
2022-03-07 16:35:15 +00:00
Nick Mathewson 3aed633559 arti-testing: add support for black-holing TCP connections. 2022-03-07 11:26:13 -05:00
Ian Jackson 1b1ce8cc82 Drop remaining conversion from FooConfig to FooConfigBuilder 2022-03-07 15:58:53 +00:00
Ian Jackson 56cb1cef4e Have ArtiConfig contain a TorClientConfig, and drop builder retcon
Replace the recapitulation of TorClientConfig fields in ArtiConfig and
instead just have it contain one.  This is part of #374.

The conversions from ArtiConfig back to ArtiConfigBuilder and
TorClientConfigBuilder would need to change, but, since we don't want
them anyway,

No longer impl Deserialize for ArtiConfig.  (As per #371 this will
want to become a private type.)

No longer impl From<ArtiConfig> for ArtiConfigBuilder and
TorClientConfigBuilder.  And abolish tests of that code.

(This all has to be in one commit, because previously
ArtiConfig::tor_client_config used the validated-to-builder config
retcon.)
2022-03-07 15:58:53 +00:00
Ian Jackson 92d1855b0e Provide way to get TorConfigBuilder from ArtiConfigBuilder
This is needed according to #372, where we observe that builders ought
not to be generated from validated structs.  So we need this
conversion.
2022-03-07 15:58:53 +00:00
Ian Jackson 56bd15b671 Derive Deserialize for handwritten ArtiConfigBuilder 2022-03-07 15:58:53 +00:00
Ian Jackson 5203311a97 Derive Deserialize for derive-builder-generated config builders
I used
  git-grep -P '\#\[serde\((?!default|deny_unknown)'
to find places where I needed to add additional attributes on the
builder method fields.

This is currently a bit duplicative, but when #371 is completely done,
the validated (non-builder) configs won't need to be Deserialize any
more.

This is part of #371 and #372.
2022-03-07 15:58:53 +00:00
Ian Jackson 416b56d852 Use git source for derive_builder for now, for attrs feature
We are going to want to specify custom attributes on fields of the
builder struct.  This feature was missing from derive_builder.

This commitid is the current head of my MR branch
  https://github.com/colin-kiegel/rust-derive-builder/pull/237
  https://github.com/ijackson/rust-derive-builder/tree/builder-field-attrs
Using the commitid prevents surprises if that branch is updated.

We will require this newer version of derive_builder.  The version
will need to be bumped again later, assuming the upstream MR is merged
and upstream do a release containing the needed changes.
2022-03-07 15:56:57 +00:00
Nick Mathewson 5d91fa7f5a arti-testing: CLI for making TCP connections break. 2022-03-07 10:52:04 -05:00
eta c27dde59b6 Merge branch 'moderately-enthused-eyeballs' into 'main'
Implement a basic form of RFC 8305 ("happy eyeballs") for channels

See merge request tpo/core/arti!382
2022-03-07 15:26:37 +00:00
Nick Mathewson ca6070b962 arti-testing: Initial support for broken TCP.
This commit adds support for a BrokenTcp provider that can make
connection attempts fail or time out.  It doesn't yet have a way to
turn on the failure.
2022-03-07 10:12:48 -05:00
eta c98d9dc5fe Implement a basic form of RFC 8305 ("happy eyeballs") for channels
This makes Arti usable in IPv6-only environments (arti#92) by letting us
attempt multiple connections to a given relay using all of its
addresses instead of just using the first (probably IPv4) one, using the
strategy from RFC 8305 § 5.

This isn't a complete implementation of Happy Eyeballs; ideally, we'd
sort the address list before doing concurrent connections. However, it
works (and has been tested inside an IPv6-only container inside eta's
network :p)
2022-03-07 14:56:51 +00:00
Nick Mathewson cc6ba72dc3 Merge branch 'bench_multicirc' into 'main'
arti-bench: support multiple streams per circuit, multiple circuits per sample.

Closes #380

See merge request tpo/core/arti!384
2022-03-07 14:11:53 +00:00
Nick Mathewson 49216a5202 arti-bench: documentation fixes 2022-03-07 08:42:20 -05:00
Nick Mathewson 22ee7671f8 arti-bench: record streams_per_circ and circs_per_sample. 2022-03-07 08:38:02 -05:00
Ian Jackson 6193c9d974 humantime_serde_option: New module in tor-basic-utils
This will be used to allow our config *builder* structs to be
Deserialize.
2022-03-07 12:38:30 +00:00
Michael 206714f2e4 Test shell variable expansion on windows 2022-03-05 01:40:42 +00:00
Lennart Kloock ef21559df1
Add config paths table in `arti` docs 2022-03-04 22:43:14 +01:00
Nick Mathewson cb846de367 Merge branch 'reproducible-build-fix-summary-git' into 'main'
fix reproducible build summary giving invalid branch and commit id

Closes #378

See merge request tpo/core/arti!383
2022-03-04 19:52:19 +00:00
Nick Mathewson 994a8f7cfa Merge branch 'errors-for-dirprovider' into 'main'
Errors for dirprovider

Closes #370

See merge request tpo/core/arti!385
2022-03-04 19:51:23 +00:00
trinity-1686a 5e5864802d fix reproducible build summary giving invalid branch and commit id
fix #378
also fix unrelated error in nightly rustdoc CI
2022-03-04 19:25:40 +01:00
Ian Jackson b095265257 Merge branch 'educe-traits' into 'main'
Replace many manual trait impls with use of educe

See merge request tpo/core/arti!375
2022-03-04 18:00:17 +00:00
Nick Mathewson e4ed8241ba Fix an unwrap() error. 2022-03-04 12:37:22 -05:00
Ian Jackson 055b7399f2 Provide ErrorKind::Other 2022-03-04 17:35:34 +00:00
Ian Jackson 4bff33c397 Provide an error variant for external directory providers 2022-03-04 17:35:34 +00:00
Ian Jackson 952aef1be3 Merge branch 'hex_decode' into 'main'
Speed up RsaIdentity decoding

Closes #377

See merge request tpo/core/arti!381
2022-03-04 17:22:52 +00:00
Nick Mathewson 8c34896651 arti-bench: use isolation to support multiple circuits per run. 2022-03-04 12:22:43 -05:00
Ian Jackson 0cfb6a0e0f tor-basic-utils: copy the README into lib.rs
The doc include rune does not work with our MSRV; it needs 1.54.

The alternative would be some kind of cfg() but that would
  - not provide the crate-level doc on Rust 1.53
  - involve the use of cfg_attr

Instead, just do it the old way.
2022-03-04 16:54:54 +00:00
Nick Mathewson 42245de9f5 arti-bench: simplify future-creation code slightly 2022-03-04 11:45:45 -05:00
Nick Mathewson c978a0d78b arti-bench: refactor stream construction to connect in parallel
Previously we tried to do each connection in a run, and only then did we
start transferring data over them.  Now we collect a bunch of the
futures that return an open stream, and run them all in parallel
with using them.  This change includes connect-time in our
benchmarks, and allows us to test contention in our connect code.

Instead of using a Stream, I've changed the connection-generation
code to call a future-returning function directly, so we have a way
to explicitly pass which run we're in.
2022-03-04 11:41:51 -05:00
eta 8d26726fdf Merge branch 'arti-testing-part1' into 'main'
arti-testing: Initial implementation

See merge request tpo/core/arti!378
2022-03-04 16:29:41 +00:00
Nick Mathewson c23dad8c2d Use RsaIdentity::from_hex() and hex::decode_to_slice in more places
These aren't critical-path, but they do make the code a little nicer.
2022-03-04 09:23:26 -05:00
Nick Mathewson caf372ac0f tor-netdoc: Use RsaIdentity::from_hex() in critical path.
This commit changes the main parsing code for RsaIdentity in
tor-netdoc, and .

Previously, parse_hex_ident was something like 10% of our startup
CPU time; now it's only like ~2%.  (Still not perfect, but way
better.)

Closes #377.
2022-03-04 09:05:04 -05:00
Nick Mathewson b6452b4812 Add a from_hex method for RsaIdentity.
We perform this operation in a bunch of places, and most of them
use hex::decode().  That's not great, since hex::decode() has to do
heap allocation.  This implementation uses hex::decode_to_slice(),
which should be faster.

(In the future we might choose to use one of the faster hex
implementations, but I'm hoping that this change will be sufficient
to get hex decoding out of our profiles.)

Part of #377.
2022-03-04 08:55:34 -05:00
Ian Jackson 6e1da0145b Remove now-obsolete note about location of skip_fmt
Now the diff from the merge base does not contain any hits for ^\+.*bytes
2022-03-04 11:45:24 +00:00
Ian Jackson ebfd734956 Move skip_fmt into tor-basic-utils
Code motion and the minimal mechanical changes.

As per
  https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/375#note_2783078
2022-03-04 11:45:24 +00:00
Ian Jackson aba68088d1 Introduce tor-basic-utils
Empty crate right now
2022-03-04 11:45:24 +00:00
Ian Jackson eb66d0af6f Merge branch 'derive-traits' into 'main'
Tidy up many open-coded trait impls

See merge request tpo/core/arti!374
2022-03-04 11:25:57 +00:00
Ian Jackson 64cc5b8537 Move educe to dev-dependencies of tor-bytes
As per
  https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/375#note_2783080

This is going to move to a new crate, but doing this now makes
things slightly clearer for me.
2022-03-04 11:03:04 +00:00
eta e9f8ddfb9c Fix introduced junk in doc comment. 2022-03-04 10:56:40 +00:00
Ian Jackson 2a7915e6f6 Merge branch 'return_ref_arc' into 'main'
arti-client: Make dirmgr() and circmgr() return &Arc<..>

Closes #369

See merge request tpo/core/arti!379
2022-03-04 10:55:21 +00:00
Nick Mathewson a4914b990b arti-client: Make dirmgr() and circmgr() return &Arc<..>
Previously they returned an Arc, which wasn't necessary unless the
client actually _wanted_ a new Arc.

This would be an API break, except that these functions are marked
'experimental-api', so semver does not apply; nonetheless I've noted
the break in semver_status.md, just in case we care.

Closes #369
2022-03-03 14:08:27 -05:00
Nick Mathewson 06f0339b97 Allow println in arti-testing. 2022-03-03 14:01:44 -05:00