Commit Graph

547 Commits

Author SHA1 Message Date
Nick Mathewson d1c362f308 bootstrap reporting: Documentation fixups from review. 2022-01-19 08:22:42 -05:00
Nick Mathewson f34faec7d9 tor-dirmgr: Create a bootstrap-status exporting mechanism.
The interface is similar to the one exposed by `arti-client`: it
internally uses postage::watch to give a series of events showing
when a bootstrap status is changing.

Thanks to the existing state/driver separation in the DirMgr design
we don't need much new logic: each download state needs to expose
(internally) how far along it is in its download, which the
bootstrap code passes to the DirMgr if it has changed.

I believe that in the long run, we'll probably want to expose more
(or different) information here, and we'll want to process it
differently.  With that in mind, I've made the API for
`DirBootstrapStatus` deliberately narrow, so that we can change its
of its internal later on without breaking code that depends on it.

(The information exposed by this commit is not yet summarized in
`arti-client`.)

Part of #96.
2022-01-18 13:06:08 -05:00
Nick Mathewson 7d3482ca1a Bump all crate versions to 0.0.3. 2022-01-11 09:40:32 -05:00
eta de1b19c22a Fixup broken doc comment from arti!220
That's what I get for blindly trusting @nickm :p
2022-01-10 15:10:41 +00:00
eta da848a1b9c Merge branch 'ticket_178' into 'main'
Fix ticket 178: Don't use a NetDir until we have microdescriptors for all of our primary guards.

Closes #178

See merge request tpo/core/arti!220
2022-01-10 14:02:24 +00:00
eta 4b1a3976cb Merge branch 'bug274' into 'main'
Fix busy-loop when consensus is near expiring.

Closes #274

See merge request tpo/core/arti!223
2022-01-10 13:47:52 +00:00
Nick Mathewson 4841b50c9f Minimize the required version for each dependency.
I found these versions empirically, by using the following process:

First, I used `cargo tree --depth 1 --kind all` to get a list of
every immediate dependency we had.

Then, I used `cargo upgrade --workspace package@version` to change
each dependency to the earliest version with which (in theory) the
current version is semver-compatible.  IOW, if the current version
was 3.2.3, I picked "3".  If the current version was 0.12.8, I
picked "0.12".

Then, I used `cargo +nightly upgrade -Z minimal-versions` to
downgrade Cargo.lock to the minimal listed version for each
dependency.  (I had to override a few packages; see .gitlab-ci.yml
for details).

Finally, I repeatedly increased the version of each of our
dependencies until our code compiled and the tests passed.  Here's
what I found that we need:

anyhow >= 1.0.5: Earlier versions break our hyper example.

async-broadcast >= 0.3.2: Earlier versions fail our tests.

async-compression 0.3.5: Earlier versions handled futures and tokio
    differently.

async-trait >= 0.1.2: Earlier versions are too buggy to compile our
    code.

clap 2.33.0: For Arg::default_value_os().

coarsetime >= 0.1.20: exposed as_ticks() function.

curve25519-dalek >= 3.2: For is_identity().

generic-array 0.14.3: Earlier versions don't implement
    From<&[T; 32]>

httparse >= 1.2: Earlier versions didn't implement Error.

itertools at 0.10.1: For at_most_once.

rusqlite >= 0.26.3: for backward compatibility with older rustc.

serde 1.0.103: Older versions break our code.

serde_json >= 1.0.50: Since we need its Value type to implement Eq.

shellexpand >= 2.1: To avoid a broken dirs crate version.

tokio >= 1.4: For Handle::block_on().

tracing >= 0.1.18: Previously, tracing_core and tracing had separate
    LevelFilter types.

typenum >= 1.12: Compatibility with rust-crypto crates

x25519-dalek >= 1.2.0: For was_contributory().

Closes #275.
2022-01-07 19:08:58 -05:00
Nick Mathewson d9b423c17f Fix busy-loop when consensus is near expiring.
When our current consensus is getting close to being invalid (but
it isn't invalid yet), we try to get a new one. So far, so good.

But we had a bug: when we went to get a new consensus, we'd see that we
had a perfectly fine not-yet-invalid consensus in our cache, reload it,
find that it was ready, and continue!

This patch fixes our behavior: If we have a usable consensus, then
when we reset the bootstrapping process, we ignore any cached consensus.

Fixes bug #274.
2022-01-07 13:16:06 -05:00
Nick Mathewson e8e0f67336 Only replace netdir once we have enough guard MDs.
This prevents a security-failure condition that could happen if our
directory caches don't give us these microdescriptors, but we
nevertheless decide that the directory is usable.

Closes #178
2022-01-06 15:58:43 -05:00
Daniel Eades 592642a9e6 extend lints to include 'clippy::all' 2021-12-28 20:15:40 +00:00
Tor CI Release e17359bc18 fallbackdir: Update list generated on December 15, 2021
Signed-off-by: Tor CI Release <no-email@torproject.org>
2021-12-16 11:52:22 -05:00
Nick Mathewson 9f6e09b4d1 tor-dirmgr: retain cache_usage on reset.
Previously we didn't retain the value of our cache_usage field when
calling reset() from GetMicrodescsState.

This resolves an XXXX comment.
2021-12-16 09:31:23 -05:00
eta e62212112c Merge branch 'flag-publish-2' into 'main'
Refactor directory events to use a new FlagPublisher mechanism.

See merge request tpo/core/arti!188
2021-12-16 14:28:51 +00:00
Nick Mathewson b5672cb2c7 tor-dirmgr: Warn on a bug case.
This warning occurs if we ask for microdescriptors from our local
cache, and our cache gives us something we didn't ask for.  It
shouldn't be possible, so let's warn when it occurs.

This patch resolves an XXXX.
2021-12-16 08:51:34 -05:00
Nick Mathewson 7f4578bec1 Refactor directory events to use a FlagPublisher mechanism.
This approach tries to preserve the current interface, but uses a
counter-based event backend to implement a coalescing stream of
events that can be represented as small integers.  The advantage
here is that publishing events no longer needs to be a blocking
operation, since there is no queue to fill up.
2021-12-14 14:05:02 -05:00
eta 8040f7afb2 Merge branch 'reconfigure' into 'main'
Make most arti-client fields reconfigurable.

See merge request tpo/core/arti!181
2021-12-13 14:49:23 +00:00
Trinity Pointard 9753a7ee06 fix nightly clippy errors 2021-12-09 12:16:23 +01:00
Trinity Pointard 7dd1a20d42 update rusqlite and revert minimal version change 2021-12-09 11:51:39 +01:00
Nick Mathewson 4536c2ac87 Upgrade to digest v0.10.0
We generally try to track the latest rust-crypto traits when we can:
fortunately, this upgrade didn't break much, considering.
2021-12-07 20:33:46 -05:00
Nick Mathewson b3b6a92145 Add a few tests to tor-config. 2021-12-07 19:52:49 -05:00
Nick Mathewson 42ae8c7a2a Make override_net_params take effect sooner.
This is still not as soon as I'd like: a real change here will require
refactoring DirMgr::notify().
2021-12-07 19:26:01 -05:00
Nick Mathewson e8ca0d7742 Make much of DirMgrConfig reconfigurable.
We can't change the authorities while in-flight: that would be pretty
miserable to implement.

Similarly we can't change the cache while in-flight.

Everything else should be fair game, though there are a couple of tricky
bits.  I've tried to document those.
2021-12-07 17:36:16 -05:00
Nick Mathewson 606d64eac5 Sketch API for reconfiguration.
This patch doesn't actually make anything reconfigurable, but it
does create an API that will tell you "you can't change the value of
that!"  If the API looks reasonable, I can start making it possible
to change the values of individual items.
2021-12-07 15:42:14 -05:00
Nick Mathewson 31b385c5b2 Resolve roughly half of the XXXXs.
We want to only use TODO in the codebase for non-blockers, and open
tickets for anything that is a bigger blocker than a TODO.  These
XXXXs seem like definite non-blockers to me.

Part of arti#231.
2021-12-06 15:11:03 -05:00
eta a4ec8476a7 Merge branch 'tor-dirmgr' into 'main'
Don't warn in bootstrap_from_config when error is  Error::ManagerDropped

See merge request tpo/core/arti!157
2021-12-01 13:00:39 +00:00
Nick Mathewson a49ee4d5f4 Merge remote-tracking branch 'origin/mr/159' 2021-11-30 13:53:53 -05:00
Nick Mathewson 3b072c5420 Merge branch 'readme_fixes' 2021-11-30 09:12:51 -05:00
dagon 9a8d883f35 remember utf8 check 2021-11-30 23:26:47 +10:00
dagon 0122e15339 dont warn on bootstrap error when error == manager dropped 2021-11-30 21:50:27 +10:00
Nick Mathewson eef81d9d57 Bump every crate by one patch version. 2021-11-29 15:21:58 -05:00
Nick Mathewson eb861b7edd Merge branch 'config-updates-and-tests' 2021-11-29 13:59:41 -05:00
dagon d5c48c616f run ./maint/readmes.sh 2021-11-29 21:29:28 +10:00
Nick Mathewson f2034ac6e2 Add basic tests for high-level builders
Make sure that we can change elements, and we can reconstruct builders
that give us the same thing.
2021-11-25 09:40:16 -05:00
Daniel Eades db16d13df4 add semicolons if nothing returned 2021-11-25 13:20:37 +00:00
Daniel Eades 052f51ff71 deglob some enums, use concise iteration syntax 2021-11-25 12:39:52 +00:00
Nick Mathewson 3e7e599a22 More typo fixes that I forgot to save :( 2021-11-24 18:23:12 -05:00
Nick Mathewson 40ff7113d4 Make every Config type implement Eq.
Doing this is necessary for reconfiguration support, and will help a lot
with testing, too.
2021-11-21 12:06:15 -05:00
Nick Mathewson 97f5a7a357 Give every ConfigBuilder a From<Config> implementation.
This will make it more convenient to reconfigure things.
2021-11-21 10:54:34 -05:00
Nick Mathewson 934412586e Use named fields for the elements of ConfigBuildError 2021-11-18 14:31:34 -05:00
Nick Mathewson 96659a850b Rename RetryConfig to DownloadSchedule, fold in parallelism. 2021-11-18 12:33:08 -05:00
Nick Mathewson 5184f5ba84 Move top-level configuration downwards from `arti` to `arti-config`.
To do this at all neatly, I had to split out `tor-config` from
`arti-config` again, and putting the lower level stuff (paths,
builder errors) into tor-config.  I also changed our use of
derive_builder to always use a common error type, to avoid
error type proliferation.
2021-11-18 11:37:48 -05:00
Nick Mathewson 2439129899 tor-dirclient: Put routerdesc download behind a feature.
Part of #125
2021-11-12 13:38:02 -05:00
Nick Mathewson e9ea7515bc tor-dirmgr: put routerdesc storage behind a feature.
(We keep routerdescs in the schema, since we don't want _that_ to
fragment.)

Part of #125.
2021-11-12 12:15:48 -05:00
Dimitris Apostolou ad3c18a456
Fix typos 2021-11-12 13:54:50 +02:00
Nick Mathewson 99aefe4bb8 Remove usage of tracing-test 0.1
It requires tracing-subscriber 0.2, which is a lower version than we
want, and which causes trouble with our minimal-versions CI test.

There is a pending issue to fix this; we can reinstate tracing-test
once it is merged:  https://github.com/dbrgn/tracing-test/pull/11
2021-11-11 11:01:32 -05:00
Nick Mathewson 4717bb270f Tests for tor-dirmgr::bootstrap 2021-11-11 08:58:48 -05:00
Nick Mathewson 3609aff10e Minor tests for DirMgr::query_into_requests 2021-11-11 07:31:23 -05:00
Nick Mathewson 585c6c4f9e tor-dirmgr: tests for making and expanding consensus requests. 2021-11-10 09:04:21 -05:00
Nick Mathewson 248aca98f1 tor-dirmgr: Tests for high-level loading functions. 2021-11-10 08:18:30 -05:00
Nick Mathewson 6899b2599b More tests on tor-dirmgr::state
These test our download schedules, resetting to the original state,
and storing downloaded objects.
2021-11-08 08:21:44 -05:00
Nick Mathewson 8833f7a3a2 Remove one more test println!(). 2021-11-04 11:11:26 -04:00
Nick Mathewson e3f8a9f94f tor-dirmgr: tests for docid module. 2021-11-04 09:47:13 -04:00
Nick Mathewson cdae4c033c Remove some dbg!() calls in real code. 2021-11-02 09:50:55 -04:00
Nick Mathewson e6e740646a Bump all crate versions to 0.0.1 2021-10-29 11:05:51 -04:00
Nick Mathewson c11fe200d1 Try writing a bit more documentation for configuration types. 2021-10-28 20:49:17 -04:00
Nick Mathewson b160177b63 Oops: add missing data files 2021-10-28 19:51:16 -04:00
Nick Mathewson 4a721da54c tor-dirmgr: tests for GetMicrodescsState 2021-10-28 14:21:21 -04:00
Nick Mathewson 2f149621fd tor-dirmgr: tests for download schedules. 2021-10-28 13:08:46 -04:00
Nick Mathewson 7e8891b861 tor-dirmgr: Test for GetCertsState 2021-10-28 10:21:56 -04:00
Nick Mathewson 8956e6cd8b tor-dirmgr: Tests for GetConsensusState.
This test uses a consensus that I've copied from
tor-netdoc/testdata.  I would include it directly, but I think that
will cause trouble when it comes time to run "cargo package".
2021-10-28 09:35:18 -04:00
Nick Mathewson 5e27c46d24 tor-dirmgr::state: Mock the view of current time.
This will let us test the state processing code without having to
give it up-to-date directory objects.
2021-10-28 08:41:59 -04:00
Nick Mathewson 2fae63b5ae Upgrade to fslock version 0.2
This version makes all locks per-handle rather than per-process, by
moving from lockf() to flock() on unix.
2021-10-27 14:52:09 -04:00
Nick Mathewson 968ffa3d6c Migrate tor-dirmgr from chrono to time 0.3
(This appears to be the emerging consensus of how to handle
RUSTSEC-2020-0159.)
2021-10-24 11:12:06 -04:00
Nick Mathewson 730be38867 Replace references to arti-client in the documentation. 2021-10-21 14:22:21 -04:00
Nick Mathewson 6af5700c5d Remove #![allow(unreachable_pub)] in shared_ref.rs 2021-10-21 14:00:38 -04:00
Nick Mathewson f3dc66d964 Merge branch 'share_state' 2021-10-21 13:34:38 -04:00
David Goulet b863e63a21 fallbackdir: Regenerate list for October 2021
Closes #200

Signed-off-by: David Goulet <dgoulet@torproject.org>
2021-10-21 09:18:03 -04:00
Nick Mathewson fdddb74de4 Mark consensus as "not-pending" even if its microdescs come from cache.
Previously our code would clear the 'pending' flag on a consensus
only when a _downloaded_ md made it become usable.

Closes #199.
2021-10-20 14:04:54 -04:00
Nick Mathewson 43506601dc Move tor-dirmgr to use a sync::Mutex.
The futures:🔒:Mutex was unnecessary, since we never held it
when we were suspending.
2021-10-20 13:46:44 -04:00
Nick Mathewson 445ec6d220 Fix most warnings from nightly.
(One represents code that I forgot to write.)
2021-10-19 16:21:12 -04:00
Nick Mathewson 36353aacd8 Fix some typos in comments.
Also, tell the "typos" tool to ignore Cargo.lock.
2021-10-19 10:40:32 -04:00
Nick Mathewson 457e7f064c Reverse two swapped log messages 2021-10-19 10:24:19 -04:00
Nick Mathewson 26e1b6fb72 Improve logging when trying to get the lock on the directory cache.
Previously we'd say that we were "waiting for the other process to
bootstrap" even if it was already bootstrapped: and we wouldn't
actually declare success when it was done.
2021-10-19 10:18:02 -04:00
Jani Monoses f683d4f3cd Remove useless into() conversions caught by clippy. 2021-10-18 13:22:53 +03:00
Jani Monoses 8b8228190d Remove anyhow dependency from tor-dirmgr. 2021-10-18 13:22:53 +03:00
Jani Monoses 6181e623f8 Use unwrap in tests.
For now, this avoids having to separately handle
AuthorityBuilderError, DirMgrConfigBuilderError, DownloadScheduleConfigBuilderError,
NetworkConfigBuilderError and FallbackDirBuilderError when anyhow is not
used.

Turn off a clippy warning.
2021-10-18 13:22:39 +03:00
Nick Mathewson 3b7b43ae5f Turn off default-features in chrono where possible. 2021-10-17 10:37:30 -04:00
Nick Mathewson 02b20edeb3 Add a few tracing directives to tor-dirmgr. 2021-10-13 13:37:18 -04:00
Nick Mathewson e9399ec6c0 tor-dirmgr: report bootstrap success on all successful cases.
Previously we would sometimes fail to report that we had
successfully bootstrapped.
2021-10-13 13:01:55 -04:00
Nick Mathewson 8113a8ba12 Don't report the bootstrap as completed unless it actually succeeds.
(Previously we'd report it as successful even if the inner download
task was a failure.)
2021-10-13 12:19:40 -04:00
Nick Mathewson 33ba697b5d Notify guard manager on network change and state flush. 2021-10-10 12:38:50 -04:00
Nick Mathewson 58da76c6c4 Lock down some dirmgr config functions. 2021-10-09 17:30:06 -04:00
Nick Mathewson 58355d7d54 Re-export configuration types from tor-client. 2021-10-09 17:30:06 -04:00
Nick Mathewson af7c9d5a0b enable checked_conversions lint. 2021-10-09 16:53:13 -04:00
Nick Mathewson 6c6785c04d Fix a type inference complaint in tor-dirmgr's tests.
I have no idea why this just turned up now.
2021-10-07 09:30:08 -04:00
Nick Mathewson 7bdb6e7633 Update cargo.lock; upgrade to newer rusqlite. 2021-10-05 08:10:04 -04:00
Trinity Pointard 954395573a update fslock to compile android 2021-09-25 12:01:45 +02:00
Nick Mathewson b51e2ff5b3 Upgrade memmap2, directories, and dirs dependencies. 2021-09-23 08:26:29 -04:00
Daniel Eades 9a3b183679 remove unused dep 2021-09-19 09:47:57 +01:00
Jani Monoses b0bb7bd100 Do not use set_ prefix on derive_builder setters 2021-09-16 08:41:04 +03:00
Nick Mathewson 0949582fab Use derive_builder for Authority and FallbackDir. 2021-09-10 10:03:33 -04:00
Nick Mathewson 1c97918d67 Typo fixes 2021-09-09 13:06:05 -04:00
Nick Mathewson a00248723d Use derive_builder for the rest of tor_dirmgr::config 2021-09-09 07:38:12 -04:00
Nick Mathewson 372c38e041 Use derive_builder for DownloadScheduleConfig 2021-09-08 17:55:18 -04:00
Nick Mathewson d42d910a4f Use derive_builder for NetworkConfig. 2021-09-08 17:46:00 -04:00
Daniel Eades fb3b8b84b5 fix/silence clippy lints in test modules 2021-09-08 17:28:31 +02:00
Nick Mathewson 557a0ff40b Move all crates into a `crates` subdirectory.
This will cause some pain for now, but now is really the best time
to do this kind of thing.
2021-08-27 09:53:09 -04:00