Now it contains either an `OwnedChanTarget` or an `OwnedCircTarget`,
which will let `GuardMgr` return bridges that can be used to make
circuits.
As part of this change, it was necessary to revise some
address-modification functions that applied to filters and
`OwnedChanTarget`. Now they do the smart thing, and remove only the
address that are in the `ChanMethod`. This means that the addresses
from HasAddrs are still accurate about which addresses the relay
"has".
The most important part of this commit is to make sure that each
`FirstHopId` includes the `GuardSetSelector` from which the guard
was selected. Doing this lets us be certain that when we report
that a guard has succeeded or failed, we're reporting it in the
right context.
Additionally, this commit uses strum to make an iterator over the
samples, so that we can make sure that our "for each sample" code is
robust against future changes, and we don't miss the bridge sample.
Now keyed by Arc<BridgeConfig>, and the values can be errors.
Currently there is no implementation so there can't be any errors,
but the error enum will become nonempty.
The feature we want is `#[doc = include_str!("README.md")]`, which is
stable since 1.54 and our MSRV is now 1.56.
This commit is precisely the result of the following Perl rune:
perl -i~ -0777 -pe 's{(^//!(?!.*\@\@).*\n)+}{#![doc = include_str!("../README.md")]\n}m' crates/*/src/lib.rs
HasAddr used to mean "Here are addresses that I have, at which I can
be contacted." But "Where (and how) can I be contacted?" is now a
question for HasChannelMethod to answer.
(We still need to have "HasAddr", though, so we can answer things
like "what country is this relay in" and "are these relays in the
same /8?")
So this commit introduces:
* A new trait for adding an implementation of HasChannelMethod in
terms of HasAddr.
* A requirement on ChanTarget that it needs to implement
HasChannelMethod.
There is some temporary breakage here, marked with "TODO pt-client",
that I'll fix later in this branch.
Also add a BridgeRelayWithDesc type (name tbd) to guarantee that
a bridge relay really does have a known descriptor before you
try to build a circuit with it.
This is the one we'll actually use to connect to bridges. It
has a `Bridge` line, and an optional `BridgeDesc`.
Maybe this will turn into a `BridgeRelay<'a>` by analogy to `Relay`
some time; I'm not sure.
BridgeDesc is a separate type to make sure that we do not confuse
bridges' descriptors with the descriptors from other routers down
the road. (Bridges' descriptors need to be used differently, and
treated as more private.)
With this code, BridgerDescList is now just an alias for
`ByRelayIds<BridgeDesc>`, which is pretty keen.
Nightly rust gives a warning about this "pub use", but the warning
is a false positive. Since it doesn't seem to be going away in a
hurry, let's suppress it for now.
Include the offending word in all the applicable errors.
Always print it with {word:?}.
As a consequence, there are no From impls any more and error
generation/conversion is by hand in all cases.
Clarify InvalidPtOrAddr vs InvalidIAddrorPt, and don't make the
attempted parse be a source error for those.
Where we still have source errors, don't print them in Display.
(Since the APIs for the `Schedule::sleep*` functions changed, this
is a breaking change in tor-rtcompat. Therefore, the Runtime trait
in tor-rtcompat is now a different trait. Therefore, anything that
uses the Runtime trait in its APIs has also broken.)
This covers only the most basic notions of working with bridges:
that we need a separate set of guards, and that they have to
come from the list of known bridges.
Because we want to work more on ensuring that our semver stability
story is solid, we are _not_ bumping arti-client to 1.0.0 right now.
Here are the bumps we _are_ doing. Crates with "minor" bumps have
had API breaks; crates with "patch" bumps have had new APIs added.
Note that `tor-congestion` is not bumped here: it's a new crate, and
hasn't been published before.
```
tor-basic-utils minor
fs-mistrust minor
tor-config minor
tor-rtcompat minor
tor-rtmock minor
tor-llcrypto patch
tor-bytes patch
tor-linkspec minor
tor-cell minor
tor-proto minor
tor-netdoc patch
tor-netdir minor
tor-persist patch
tor-chanmgr minor
tor-guardmgr minor
tor-circmgr minor
tor-dirmgr minor
arti-client minor
arti-hyper minor
arti major
arti-bench minor
arti-testing minor
```
With this change, each individual identity type becomes optional.
The functions that expose them unconditionally are now in a "legacy"
trait that only some downstream types are expected to implement.
There are new convenience APIs in HasRelayIds:
* to return Option<&keytype>,
* to see if one identity-set contains another.
This commit will break several downstream crates! For the
reviewer's convenience, I will put the fixes for those crates into a
series of squash! commits on this one.
tor-netdir
----------
Revise tor-netdir to accept optional identities. This required some
caveats and workarounds about the cases where we have to deal with a
key type that the tor-netdir code does not currently recognize at
all. If we start to add more identity types in the future, we may
well want more internal indices in this code.
tor-proto
---------
In order to make tor-proto support optional identities, there were
fewer changes than I thought. Some "check" functions needed to start
looking at "all the ids we want" rather than at "the two known IDs";
they also needed to accommodate that case where we don't have an ID
that we demand.
This change will also help with bridges, since we want to be able to
connect to a bridge without knowing all of its IDs up front.
The protocol currently _requires_ the two current ID types in some
places. To deal with that, I added a new `MissingId` error.
I also removed a couple of unconditional identity accessors for
chanmgr; code should use `target().identity(...)` instead.
tor-chanmgr
-----------
This is an incomplete conversion: it does not at all handle channel
targets without Ed25519 identities yet. It still uses those
identities to index its internal map from identity to channel; but
it gives a new `MissingId` error type if it's given a channel target
that doesn't have one.
We'll want to revise the map type again down the road when we
implement bridges, but I'd rather not step on the channel-padding
work in progress right now.
tor-guardmgr
------------
This change is mostly a matter of constructing owned identity types
more sensibly, rather than unwrapping them directly.
There are some places marked with TODOs where we still depend on
particular identity types, because of how the directory protocol
works. This will need revisiting when we add bridge support here.
tor-circmgr
-----------
These changes are just relatively simple API changes in the tests.
We want the set of identities supported by a relay to be extensible
in the future with minimal fuss; we'd also like to make working
with these ID sets more convenient. To handle that, this commit
adds a new trait for "Something that has the same IDs as a relay"
and a new object for "an owned representation of a relay's IDs."
This commit introduces a similar trait for "Something with a list of
SocketAddr, like a relay has." There's no owned equivelent for
that, since Vec<SocketAddr> is already a thing.
Closes#428.
Do _not_ bump the dependency versions on crates that have had no
changes since arti 0.0.5, since those crates do not depend on the
new APIs.
```
cargo set-version -p tor-basic-utils --bump patch
cargo set-version -p tor-llcrypto --bump patch
git restore crates/tor-checkable
git restore crates/tor-consdiff
git restore crates/tor-rtmock
```
This performs the transitive closure of the last operation:
everything that depends on a crate with a breaking change gets the
version which it depends on bumped.
```
cargo set-version -p tor-proto --bump minor
cargo set-version -p tor-netdoc --bump minor
cargo set-version -p arti-hyper --bump minor
cargo set-version -p arti-bench --bump minor
cargo set-version -p arti-testing --bump minor
cargo set-version -p tor-config --bump minor
```
In order to mitigate syntax highlighting issues and a rust-analyzer bug
(https://github.com/rust-analyzer/rust-analyzer/issues/10178), rename
files that are included with the `include!` macro to have a `.rs`
extension.
Make sure the included files are outside `src/`, in order to not confuse
humans and automated editing tools that might mistake them for valid
Rust modules.
fixes arti#381
Found these by disabling the nightly dbg macro special case. Now, we
have a mechanism for globally adding suppressions to tests, we can use
that instead.
At the site of modify_hop, we now have a comment explaining the
internal-error issue.
To make the internal error less likely, we lower the modify_hop call
in lib.rs into GuardSet, where it can make sure it's looking at the
same filter as was used to select the guard.
The function name "pick_guard_ext" is not permanent; I'm going to
rename it in the next commit.
Previously, the API said "you need to call this in a loop till it
returns false". We did that in one place, but not another.
With the introduction of filters, forgetting to loop here becomes a
bug: so instead, change the behavior of extend_sample_as_needed so
it handles looping itself.
The fake network we construct for these tests is small in ways that
can sometimes cause weird behavior with guard filters. We fix this
by adjusting the parameters of the guard selection algorithm
accordingly in the tests.
With these new parameters, #491 no longer occurs.
This commit also adds comments to explain why the parameters are set
as they are.
Closes#491.
guard-spec.txt specifies that we have multiple separate samples of
guards that we can use depending on whether the filter is
restrictive or not. Here we implement the rules for switching
between samples.
When we're filtering guards, we have to check whether the filter is
"restrictive": if it forbids most of the guards (by bandwidth), we
keep its guards separated from the main set. If it is
super-restrictive, we also warn.
This functionality is specified in guard-spec.txt.
Since a guard can have a bunch of addresses, and the guard is
permitted if any one of those addresses is permitted, then we might
decide to use a guard with some non-permitted addresses. Thus, we
need to filter those addresses before returning the view of the
guard as a FirstHop.
Some of our existing code optionally takes a netdir from the
caller. When it doesn't give us one, use the netdir from the
installed NetDirProvider.
(Possibly someday we should remove the NetDir arguments
entirely. I'm deferring that because there are only two APIs
affected, and because making this change would force us to rewrite a
pretty large mess of unit tests.)
Previously it was the job of a task in CircMgr to do this; but we're
going to want to give GuardMgr full access to the latest NetDir for
this, and for other code-simplification reasons.
With this change I'm deprecating a couple of functions in
tor-circmgr. It's no longer necessary for us to have an artificial
external way for you to feed new NetDirs to a circmgr. (I could
just remove them, but I want practice deprecating.)
This only affects uses of thread_rng(), and affects them all more or
less indiscriminately. One test does not work with
ARTI_TEST_PRNG=deterministic; the next commit will fix it.
This commit was made by reverting the previous commit, then
re-running the script I used to generate it. In theory there should
be no semantic changes: only changes due to improved formatting from
cargo edit.
I followed the following procedure to make these changes:
* I used maint/changed_crates to find out which crates had changed
since 0.3.0.
* I used grep and maint/list_crates to sort those crates in
topological (dependency) order.
* I looked through semver_status to find which crates were listed as
having semver-relevant changes (new APIs and breaking changes).
* I scanned through the git logs of the crates with no
semver-relevant changes listed to confirm that, indeed, they had
no changes. For those crates, I incremented their patch-level
version _without_ changing the version that other crates depend on.
* I scanned through the git logs of the crates with no
semver-relevant changes listed to confirm that, indeed, they had
no obvious breaking changes.
* I treated all crates that depend on `arti` and/or `arti-client` as
having breaking changes.
* I identified crates that depend on crates that have changed, even
if they have not changed themselves, and identified them as having
a non-breaking change.
* For all of the crates, I used `cargo set-version -p $CRATE --bump
$STATUS` (where `STATUS` is `patch` or `minor`) to update the
versions, and the depended-upon versions.
Test the Deserialize impl of every config struct.
This detects bugs like the one fixed in !502.
The macro now becomes more complex because it needs to take options.
Right now this tt-munching option parser is overkill, but this
leave space for further options in the future.
I have Plans for this macro. In particular:
* I have a wip branch which tests that the Builder can be
deserialised from an empty config (ie, that config reading
of a config with a blank section for this item works).
* I think we should autogenerate $Config::builder(),
and promote that, rather than $ConfigBuilder::default().
This macro could do that.
For reference, the git source for this crate (and the others in its
workspace) currently lives in my personal github account (ijackson).
If this fork turns out to be long-lived and gains features and/or
users, it would be good to move it to a gitlab somewhere.
I have granted Nick crate ownership on the crates.io system.
* Builders additionally derive: Debug, Serialize, Deserialize.
* Validated structs no longer derive: Serialize, Deserialize
and all related attributes deleted.
* As a consequence, all the `#[serde(deny_unknown_fields)]`
are gone. That means that right now unknown fields are totally
ignored. This is good for compatibility but poor for useability.
Doing something better here is arti#417, in progress.
* As a consequence, delete tor_dirmgr::retry::default_parallelism.
(The default value was already duplicated into a builder attr.)
And drop the ad-hoc orport() method. This brings FallbackDir's
orports field in line with our list builder API.
The general semver note in "configuation" seems to cover most of this.
Although these do not appear in the config, it does have a builder.
It seems sensible to get rid of this ad-hoc list manipulation site,
and replace it with our standard list builder API.
define_list_builder_helper requires that the builder element type be
Deserialize. Currently GuardUsageRestriction is a transparent, public
enum, so we aren't really exposing anything.
We could introduce GuardUsageRestrictionBuilder now, but
since it's not in the config and thereofore only in the public API of
the lower crates, we can definitely put that off.
The new API is (roughly) as discussed in
https://gitlab.torproject.org/tpo/core/arti/-/issues/451
This is quite a large commit and it is not convenient to split it up.
It contains the following changes:
* Redo the list builder and accessor macros implemnetation,
including docs and tests.
* Change uses of define_list_config_builder. In each case:
- Move the docs about the default value to the containing field.
- Remove the other docs (which were just recapitulations, and
are now not needed since the ListBuilder is no longer public).
- Rewmove or replace `pub` in the define_list_builder_helper call,
so that the builder is no longer public.
- Change the main macro call site to use define_list_builder_helper.
- Add a call to define_list_builder_accessors.
* Make the module `list_builder` pub so that we have somewhere to
put the overview documentation.
* Consequential changes:
- Change `outer.inner().replace(X)` to `outer.set_inner(X)`
- Consequential changes to imports (`use` statements).
The `[patch]` approach causes the tree not to build when used as a
dependency, unless the `[patch]` is replicated into the depending
project.
Instead, replace our `derive_builer =` dependencies with a reference
to a specific git commit:
perl -i~ -pe 'next unless m/^derive_builder/; s#"(0\.11\.2)"#{ version = "$1", git = "https://github.com/ijackson/rust-derive-builder", rev = "ba0c1a5311bd9f93ddf5f5b8ec2a5f6f03b22fbe" }#' crates/*/Cargo.toml
Note that the commitid has changed. This is because derive_builder is
in fact a workspace of 4 crates. 3 of them are of interest to arti
itself (the 4th exists only for testing). So the same "add git
revision" treatment had to be done to the `derive_builder` and
`derive_builder_macro` crates. Each dependency edge involves a new
commit in the derive_builder workspace, since we can't create a git
commit containing its own commitid. (We want to use commits, rather
than a branch, so that what we are depending on is actually properly
defined, and not subject to the whims of my personal github
namespace.)
There are no actual code changes in derive_builder.
And add an imprecation in define_list_config_builder's doc comment do
do so in future for other invocations of the macro.
Add add the missing full stops.
This is an automated change made with a perl one-liner and verified
with grep -L and grep -l.
Some warnings are introduced with this change; they will be removed
in subsequent commits.
See arti#208 for older discussion on this issue.
Now the network fallbacks configuration wants to Deserialize
a Vec<FallbackDirBuilder>, rather than validated Vec<FallbackDir>.
Methods on FallbackListBuilder are as per
https://gitlab.torproject.org/tpo/core/arti/-/merge_requests/462#note_2797697
mutatis mutandi for the fact that this struct has only fallbacks in it.
This is where the FallbackList type is. We are going to want to
provide a builder too, which ought to impl Default.
This means that the default value for the type must be next to the
type. In any case, it was anomalous that it wasn't.
This commit is pure code motion.
Instead of just having a function that recalculates the latest clock
skew, instead recalculate the clock skew when it may have changed,
and notify other processes via a postage::watch.
Apparently on OSX you are not allowed to construct an Instant that is a
long time before the time when the test is running.
Also, fix the length of a year in this test.
This time, our estimator discards outliers, takes the mean of what's
left, and uses the standard deviation to try to figure out how
seriously to take our report of skew/not-skew.
These estimates are still not actually used.
The old version of this function was error-prone, and in fact had
errors: it was too easy to forget to add non-persistent fields, and
that's exactly what we forgot in a few cases
(`microdescriptor_missing`, `circ_history`, and
`suspicious_behavior_warned`).
The new version of this function consumes both of the incoming
Guards, and constructs every field explicitly so that we can't
forget to list any.
Closes#429.
Previously, we treated successfully building a circuit to a guard as
a "success", and any failure, including a directory cache failure,
as a failure. With this change, guards now have separate
success/failure and retry status for circuit usage and directory
usage.
This change is needed for guard-as-directory retry to have
reasonable behavior. Otherwise, when a guard succeeds at building a
circuit, that clears the directory-is-failing status and makes us
retry the guards to quickly.
Not all of these strictly need to be bumped to 0.2.0; many could go
to 0.1.1 instead. But since everything at the tor-rtcompat and
higher layers has had breaking API changes, it seems not so useful
to distinguish. (It seems unlikely that anybody at this stage is
depending on e.g. tor-protover but not arti-client.)
The FirstHopId type now records an enum that stores whether the hop
is a guard or a fallback. This change addresses concerns about
remembering to check the type or source of an Id before passing it
down to the FallbackState or GuardSet.
Making this change required an API change, so that dirmgr can
report success/failure status without actually knowing whether it's
using a fallback or a guard.
The code here uses a new iterator type, since I couldn't find one of
these on crates.io. I tried writing the code without it, but it was
harder to follow and test.
We do this by creating a new FallbackSet type that includes status
information, and updating the GuardMgr APIs to record success and
failure about it when appropriate. We can use this to mark
FallbackDirs retriable (or not).
With this change, FallbackDir is now stored internally as a Guard in
the GuardMgr crate. That's fine: the FallbackDir type really only
matters for configuration.
We only do this when we fail to get a regular guard (e.g., because
they're all down), and when we have been asked for a guard for a
one-hop directory.
Most of the change in this commit is plumbing to make all of the
types match up.
As before, compilation may still be broken.
We need to extend our notion of "the origin of a guard" to include
"somewhere outside the guard list"; we need the ability to return a
FallbackDir as a Guard; and we need to remember a few more pieces of
information in each pending request.
As before, this commit may break compilation; it will be restored soon.
The guard manager is responsible for handing out the first hops of
tor circuits, keeping track of their successes and failures, and
remembering their states. Given that, it makes sense to store this
information here. It is not yet used; I'll be fixing that in
upcoming commits.
Arguably, this information no longer belongs in the directory
manager: I've added a todo about moving it.
This commit will break compilation on its own in a couple of places;
subsequent commits will fix it up.
This is the logical place for it, I think: the GuardMgr's job is to
pick the first hop for a circuit depending on remembered status for
possible first hops. Making this change will let us streamline the
code that interacts with these objects.
When all guards are down, we would previously mark them all as up,
and retry aggressively. But that's far too aggressive: if there's
something wrong with our ability to connect to guards, it makes us
hammer the network over and over, ignoring all the guard retry
timeouts in practice.
Instead,
* We now allow the `pick_guard()` function to fail without
automatically retrying.
* We give different errors in the cases when all our guards are
down, and when all of the guards selected by our active usage
are down.
* Our "guards are down" error includes the time at which a guard
will next be retriable.
This is part of #407.
C tor used one schedule, and guard-spec specified another. But in
reality we should probably use a randomized schedule to retry
guards, for the reasons explained in the documentation for
RetrySchedule.
I've chosen the minima to be not too far from our previous minima
for primary and non-primary guards.
This is part of #407.
This has the different syntax for builder field attributes than what I
originally proposed in my MR, and which therefore is in the pinned
branch.
My upstream MR for the field attributes feature was morged:
https://github.com/colin-kiegel/rust-derive-builder/issues/239
This has the humantime_serde::option module, which we have upstreamed
and are about to switch to.
The remaining dependency with version = "1" is going to be removed
in a moment.
We are going to want to specify custom attributes on fields of the
builder struct. This feature was missing from derive_builder.
This commitid is the current head of my MR branch
https://github.com/colin-kiegel/rust-derive-builder/pull/237https://github.com/ijackson/rust-derive-builder/tree/builder-field-attrs
Using the commitid prevents surprises if that branch is updated.
We will require this newer version of derive_builder. The version
will need to be bumped again later, assuming the upstream MR is merged
and upstream do a release containing the needed changes.