This function isn't actually needed (it's not the responsibility of
`KeyType` to encode keys).
This commit also rewrites `ArtiNativeKeystore::insert` to use the new
`as_ssh_keypair_data` function instead of `to_ssh_format`.
The `EncodableKey::to_bytes` function didn't make much sense, because
not all keys have a canonical byte representation.
This commit replaces `EncodableKey::to_bytes` with
`EncodableKey::as_ssh_keypair_data`. In the future, `EncodableKey` will
grow functions for encoding keys in other storage formats too.
Closes#965
There are some places we might improve this, maybe testing more data
types and shapes. This patch just makes the minimal changes necessary
to get it working: adds allocation logic to the fuzzer itself,
and adds visibility for the bucket_array::mem interface.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
In response to review feedback, explain that 'seed' here is more
for compatibility and convenience and not central to our goal of
fuzzing the program generator.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
Fuzz testing for HashX. Uses a hook into the pseudorandom number
stream to test the program generator deeply on input that can
be mutated by the fuzzer. Confirms program generation by running
a small number of arbitrary test hashes, so we don't need to
understand the implementation-specific program format to test the
program generator.
We test four implementations in parallel this way, the compiled and
interpreted implementations included in both this crate and c-tor.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
The solve tests are all tunable so that we can balance execution time
with test coverage. A longer solve will test more random programs and it
will test more of the nonce increment function, minor benefits at the
cost of much more CPU.
The starting nonce in solve_effort1k_aa_41_01 was set so that we would
exercise a rollover in bit 7 of the nonce increment before the full
width rollover, but this wasn't a particularly helpful place to test
and certainly not worth the 13+ seconds it takes on my machine.
This patch bumps the starting nonce to a value much closer to the
target, and still including the full-width rollover.
Brings solve_effort1k_aa_41_01 down from 13.2 seconds to 0.5 sec for me.
For ticket #991
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
This test had one large function for all the verifies and one for all
the solves. The solve test was slow enough to be a bottleneck,
documented in ticket #991.
This patch splits the existing tests up in to one labeled function per
solve or verify configuration.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
Propagates this setting from the outer Cargo.toml to the new
benchmark crates, since they no longer get the setting by
being included in the main workspace.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
It might be useful to keep these locked down for benchmark
reproducibility. Currently the hashx and equix crates are
fully separate.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
This is a small batch of improvements for the equix and hashx
benchmarks. The headline feature is that we are now including
the C implementations (slightly modified from tevador's, hosted
as part of c-tor) and using them in apples-to-apples comparisons.
Minor features:
- Benchmarks moved to new nested crates, preventing their
dependencies from spilling into the main workspace build.
- Tests are now grouped
- We also test the performance of memory reuse where possible
- Code cleanup for per-runtime options
These benchmark builds will now automatically pull in the c-tor
git repo and build portions of it with a Rust wrapper. This uses
the 'cc' and 'bindgen' crates, so it requires a C compiler and
libclang on the host system.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
These crates have had trivial changes only: typically,
changes to documentation or to clippy warnings. There's no
good reason to update which version of them other crates depend on,
so we only bump _their_ patchlevels.
```
tor-async-utils
caret
safelog
tor-events
tor-units
tor-rtcompat
tor-rpcbase
tor-llcrypto
tor-protover
tor-bytes
tor-hscrypto
tor-socksproto
tor-cert
tor-cell
tor-consdiff
tor-congestion
arti-rpcserver
arti-testing
arti-bench
arti-config
arti-hyper
```
These crates are at version 0.x.y, so we don't need to distinguish
new-feature changes from other changes:
```
tor-basic-utils
fs-mistrust
tor-error
tor-geoip
tor-checkable
tor-linkspec
tor-netdoc
tor-netdir
tor-persist
tor-ptmgr
tor-hsservice
```
This crate has a breaking change, but only when the semver-breaking
feature `experimental-api` is enabled:
```
tor-config
```
This crate is at version 1.x.y, but has no new public APIs, and
therefore does not need a minor version bump:
```
arti
```
These crates had first-order breaking changes:
```
retry-error
tor-keymgr
tor-proto
tor-hsclient
tor-rtmock
```
Additionally, these broke because they re-exposed RetryError:
```
tor-circmgr
```
Additionally, these broke because they may re-expose something from
tor-proto:
```
arti-client
tor-chanmgr
tor-dirclient
tor-dirmgr
tor-guardmgr
```
Additionally, these broke for other fiddly reasons:
`tor-ptmgr` implements traits from tor-chanmgr, which has a breaking
change above.
`arti-hyper` exposes types from arti-client in its API.
This replaces the 'TODO' marker from earlier commits, using tevador's
copyright and license (LGPL 3.0 only) for the hashx and equix crates.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
Challenges are now built using try_extend_from_slice instead of
iterators. There's no CPU benchmark in this crate yet, but I can confirm
that the resulting code is shorter. With this patch, the entirety of
Challenge:new() is automatically inlined at call sites.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
I originally wrote this in an overcomplicated way, to avoid
frequent initialization of a RegisterWriter array. It turns out
that RegisterWriter can be fairly compact, so this extra level of
indirection isn't necessary or measurably helpful.
This still manages to avoid declaring RegisterWriter as Copy, by
using Default to initialize the array instead of an array constructor.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
In response to review feedback. The byte output is only needed
for unit tests right now, since Equi-X uses u64 output exclusively.
The optimization for shorter output widths can shave tiny amounts of
time off hash benchmarks, but in this case it's more helpful to avoid
introducing APIs that offer parameters with incomplete compile-time
range checking.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
This refactors the random number generator used within HashX's program
generator so that it uses the rand::RngCore trait. The basic SipHash
powered u64 generator now implements RngCore, while a buffer layer
wraps this and provides u8 and u32 values as needed by the generator.
Some of this new RngCore layer is now exposed to the hashx crate's
public API. The intent is to allow external code to test, benchmark, or
fuzz the program generator by supplying its own random number stream.
Benchmarks show a small but confusing performance improvement
associated with this patch. About a 2% improvement in generation.
This could be due to the Rng changes. No change in compiled hash
execution performance. Even though this patch only touches program
generation, benchmarks show a 4% speedup in interpreted execution.
This seems most likely explained by instruction cache effects,
but I'm not sure.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
This splits up bucket_array into two smaller modules, one for the hash
table behavior and one for the MaybeUninit memory management.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
I was hoping most of the program generator would get inlined, so we can
resolve a lot of the edge cases at compile-time. This patch gets us
close to that, adding many inline attrs and rewriting RegisterSet with
explicit unrolling and storage types that are easier for the optimizer
to reason about.
From the disassembly of the program generator, it's now mostly one big
function with a jump table. From callgrind instruction profiles, there
are no longer obvious hotspots in register set scanning loops. It also
looks like we're often keeping per-register schedule information all
loaded into machine registers now.
Keeping the Rng entry points non-inlined for now seems to be slightly
better, by a percent or two.
There's some work left to do in compiled programs, and maybe room for
improvement in the Program representation too. That will be in a future
patch.
Benchmark shows about 20% improvement on my machine,
generate-interp time: [75.440 µs 75.551 µs 75.684 µs]
change: [-24.083% -23.775% -23.483%] (p = 0.00 < 0.05)
Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
5 (5.00%) high mild
6 (6.00%) high severe
generate-x86_64 time: [96.068 µs 96.273 µs 96.540 µs]
change: [-18.699% -18.381% -18.013%] (p = 0.00 < 0.05)
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
4 (4.00%) high mild
6 (6.00%) high severe
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
This adds a new tor-hspow crate with the first layers of support in
place for onion service client puzzles as described in Proposal 327.
The API here is experimental, and it's currently only implementing
the self-contained parts of the client puzzle. So, it can verify and
solve puzzles, but it has no event loop integration or nonce replay
tracking or prioritization code yet. These things seem like they would
eventually live in the same crate.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
This is a new pure Rust implementation of the Equi-X algorithm
designed by tevador for Tor's onion service proof of work puzzle v1.
Equi-X is an asymmetric puzzle algorithm based on Equihash, with
N=60, K=3, the XOR replaced with modular addition, a 16-bit index
space, and HashX as the inner hash function.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
This is a new pure Rust implementation of the HashX algorithm
designed by tevador for Tor's onion service proof of work puzzle v1.
HashX is a lightweight family of randomly generated hash functions.
A seed, via blake2 and siphash, drives a program generation model
which randomly selects opcodes and registers while following some
constraints that avoid timing stalls or insufficient hash mixing.
The execution of these hash funcions can be done using a pure Rust
interpreter, or about 20x faster using a very simple just in time
compiler based on the dynasm assembler crate. This has been
implemented for x86_64 and aarch64.
Signed-off-by: Micah Elizabeth Scott <beth@torproject.org>
I think it's safe to handle `ChanMsg::Create` separately, because
there's nothing for the reactor to do until the first hop of the circuit
is created (so blocking on this _should_ be alright).
This logic from `create_firsthop()` was extracted (copied) from
`Reactor::run_once()`. A future commit will update `Reactor::run_once()`
to use `create_firsthop()`.
rustfmt has grown opinions about how let ... else ... ought to be
formatted. They don't always agree with our previous manual
decisions.
I think our policy is to always insist on rustfmt. When that version
of rustfmt hits stable, our CI will start to fail for everyone.
(Right now this discrepancy just causes trouble for contributors who
are using nightly by default.)
This will enable us to test the parts of `ArtiNativeKeystore::insert`
that _are_ implemented (such as the part where it creates the missing
directories).
cargo doc --locked --document-private-items --workspace --all-features
warning: unclosed HTML tag `CountryCode`
--> crates/tor-geoip/src/lib.rs:90:54
|
90 | /// We store these as NonZeroU8 so that an Option<CountryCode> only has to
| ^^^^^^^^^^^^^
|
= note: `#[warn(rustdoc::invalid_html_tags)]` on by default
The concept of a "key bundle" would introduce a lot of complexity while
providing little to no gain.
Some context:
```
Originally, "key bundles" were meant to be the answer to the question
"which keystore should insert place keys in?":
36606a66dd/crates/tor-keymgr/src/mgr.rs (L60-69)
However, I'm not so sure anymore that "key bundles" are the answer. I
don't think there is any way we can "guess" where a key should go. When
inserting/generating a new key, we should either:
always write to the same, primary key store, OR require the user to be
explicit about which key store the new key should go in (by assigning an
ID to each key store and expecting the user to provide it when
inserting/generating new keys)
I prefer the latter option, because it provides more flexibility, which
we're going to need when implementing the key management CLI (which I
think should allow users to generate keys anywhere they want, e.g. arti
keymgr generate <key type> --keystore hsm ...)
```
For more details, see the discussion on #903.
Closes#903
The caller uses `KeystoreSelector` to specify which keystore to insert
the new key into (only `KeystoreSelector::Id` and
`KeystoreSelector::Default` are supported for `insert`).
The ability to insert keys in a particular keystore will come in handy
when we implement the key management CLI (the CLI will have an option
for specifying the keystore to access/modify).
This error will be returned by `KeyMgr` if the caller tries to access a
keystore that does not exist, or if the requested `KeystoreSelector`
cannot be applied.
This will enable the `KeyMgr` to look up `Keystore`s by ID (which is
a requirement for disambiguating the semantics of `insert`, which
currently tries to "guess" which keystore it should be using).
Hiding the underlying value of `enabled` enables us to give it a
different `auto` value depending on whether the `keymgr` feature is
enabled or not (it defaults to `true` if `keymgr` is enabled, and
`false` otherwise).
The `experimental-api` was only meant to apply to the use of the
unstable `ArtiNativeKeystoreConfig` in the Arti config.
`experimental-api` was _not_ supposed to be used for enabling/disabling
the keystore (that's what the `enabled` flag is for).
This would be a break in higher-layer crates which incorproate this
error but:
1. That's just arti-client which hides it behind the detailed errors
cargo feature
2. I'm hoping cargo-semver-checks would spot it, anyway.
The effect is that everywhere a RetryError is used, the error sources
for the contained errors will be Display'd.
In tor-hsclient we no longer need to explicitly wrap things up in
tor_error::Report.
We don't use OsString now except where it appears in our public API,
or where we get it from std::env.
Moving the `use` statements into the use sites enabled me to see
that I had found all the places I wanted to change.
users is unmaintained. pwd-grp is the crate I have just written to
replace it. In this commit:
Change the cargo dependency and imports.
Replace the cacheing arrangements. users has a built-in cache;
pwd-grp doesn't. Now, instead of cashing individual lookups, we cache
the trusted user and trusted gid calculation results.
This saves on some syscalls, and is also more convenient to write.
(Mocking is still done via the dependency.)
Many systematic consequential changes of details:
* The entrypoint names to the library are different:
pwd-grp uses the names of the corresponding Unix functions.
* pwd-grp's returned structs are transparent, so we don't
call accessors for .uid(), .name(), etc.
* pwd-grp's methods are much more often fallible
(returning io::Result<Option<...>)
* We're using the non-UTF-8 pwd-grp API, which means we must
use turbofish syntax in some places.
* The mocking API is a bit different.
Add some wrapper functions for convenience.
The pwd-grp crate has a richer and more faithful, but not so
convenient, way of creating dummy user/group entries. Also the type
names are all going to change.
Doing this now reduces churn.
The actual underlying operations here *are* fallible.
The `users` crate hides those errors in several cases.
(Failures are very rare (at least unless NIS is involved), so this is
not of much practical import, but it's going to be necessary when we
use the more careful pwd-grp crate.
When we implemented bridges, we added code in 08473872ab to
conditionally mark their directory info as present or not present.
But the we didn't remove the old code to mark them present
unconditionally!
Fixes#638.
- When the `geoip` feature flag of `tor-netdir` is enabled, perform
GeoIP lookups for all relays added to the directory and add the
resulting country code to the `Relay` struct.
- The GeoIP database is provided in a new
`PartialNetDir::new_with_geoip` constructor.
- A new trait was also added to `tor-linkspec`, `HasCountryCode`, to
enable getting this data out from other crates.
Part of onionmasq#47.
Previously, the keystore config consisted of a single field in
`StorageConfig`, which encoded 2 bits of information: whether the
keystore is enabled, and its root directory:
```
[storage]
# use this path, fail if compiled out
# keystore = "/path/to/arti/keystore"
#
# use default path, fail if compiled out
# keystore = true
#
# disable
# keystore = false
```
This commit adds `ArtiNativeKeystoreConfig`, which will replace the
multi-purpose `keystore` field. The new config will look like this:
```
#[storage.keystore]
# Whether the keystore is enabled.
#
# If the `keymgr` feature is enabled and this option is:
# * set to false, we will ignore the configured keystore path.
# * set to "auto", the configured keystore, or the default keystore, if the
# keystore path is not specified, will be used
# * set to true, the configured keystore, or the default keystore, if the
# keystore path is not specified, will be used
#
# If the `keymgr` feature is disabled and this option is:
# * set to false, we will ignore the configured keystore path.
# * set to "auto", we will ignore the configured keystore path.
#
# Setting this option to true when the `keymgr` feature is disabled is a
# configuration error.
#enabled = "auto"
# The root directory of the arti keystore
#path = "${ARTI_LOCAL_DATA}/keystore"
```
While `ArtiNativeKeystoreConfig` currently only has 2 fields, `enabled`
and `path`, future versions of the keystore might require additional
config options.
There are a number of places where we generate random Durations
in a range which starts at zero.
These call sites currently (i) have to write out Duration::ZERO
or equivalent, and (ii) would have to use gen_range_checked and expect
the result, even though it can be statically proven to be OK.
To make this slightly smoother, provide `GenRangeInfallible` and
`gen_range_infallible`.
Ideally we would be allowed to use vanilla gen_range() here, but there
doesn't seem to be a way to allow a specific clippy-forbidden method
using #[allow] and we probably don't want to make a blanket allow.
This launders the closure so that clippy's
clippy::redundant_closure_call can't see it.
We can't have a local #[allow] because it would be on an expression,
which isn't allowed on stable.
This avoids having to use more clumsy idioms at call sites.
Originally they didn't check err.kind(), since err.kind() can never
increase their severity. We lost that behavior with !1386, and we
became dependent on it with arti!1383. Since they both merged at
the same time, CI broke.
This patch restores their original behavior.
This abolishes some quintuplication.
The output is identical except that:
* The syntax display in the rustdoc output for the resulting macros
seems to have somewhat less whitepsace.
* The whimsical error messages in the examples are all identical.
Ah well.
I identified the cases to replace by searching for the string
`.report()`. There are a few that I didn't change:
* A couple of cases that used anyhow::Error,
* One case that reported two Errors.
* Two cases in `tor_hsclient::err` that just did
`error!("Bug: {}")`.
I have also not audited the cases in `tor-hsclient` where we're using
`tor_error::Report` manually.
Nonetheless, closes#949.
Many of these call sites would panic if, somehow, the upper bound was
zero. In most cases it is very complicated to see if whether this
could happen.
However, there is a better answer:
Durations are (conceptually) dense, so picking the closed set (which
includes its boundary) rather than the open one (which doesn't) will
make little practical difference.
So change four call sites to use `..=` instead of just `..`.
Since we're going to be using `Option<CountryCode>` all over, let's
save the extra byte.
Sadly this required std::mem::transmute(), which is unsafe, so maybe
we should think twice.
The main contribution here is a set of convenience macros for
logging error `Report`s. Notably, this macros always logs
`Internal` and `BadAspiUsage` errors at `WARN`, unless they
are already at `ERROR` or more.
This is a little tricky because `tracing::event!()` requires
its Level argument to be a constant.
Drain a number of events, not just one. The stream might yield many
events, as explained in this new comment.
This fails every time with MockExecutor::try_test_with_various().
I think it might fail with the tokio exeuctor too, but evidently not
with high probability or we would have noticed.
Nothing in our tree actually *uses* the UDP in tests.
We want a mock UDP provider that isn't part of a real runtime, so that
we can make a totally-mock runtime for properly controlled testing.
It seems best to make this part of MockNetProvider rather than a
separate type.
This deduplicates some trait delegation.
We want this now because we're about to introduce a third mock
runtime, so this would become triplication otherwise.
I was unable to assure myself that tracing_subscriber could
withstand a panicking Timer, so instead I'm trying to make sure
our own LogTimer can't panic.
The original version of our panic handler would allocate a string
for the panic `Location`. But if we're panicking we'd like to keep
allocations to a minimum: so instead format the `Location`
conditionally.
This also drops a useless `std::borrow::Cow`. (The possibility of a
"Don't have a `Cow`" commit message was considered and rejected.)
These crates were started, but they aren't integrated into
Arti (yet?). As such it doesn't make sense to keep publishing
updated versions of them every time we tweak our standard
lints.
Was UninstallHandler. We are going to talk more about conversations
and less about handlers (although, the fact of there being a handler
will still be visible).
Done with the commands below.
The following crates have had various changes, and should get a
patchlevel bump. Since they are pre-1.0, we do not need to
distinguish new APIs from other changes.
```
cargo set-version --bump patch -p arti-client
cargo set-version --bump patch -p safelog
cargo set-version --bump patch -p tor-bytes
cargo set-version --bump patch -p tor-cert
cargo set-version --bump patch -p tor-circmgr
cargo set-version --bump patch -p tor-config
cargo set-version --bump patch -p tor-consdiff
cargo set-version --bump patch -p tor-dirclient
cargo set-version --bump patch -p tor-dirmgr
cargo set-version --bump patch -p tor-error
cargo set-version --bump patch -p tor-hsservice
cargo set-version --bump patch -p tor-linkspec
cargo set-version --bump patch -p tor-llcrypto
cargo set-version --bump patch -p tor-netdir
cargo set-version --bump patch -p tor-netdoc
cargo set-version --bump patch -p tor-proto
cargo set-version --bump patch -p tor-rpcbase
cargo set-version --bump patch -p tor-socksproto
```
This crate has new features, but no new non-experimental Rust APIs.
So even though it is post-1.0, it gets a patchlevel bump.
```
cargo set-version --bump patch -p arti
```
These crates have had nonfunctional changes only, mostly due to !1271,
or ac90cb7500, or documentation changes: crates that depend on
them do not require a version bump.
```
arti-bench
arti-config
arti-hyper
arti-testing
caret
fs-mistrust
retry-error
tor-async-utils
tor-basic-utils
tor-chanmgr
tor-checkable
tor-congestion
tor-consdiff
tor-events
tor-guardmgr
tor-persist
tor-protover
tor-ptmgr
tor-rtcompat
tor-rtmock
tor-units
```
Gabriela Moldovan
Micah Elizabeth Scott
Nick Mathewson
Ian Jackson
Saksham Mittal
sw1tch
Ian Jackson
Dimitris Apostolou
juga
Alexander Færøy
eta
Kunal Mehta
trinity-1686a