Commit Graph

152 Commits

Author SHA1 Message Date
Ian Jackson d65c0023c5 CI: Break out some fastish Rust checks into new "build" step 2022-06-08 19:32:59 +01:00
Ian Jackson 9a8d75b844 CI: Break out some fast checks using debian image 2022-06-08 19:30:16 +01:00
Ian Jackson 84d5845eb6 maint/check_toposort: Run in CI 2022-06-08 18:35:56 +01:00
Ian Jackson 73c65d4c1a clippy: Allow dbg in tests *again*
*sigh*
2022-06-08 17:08:40 +01:00
Ian Jackson 6f96736278 Merge branch 'check-license-fixes' into 'main'
Update check_licenses and add it to CI.

Closes #462

See merge request tpo/core/arti!559
2022-06-02 13:59:35 +00:00
Nick Mathewson c8997b652a Add check_licenses to CI.
The artifact-caching logic is taken from other examples in the
.gitlab-ci.yml.
2022-05-31 14:01:21 -04:00
Ian Jackson 3c3cdbcf9a maint/add_warning: Run with --check in CI 2022-05-31 18:54:08 +01:00
Orhun Parmaksız bfd41ddb5f
Lexically sort Cargo.toml dependencies
Utilize cargo-sort: https://github.com/DevinR528/cargo-sort

Signed-off-by: Orhun Parmaksız <orhunparmaksiz@gmail.com>
2022-05-28 20:05:51 +03:00
Nick Mathewson 68b0419752 Merge branch 'standardise-shell' into 'main'
Standardise shell

Closes #425

See merge request tpo/core/arti!533
2022-05-26 12:53:14 +00:00
trinity-1686a 8eb843f2af make almost all scripts use bash and set euo pipefail 2022-05-25 22:25:58 +02:00
Alex Xu (Hello71) 4797c120d0 ci: set umask instead of ARTI_FS_DISABLE_PERMISSION_CHECKS 2022-05-25 10:27:45 -04:00
eta b137d64e18 Update Gitlab CI config for Gitlab 14.10
see https://gitlab.com/gitlab-org/gitlab/-/issues/344533
2022-05-23 10:14:09 +00:00
Nick Mathewson 78e86e6b81 Disable fs-mistrust in coverage CI.
The CI runners like to run as root with umask 000, which our code
rightly freaks out about.
2022-05-10 08:14:13 -04:00
Nick Mathewson 4262e9d0ec Merge branch 'use-fs-mistrust' 2022-05-09 15:27:25 -04:00
trinity-1686a 98de5cff3c only deploy pages on main 2022-05-05 22:21:09 +02:00
trinity-1686a 9ed5983c76 add schedulable job to generate aggregated coverage 2022-05-05 21:05:42 +02:00
Nick Mathewson b9c54af707 gitlab-ci: Disable fs mistrust in integration test
The environment where the test runner does its thing seems to want
to run as root with umask 000, which naturally makes the fs
permissions checks freak out.
2022-05-05 09:47:13 -04:00
Ian Jackson 875f3e4352 CI: Check that the lockfile is up to date.
For at least one job, run the first cargo run with --locked.  This
will fail if the lockfile needs updating.

I have verified that this correctly detects this situation:
  https://gitlab.torproject.org/Diziet/arti/-/pipelines/37692
failed.  Now I have rebased this branch onto main to get the fix to
Cargo.lock.
2022-05-04 14:31:20 +01:00
Nick Mathewson 6f3f351140 Increase our MSRV to 1.56.
Our support policy says that we can update to any Rust released at
least 6 months ago; 1.56 came out on 21 October 2021.

This doesn't yet change any code: it just increases the version
we say we need in our README, and the version we test against in
CI.

Our main justification for this change is to be able to upgrade to
newer versions of our dependencies, including `async_executors` >=
0.5, `aes` >= 0.8, and `cipher` >= 0.4.
2022-04-25 13:03:31 -04:00
trinity-1686a c113c607ac use rustc 1.60 (stable) for coverage 2022-04-07 23:04:10 +02:00
trinity-1686a b7daa9ff12 add integration test and fill semver_status 2022-03-14 21:19:19 +01:00
Nick Mathewson dcd74e7a64 tests/chutney: Merge chutney-launching scripts
Formerly, the "launch a chutney network" code was duplicated in
"setup" and "arti-bench", since "setup" always launched an arti
proxy, while "arti-bench" didn't want that functionality.

Now the "setup" script launches an arti proxy conditionally,
depending on whether the "proxy" argument is given.
2022-03-10 12:29:28 -05:00
Nick Mathewson e15c47f762 Reproducible build: Bump to Rust 1.59 and Alpine 3.15
The Rust upgrade is necessary since our Cargo.lock file now requires
`ed25519` 1.4.0, which requires edition2021, which requires Rust
1.56 or later.

The Alpine upgrade is opportunistic: we might as well.

I've also added comments to remind us to keep the .gitlab-ci.yml
and docker_reproducible_build files in sync, since my first version
of this commit messed that up.

Closes #376.
2022-03-04 10:09:31 -05:00
trinity-1686a ee4ea74698 attempt at making CI faster
Move rust-nightly to stage test so it runs in parallel with coverage,
which are the two longest jobs, and currently run in sequence.
Don't document dependancies, should make the step about 50% faster
Use about 12M of cache to not recompile grcov each time
Don't compile grcov with coverage, we don't need it, it's probably
slower both to compile and execute
2022-02-25 18:08:14 +01:00
Ian Jackson 9aefbf4227 rename *.sh scripts to bare name
As per #309

Update all references.  There is one remaining hit for
  git-grep '\.sh\b'
namely "build.sh" in maint/reproducible_build, which I think is
the build script for osxcross - ie, an external project.
2022-02-18 11:46:06 +00:00
Nick Mathewson e7a38ccb55 Temporarily allow failures in the coverage CI.
Right now it's failing on nightly through no fault of our own.
2022-02-09 08:29:00 -05:00
eta 4413645c1b Merge branch 'eta/arti-bench-ci' into 'main'
arti-bench: run the benchmarks in CI, and keep the results

See merge request tpo/core/arti!283
2022-02-03 20:35:27 +00:00
eta 05046626f3 arti-bench: run the benchmarks in CI, and keep the results
This adds `arti-bench` to the `integration` job in the CI pipelines, and
keeps around the JSON benchmark output for later comparison.
2022-02-03 18:31:32 +00:00
Daniel Schischkin 4509f8b9d4 Remove file ending of shellcheck_all and downgrade_dependencies script 2022-02-03 18:14:57 +00:00
Nick Mathewson 73bb6f342e Merge branch 'add-final-newline-force' into 'main'
Add editorconfig to force some rules (Final Newline)

See merge request tpo/core/arti!289
2022-02-03 15:27:13 +00:00
Daniel Schischkin 3a7412f4cf Add editorconfig to force some rules (Final Newline) 2022-02-03 15:27:13 +00:00
Ian Jackson 7be3bf6339 Temporarily disable some clippy lints on nightly 2022-02-02 21:57:30 +00:00
eta a71ab72bf2 Merge branch 'ci-rustup-show' into 'main'
CI: Add "rustup show" to show compiler versions in logs

See merge request tpo/core/arti!276
2022-01-31 18:33:22 +00:00
Nick Mathewson 81a49fb6ec Merge branch 'dedup-cobertura' into 'main'
add postprocessing script to deduplicate cobertura reports

See merge request tpo/core/arti!272
2022-01-31 17:19:01 +00:00
Ian Jackson ac8e28b6c7 CI: Add "rustup show" to show compiler versions in logs
Trying to debug the nightly failure here
  https://gitlab.torproject.org/Diziet/arti/-/pipelines/25417
and I can't seem to repro it.
2022-01-31 13:58:46 +00:00
Daniel Schischkin 1774de7e47 Remove redundancy/code duplication and switch to shellcheck_all.sh script 2022-01-29 00:09:36 +01:00
trinity-1686a daa1eba969 add postprocessing script to deduplicate cobertura reports 2022-01-28 20:52:50 +01:00
Daniel Schischkin 1d7fcd6d48 Undo remove document-private-items arg 2022-01-28 17:44:32 +01:00
Daniel Schischkin 01b3bc85c9 Remove argument document-private-items from nightly build 2022-01-28 17:28:25 +01:00
Daniel Schischkin 67ac91a809 Add missing file ending for downgrade_dependencies in gitlab ci 2022-01-28 15:26:14 +01:00
Daniel Schischkin 6fe0543645 Add shellcheck step into build stage 2022-01-28 15:21:28 +01:00
arturomf94 72a08cc295 Improve `coverage` job in CI 2022-01-28 00:28:06 +00:00
eta 2f39dbd587 Merge branch 'rtcompat-refactor' into 'main'
Refactor the tor-rtcompat API.

Closes #301 and #300

See merge request tpo/core/arti!263
2022-01-27 15:53:35 +00:00
eta 785ad605d7 Merge branch 'coverage-tracking' into 'main'
Add test coverage tracking with CI/CD

Closes #250

See merge request tpo/core/arti!221
2022-01-27 12:56:22 +00:00
Arturo Marquez 899691e5c7
Install bs4 in coverage job 2022-01-26 23:48:11 -06:00
Arturo Marquez b3ba32824e
Use `maint/with_coverage.sh` in coverage job 2022-01-26 23:13:15 -06:00
Nick Mathewson 474a46999d CI: Fix async-std task
Previously it didn't enable any TLS provider, since we made
native-tls optional a few commits ago.  Now it enables rustls,
so that rustls also gets a quick check along with async-std.

I've also switched this test to use "cargo clippy" in place of
"cargo test" because it's a strict superset.
2022-01-26 15:12:39 -05:00
Nick Mathewson c84c547bf4 Move our "minimal versions" code to a script.
Previously this code was in .gitlab-ci.yml, but for the purposes of
testing my fix for #275, it turned out to be much better to have it
in a script of its own.

Also, we now need to update our "quote" second-order dependency,
since otherwise our minimal first-order dependencies won't build
correctly.
2022-01-07 19:10:07 -05:00
Arturo Marquez 3633a78715
Remove lcov from coverage job 2022-01-07 00:41:04 -06:00
Arturo Marquez 4ddddceeaa
Change output type to `cobertura` 2022-01-07 00:07:03 -06:00
Arturo Marquez c86be6cdd6
Use python3.9 in coverage job 2022-01-06 22:13:15 -06:00
Arturo Marquez 86e829ec1b
Add pip3 show to coverage job 2022-01-06 21:48:09 -06:00
Arturo Marquez da3b937c82
Add pip3 to coverage job 2022-01-06 21:46:53 -06:00
Arturo Marquez 6aea21abeb
Remove build in coverage job 2022-01-06 21:26:15 -06:00
Arturo Marquez 2fc67c0145
Use stable image and download nightly in coverage
Following suggestion here:
`https://gitlab.torproject.org/tpo/core/arti/-/issues/250#note_2764010`
2022-01-06 21:18:09 -06:00
Arturo Marquez b46a562060
Add cargo build to coverage job 2022-01-06 20:34:16 -06:00
Arturo Marquez 34e3438d3b
Change target 2022-01-06 19:57:34 -06:00
Arturo Marquez 3c853126de
Add `coverage` job
Following instructions at:

`https://www.collabora.com/news-and-blog/blog/2021/03/24/rust-integrating-llvm-source-base-code-coverage-with-gitlab/`
2022-01-06 14:05:17 -06:00
Daniel Eades 88aba9e007 don't deny 'println' in examples in CI 2021-12-09 15:42:36 +00:00
Daniel Eades fe788aad5e update CI to test all targets 2021-12-09 14:30:41 +00:00
Trinity Pointard 7dd1a20d42 update rusqlite and revert minimal version change 2021-12-09 11:51:39 +01:00
Nick Mathewson 854b854764 Temporarily change minimal-versions CI to use Rust 1.56
Because of arti#257, compatibility with earlier versions of rust are
broken: we hope this is temporary, but fixing it will probably
require a new version of rusqlite.
2021-12-08 12:07:46 -05:00
Trinity Pointard 6615afda8c normalize --target to better reuse build artifacts
testing was done on a different target than building, so everything was
compiled twice (actually, it's the same target, but rust doesn't realize
that)
makes build+test about 20% faster
2021-11-12 16:30:37 +01:00
Trinity Pointard 026cac0c60 try to cache cargo-audit 2021-11-12 15:47:27 +01:00
Nick Mathewson 753cbc9626 In rust-nightly CI, forbid debugging prints.
This patch makes the rust-nightly CI task fail if it detects any
dbg!(), println!(), or eprintln!() calls in production code.

Because of clippy limitations, it may also gripe about calls to
these macros in our tests.  The preferred workarounds are to either
instead.  Both are acceptable.

We're doing this check in CI rather than unconditionally with clippy
directives, since we often want to have these calls in our code
temporarily while we're developing.  Some day we might want this
test to go into a pre-push hook.

This patch also adds #![allow()] directives for println!() and
eprintln!() in the arti crate.  Since that one isn't a library, it's
okay for it to speak to stdout/stderr.

Closes #218.
2021-11-04 11:13:29 -04:00
Nick Mathewson e9cbf8c54f minimal-versions: switch to 1.53. 2021-10-31 12:14:41 -04:00
Trinity Pointard 1fd720a926 add ci check for deps minimal versions and msrv 2021-10-31 09:54:52 +01:00
Trinity Pointard 0438a3da93 move integration tests to scripts 2021-10-28 11:28:22 -04:00
eta 4fa0122dde Improve and future-proof the `arti` CLI
This switches out `arti`'s argument-parsing library with `clap`, which
is a lot more featureful (and very widely used within the Rust
ecosystem). We also now use a lot of `clap`'s features to improve the
CLI experience:

- The CLI now expects a subcommand (currently, either "help", or "proxy"
  for the existing SOCKS proxy behaviour). This should let us add
  additional non-SOCKS-proxy features to arti in future.
- `clap` supports default values determined at runtime, so the way the
  default config file is loaded was changed: now, we determine the
  OS-specific path for said file before invoking `clap`, so the help
  command can show it properly.
  - The behaviour of `tor_config` was also changed; now, one simply
    specifies a list of configuration files to load, together with
    whether they're required.
  - That function also way overused generics; this has been fixed.
- Instead of using the ARTI_LOG environment variable to configure
  logging, one now uses the `-l, --log-level` CLI option.
  (The intent is for this option to be more discoverable by users.)
- The `proxy` subcommand allows the user to override the SOCKS port used
  on the CLI without editing the config file.
2021-10-27 19:11:48 +01:00
Nick Mathewson 7c7a388e28 Rename tor_client/arti_tor_client to arti_client.
Solves a name conflict with the existing tor_client create.

Closes #130.
2021-10-21 14:22:11 -04:00
Nick Mathewson 290fe83671 Move "cargo audit" invocation into a script. 2021-10-17 10:41:52 -04:00
Trinity Pointard 9c275a04ee update ctrlc and der
ctrlc had rustsec issues, der was yanked
2021-10-06 23:55:08 +02:00
Nick Mathewson 0c0a057bef In CI, *temporarily* ignore RUSTSEC-2021-0119 in nix crate.
The affected API (getgrouplist) isn't one that we use, directly or
transitively.  We can't just upgrade immediately, since the ctrlc
crate is currently stuck on the older version of nix.
2021-10-01 09:24:38 -04:00
Trinity Pointard e05a2ae51d add basic integration test 2021-09-27 17:35:32 +02:00
Trinity Pointard 0493c2b830 try use cache and make MacOS bin available as artifact 2021-09-20 17:48:13 +02:00
Trinity Pointard 78b6a36671 add reproducible builds for MacOS 2021-09-20 17:48:12 +02:00
Nick Mathewson 8bf394b0d5 Move version-controlled Cargo.lock to /.
Previously I'd hoped to have it only apply to reproducible builds,
but the work of keeping it up-to-date and checking it for drift
seems to have been tripping us up.  Let's try doing it the other way
for now, and we'll see how that goes.
2021-09-20 11:37:11 -04:00
Trinity Pointard 193f518b6b CI: make fail when new deps not in misc/Cargo.lock
Currently adding new deps can cause builds to not be reproducibles
because that new dependancies don't have their versions frozen in
misc/Cargo.lock.
This make so CI reject commits in that situation, with a message telling
what to do to fix the issue.
2021-09-12 10:20:53 +02:00
Trinity Pointard e69d2d45bc run CI only on amd64 runners 2021-09-08 22:15:20 +02:00
Daniel Eades c2c6a04309 run clippy on test modules 2021-09-08 17:28:40 +02:00
Trinity Pointard 9db9c0c90f use a TPA runner so shm is big enough 2021-08-31 18:28:53 +02:00
Trinity Pointard 3df2fd7a54 build repro only on schedule
build takes about 23 minutes, it's quiet long for something that is
usefull mostly on releases
2021-08-29 16:36:28 +02:00
Trinity Pointard 33474537ab make reproducible Windows build
Linux hash   : c024778da162cda77869c46d162729d012a7758bee973bc83fa78c964d080903
Windows hash : e97e171740b889b7e3722b79c3763177174baac7fa34a5c3615c999298e1bc48
2021-08-29 13:48:33 +02:00
Nick Mathewson 25d058e074 Fix for CI script.
It looks like there was a directory here I didn't update when I
moved all the crates into crates/
2021-08-27 10:18:20 -04:00
Trinity Pointard eb0522fe63 fix segfault on static bin
see https://github.com/rusqlite/rusqlite/issues/914

sha256: 684ebc4b8c270fc63beba185f6c54ceeb98734f13aa7aeca9b64acb33432a21c
2021-08-25 20:37:58 +02:00
Trinity Pointard 540302851e make arti availlable as build artifact
and make builds reproducible
sha256sum: f141c54929a43a31b9ed6b529f6f863aace87f7406818b2f8ffe2b7a5e2803fb
2021-08-25 18:16:45 +02:00
Nick Mathewson 9f61a392be Use cargo audit with "-D warnings". 2021-07-21 09:33:32 -04:00
Trinity Pointard c6fef98a83 attempt to add cargo audit to CI 2021-07-20 23:39:23 +02:00
Nick Mathewson 5a3e735e4b Try some tweaks to .gitlab-ci.yml
The "--all" flag is deprecated in favor of "--workspace", which
isn't actually needed here.

"--all-features" catches a few things that we want to make sure
we're catching.

Let's try out cargo doc and make sure that passes too.
2021-05-25 16:18:00 -04:00
Nick Mathewson 0cfff7e0e4 CI target to build with async-std. Running tests is harder, alas 2021-03-02 12:21:38 -05:00
Nick Mathewson b33bc268d5 My rustfmt is up-to-date with the gitlab ci one again 2021-03-01 07:42:45 -05:00
Nick Mathewson 66c901265d Disable our cargo fmt test on CI for now. 2021-02-26 09:59:23 -05:00
Nick Mathewson 8301ae699b gitlab-ci.yml -- on rust-latest, turn all clippy warnings into errors. 2020-12-07 16:18:21 -05:00
Nick Mathewson 6f286ba7c2 fixup! fixup! fixup! Try adding "test --all" and "fmt" and "clippy". 2020-12-07 14:04:05 -05:00
Nick Mathewson b5f17167c2 fixup! fixup! Try adding "test --all" and "fmt" and "clippy". 2020-12-07 12:51:03 -05:00
Nick Mathewson 09849c64f8 fixup! Try adding "test --all" and "fmt" and "clippy". 2020-12-07 12:41:28 -05:00
Nick Mathewson fd74f6c13f Try adding "test --all" and "fmt" and "clippy". 2020-12-07 12:16:00 -05:00
Alexander Færøy 1923c7820f Let's try explicit target? 2020-12-07 08:35:02 -05:00
Alexander Færøy dee4d7bf72 No need to install Husky hooks on CI. 2020-12-07 08:35:02 -05:00
Alexander Færøy 6fa8f2c0f5 No need to install libsqlite3-dev on CI servers. 2020-12-06 16:01:06 +00:00