With the work on #25500 (reducing CPU client usage), the HS service main loop
callback is enabled as soon as the HS service map changes which happens when
registering a new service.
Unfortunately, for an ephemeral service, we were building the onion address
*after* the registration leading to the "service->onion_address` to be an
empty string.
This broke the "HS_DESC CREATED" event which had no onion address in it. And
also, we were logging an empty onion address for that service.
Fixes#25939
Signed-off-by: David Goulet <dgoulet@torproject.org>
* REFACTORS `UnvalidatedProtoEntry::from_str` to place the bulk of the
splitting/parsing logic in to a new
`UnvalidatedProtoEntry::parse_protocol_and_version_str()` method (so that
both `from_str()` and `from_str_any_len()` can call it.)
* ADD a new `UnvalidatedProtoEntry::from_str_any_len()` method in order to
maintain compatibility with consensus methods older than 29.
* ADD a limit on the number of characters in a protocol name.
* FIXES part of #25517: https://bugs.torproject.org/25517
In protover.c, the `expand_protocol_list()` function expands a `smartlist_t` of
`proto_entry_t`s to their protocol name concatenated with each version number.
For example, given a `proto_entry_t` like so:
proto_entry_t *proto = tor_malloc(sizeof(proto_entry_t));
proto_range_t *range = tor_malloc_zero(sizeof(proto_range_t));
proto->name = tor_strdup("DoSaaaaaaaaaaaaaaaaaaaaaa[19KB]aaa");
proto->ranges = smartlist_new();
range->low = 1;
range->high = 65536;
smartlist_add(proto->ranges, range);
(Where `[19KB]` is roughly 19KB of `"a"` bytes.) This would expand in
`expand_protocol_list()` to a `smartlist_t` containing 65536 copies of the
string, e.g.:
"DoSaaaaaaaaaaaaaaaaaaaaaa[19KB]aaa=1"
"DoSaaaaaaaaaaaaaaaaaaaaaa[19KB]aaa=2"
[…]
"DoSaaaaaaaaaaaaaaaaaaaaaa[19KB]aaa=65535"
Thus constituting a potential resource exhaustion attack.
The Rust implementation is not subject to this attack, because it instead
expands the above string into a `HashMap<String, HashSet<u32>` prior to #24031,
and a `HashMap<UnvalidatedProtocol, ProtoSet>` after). Neither Rust version is
subject to this attack, because it only stores the `String` once per protocol.
(Although a related, but apparently of too minor impact to be usable, DoS bug
has been fixed in #24031. [0])
[0]: https://bugs.torproject.org/24031
* ADDS hard limit on protocol name lengths in protover.c and checks in
parse_single_entry() and expand_protocol_list().
* ADDS tests to ensure the bug is caught.
* FIXES#25517: https://bugs.torproject.org/25517
Apparently, even though I had tested on OpenSSL 1.1.1 with
no-deprecated, OpenSSL 1.1.0 is different enough that I should have
tested with that as well.
Fixes bug 26156; bugfix on 0.3.4.1-alpha where we first declared
support for this configuration.
The new gcov sometimes emits *s if there is a line containing
multiple basic blocks, and some are not executed. (The gcov
documentation says something weird about this point, so I'm trying
to interpret it into the compilerese that I'm familiar with.)
That's bug 26101.
Also, when we're looking for unique variations in our coverage, we
would like cov-diff to suppress timestamps on the diffs. That's bug
26102.
Both of these are bugfixes on 0.2.5.1-alpha when the cov-diff script
was introduced.
My apologies for the perl.
i don't know if whitespace is ok to have before preprocessing
directives on all platforms, but anyway we almost never have it,
so now things are more uniform.
Before this commit, the control events were never triggered. It was introduced
with commit 0c19ce7bde.
Fixes#26082
Signed-off-by: David Goulet <dgoulet@torproject.org>
We alloc/free X.509 structures in three ways:
1) X509 structure allocated with X509_new() and X509_free()
2) Fake X509 structure allocated with fake_x509_malloc() and fake_x509_free()
May contain valid pointers inside.
3) Empty X509 structure shell allocated with tor_malloc_zero() and
freed with tor_free()
There are three cases where this can happen: changes in our
controller events, changes in our DisableNetwork setting, and
changes in our hibernation state.
Closes ticket 26063.
Two new values in each direction. DELIVERED counts valid end-to-end circuit
data that is accepted by our end and OVERHEAD counts the slack unused data in
each of the relay command cells for those accepted cells.
Control port changes are in the next commit.
We still do this time update here, since we do it from all
callbacks, but it is no longer a reason to keep the once-per-second
callback enabled.
Closes ticket 26009.
Since we're going to be disabling the second-elapsed callback, we're
going to sometimes have long periods when no events file, and so the
current second is not updated. Handle that by having a better means
to detect "clock jumps" as opposed to "being idle for a while".
Tolerate far more of the latter.
Part of #26009.
The options_any_client_port_set() returns true if the ControlPort is set which
is wrong because we can have that port open but still not behave as a tor
client (like many relays for instance).
Fixes#26062
Signed-off-by: David Goulet <dgoulet@torproject.org>
This functions is now used outside of networkstatus.c and makes more sense to
be in config.c.
It is also renamed to options_any_client_port_set() for the config.c
namespace.
No code behavior change.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Now it has a function that can tell the rest of Tor whether any
once-a-second controller item should fire, and a function to fire
all the once-a-second events.
Previously the coverage on this function was mostly accidental,
coming as it did from test_entryconn.c. These new tests use mocking
to ensure that we actually hit the different failure and retry cases
of addressmap_get_virtual_address(), and make our test coverage a
bit more deterministic.
Closes ticket 25993.
Previously, an authority with a clock more than 60 seconds ahead could
cause a client with a correct clock to warn that the client's clock
was behind. Now the clocks of a majority of directory authorities
have to be ahead of the client before this warning will occur.
Relax the early-consensus check so that a client's clock must be 60
seconds behind the earliest time that a given sufficiently-signed
consensus could possibly be available.
Add a new unit test that calls warn_early_consensus() directly.
Fixes bug 25756; bugfix on 0.2.2.25-alpha.
construct_consensus() in test_routerlist.c created votes using a
timestamp from time(). Tests that called construct_consensus() might
have nondeterministic results if they rely on time() not changing too
much on two successive calls.
Neither existing of the two existing tests that calls
construct_consensus is likely to have a failure due to this problem.
Code movement for the refactoring for ticket 24658 didn't copy the
inclusion of sys/random.h, which is needed to get a prototype for
getentropy() on macOS 10.12 Sierra. It also didn't copy the inclusion
of sys/syscall.h, which might prevent the getrandom() syscall from
being properly detected. Move these inclusions. Bug not in any
released Tor.
Our previous algorithm had a nonzero probability of picking no
events to cancel, which is of course incorrect. The new code uses
Vitter's good old reservoir sampling "algorithm R" from 1985.
Fixes bug 26008; bugfix on 0.2.6.3-alpha.
This functionality was covered only accidentally by our voting-test
code, and as such wasn't actually tested at all. The tests that
called it made its coverage nondeterministic, depending on what time
of day you ran the tests.
Closes ticket 26014.
This is needed for libressl-2.6.4 compatibility, which we broke when
we merged a15b2c57e1 to fix bug 19981. Fixes bug 26005; bug
not in any released Tor.
We cleared this value in second_elapsed_callback. But what were we
using it for? For detecting if Libevent returned EINVAL too often!
We already have a way to detect too-frequent events, and that's with
a ratelim_t. Refactor the code to use that instead. Closes ticket
26016.
Apart from adding the newly created Module.md, this commit also adds all the
missing files we have in HACKING/ to the EXTRA_DIST.
Signed-off-by: David Goulet <dgoulet@torproject.org>
From Neel's latest patch on optimizing the hs_circ_service_get_intro_circ()
digest calculation, remove an extra white-space and clarify a comment of the
legacy key digest to inform when to use it.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This function must return false if the module is not compiled in. In order to
do that, we move the authdir_mode_v3() function out of router.c and into the
dirauth module new header file named mode.h.
It is always returning false if we don't have the module.
Closes#25990
Signed-off-by: David Goulet <dgoulet@torproject.org>
This test was using the current time to pick the time period number,
and a randomly generated hs key. Therefore, it sometimes picked an
index that would wrap around the example dht, and sometimes would
not.
The fix here is just to fix the time period and the public key.
Fixes bug 25997; bugfix on 0.3.2.1-alpha.
This test, in test_client_pick_intro(), will have different coverage
depending on whether it selects a good intro point the first time or
whether it has to try a few times. Since it produces the shorter
coverage with P=1/4, repeat this test 64 times so that it only
provides reduced coverage with P=1/2^128. The performance cost is
negligible.
Closes ticket 25996. This test was introduced in 0.3.2.1-alpha.
I'd prefer not to do this for randomized tests, but as things stand
with this test, it produces nondeterministic test coverage.
Closes ticket 25995; bugfix on 0.2.2.2-alpha when this test was
introduced.
This change should make it impossible for the monotonic time to roll
over from one EWMA tick to the next during this test, and make it so
that this test never invokes scale_active_circuits() (which it
doesn't test).
(Earlier changes during the 0.3.4 series should make this call even
rarer than it was before, since we fixed#25927 and removed
cached_gettimeofday. Because this test didn't update
cached_gettimeofday, the chance of rolling over a 10-second interval
was much higher.)
Closes ticket 25994; bugfix on 0.3.3.1-alpha when this test was
introduced.
Arguably, the conditions under which these events happen should be a
bit different, but the rules are complex enough here that I've tried
to have this commit be pure refactoring.
Closes ticket 25952.
Finally, before this code goes away, take a moment to look at the
amazing way that we used to try to have an event happen
every N seconds:
get_uptime() / N != (get_uptime()+seconds_elapsed) / N
Truly, it is a thing of wonder. I'm glad we didn't start using this
pattern everywhere else.
By doing so, it is renamed to voting_schedule_recalculate_timing(). This
required a lot of changes to include voting_schedule.h everywhere that this
function was used.
This effectively now makes voting_schedule.{c|h} not include dirauth/dirvote.h
for that symbol and thus no dependency on the dirauth module anymore.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This function doesn't need to be public from the dirvote common file (which
will get renamed in future commit) so move it to dirauth/dirvote.c and make it
static.
Part of #25988
Signed-off-by: David Goulet <dgoulet@torproject.org>
It makes more sense to be in networkstatus.c so move it there and rename it
with the "networkstatus_" prefix.
Part of #25988
Signed-off-by: David Goulet <dgoulet@torproject.org>
Apparently, we can decide our state is dirty before we create the
event to tell the mainloop that we should save it. That's not a
problem, except for the assertion failure.
This requires that when a log cb happens, the event for flushing
queued events is scheduled, so we also add the necessary machinery
to have that happen.
Note that this doesn't actually help with logs from outside the main
thread, but those were already suppressed: see #25987 for a ticket
tracking that issue.
Sometimes the logging system will queue a log message for later.
When it does this, the callback will either get flushed at the next
safe time, or from the second-elapsed callback.
But we're trying to eliminate the second-elapsed callback, so let's
make a way for the log system to tell its users about this.
Commit 0f3b765b3c added
tor_assert_nonfatal_unreached() to dirvote_add_vote() and
dirvote_add_signatures() when the dirauth module is disabled.
However, they need to return a value. Furthermore, the dirvote_add_vote()
needs to set the msg_out and status_out so it can be sent back. Else,
uninitialized values would be used.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Originally, it was made public outside of the dirauth module but it is no
longer needed. In doing so, we put it back in dirvote.c and reverted its name
to the original one:
dirvote_authority_cert_dup() --> authority_cert_dup()
Signed-off-by: David Goulet <dgoulet@torproject.org>
The --disable-module-* configure option removes code from the final binary but
we still build the unit tests with the disable module(s) so we can actually
test that code path all the time and not forget about it.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This code is only for dirauth so this commit moves it into the module in
dirvote.c.
No code behavior change.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Both functions are used for directory request but they can only be used if the
running tor instance is a directory authority.
For this reason, make those symbols visible but hard assert() if they are
called when the module is disabled. This would mean we failed to safeguard the
entry point into the module.
Signed-off-by: David Goulet <dgoulet@torproject.org>
In order to further isolate the dirauth code into its module, this moves the
handling of the directory request GET /tor/status-vote/* into the module.
Signed-off-by: David Goulet <dgoulet@torproject.org>
In order to make sr_commit_free() only used by the dirauth module, this
commits moves the commits free from a vote object into the dirvote.c file
which is now only for the module.
The function does nothing if the module is disabled.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Adds two unittests:
- First checks the path selection of basic Tor circs.
- Second checks the path selection of vanguard circs.
There is a TODO on the second unittest that we might want to test sooner than
later, but it's not trivial to do it right now.
To do these unittests we needed the following mods:
- Make some functions STATIC.
- Add some more fields to the big fake network nodes of test_entrynodes.c
- Switch fake node nicknames to base32 (because base64 does not produce valid nicknames).
This prevents a malicious RP/IP from learning the guard node in the case that
we are using only one (because we aren't using two guards, or because one of
those two guards is temporarily down).
This ensures the "strong" version of Property #6 from
https://lists.torproject.org/pipermail/tor-dev/2018-April/013098.html
(Information about the guard(s) does not leak to the website/RP at all).
The last hop in vanguard circuits can be an RP/IP/HSDir.
Since vanguard circuits are at least 3 hops (sometimes 4) before this node,
this change will not cause A - B - A paths.
When parsing a vote in routerparse.c, only dirauth extract the commits from
the vote so move all this code into dirvote.c so we can make it specific to
the dirauth module.
If the dirauth module is disabled, the commit parsing does nothing.
Signed-off-by: David Goulet <dgoulet@torproject.org>
From dirvote.c to networkstatus.c where it makes more sense both in terms of
namespace and subsystem responsability.
This removes one less dependency on the dirauth module.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Add static inline dirauth public functions used outside of the dirauth module
so they can be seen by the tor code but simply do nothing.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Move most of the shared random functions that are needed outside of the
dirauth module.
At this commit, because dirvote.c hasn't been refactor, it doesn't compile
because some SR functions need a dirvote function.
Furthermore, 5 functions haven't been touched yet because they are dirauth
only but are in used in other C files than the dirauth module ones.
No code behavior change. Only moving code around.
Signed-off-by: David Goulet <dgoulet@torproject.org>
This is a pretty big commit but it only moves these files to src/or/dirauth:
dircollate.c dirvote.c shared_random.c shared_random_state.c
dircollate.h dirvote.h shared_random.h shared_random_state.h
Then many files are modified to change the include line for those header files
that have moved into a new directory.
Without using --disable-module-dirauth, everything builds fine. When using the
flag to disable the module, tor doesn't build due to linking errors. This will
be addressed in the next commit(s).
No code behavior change.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Remove useless include.
Clearly identify functions that are used by other part of Tor, functions that
are only used by the dirauth subsystem and functions that are exposed for unit
tests.
This will help us in the dirauth modularization effort.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Many functions become static to the C file or exposed to the tests within the
PRIVATE define of dirvote.h.
This commit moves a function to the top. No code behavior change.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Don't access the AuthoritativeDir options directly. We do this so we can move
authdir_mode() to the dirauth module.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Make our build system support a disable dirauth module option. It can only be
disabled explicitly with:
$ ./configure --disable-module-dirauth
If *not* specified that is enabled, an automake conditional variable is set to
true and a defined value for the C code:
AM_CONDITIONAL: BUILD_MODULE_DIRAUTH
AC_DEFINE: HAVE_MODULE_DIRAUTH=1
This introduces the dirauth/ module directory in src/or/ for which .c files
are only compiled if the BUILD_MODULE_DIRAUTH is set.
All the header files are compiled in regardless of the support so we can use
the alternative entry point functions of the dirauth subsystem.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Because we rescan the main loop event list if the global map of services has
changed, this makes sure it does work.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Because ADD_ONION/DEL_ONION can modify the global service map (both for v2 and
v3), we need to rescan the event list so we either enable or disable the HS
service main loop event.
Fixees #25939
Signed-off-by: David Goulet <dgoulet@torproject.org>
This is done because it makes our life easier with unit tests. Also, a rescan
on an uninitialized event list will result in a stacktrace.
Signed-off-by: David Goulet <dgoulet@torproject.org>
When we change the hibernation state, rescan the main loop event list because
the new state might affect the events.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Implement the ability to set flags per events which influences the set up of
the event.
This commit only adds one flag which is "need network" meaning that the event
is not enabled if tor has disabled the network or if hibernation mode.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Needed to run tests from the tarball else the geoip unit test would fail by
not finding that file.
Signed-off-by: David Goulet <dgoulet@torproject.org>
In 25374, we created the necessary post-loop event for scheduling
connection_ap_attach_pending as needed. Before that, we were
already running this event once per mainloop. There's no reason to
also run it once per second.
Closes ticket 25933. No changes file, since the relevant change is
already in 25374. Or possibly in 17590, depending on how you look
at it.
Our main function, though accurate on all platforms, can be very
slow on 32-bit hosts. This one is faster on all 32-bit hosts, and
accurate everywhere except apple, where it will typically be off by
1%. But since 32-bit apple is a relic anyway, I think we should be
fine.
Previously were using this value to have a cheap highish-resolution
timer. But we were only using it in one place, and current dogma is
to use monotime_coarse_t for this kind of thing.
This part of the code was the only part that used "cached
getttimeofday" feature, which wasn't monotonic, which we updated at
slight expense, and which I'd rather not maintain.
The clean_consdiffmgr() callback is only for relays acting as a directory
server, not all relays.
This commit adds a role for only directory server and sets the
clean_consdiffmgr() callback to use it.
Signed-off-by: David Goulet <dgoulet@torproject.org>
We can't end up in the removed else {} condition since we first validate the
flavor we get and then we validate the flavor we parse from the given
consensus which means we can only handle the two flavors of the if/elseif
conditions.
Fixes#25914
Signed-off-by: David Goulet <dgoulet@torproject.org>
Previously, we were ignoring values _over_ EPSILON. This bug was
also causing a warning at startup because the default value is set
to -1.0.
Fixes bug 25577; bugfix on 6b1dba214d. Bug not in any released tor.
Two helper functions to enable an event and disable an event which wraps the
launch and destroy of an event but takes care of the enabled flag.
They are also idempotent that is can be called multiple time on the same event
without effect if the event was already enabled or disabled.
Signed-off-by: David Goulet <dgoulet@torproject.org>
In case we transitionned to a new role in Tor, we need to launch and/or
destroy some periodic events.
Signed-off-by: David Goulet <dgoulet@torproject.org>
In tor, we have a series of possible "roles" that the tor daemon can be
enabled for. They are:
Client, Bridge, Relay, Authority (directory or bridge) and Onion service.
They can be combined sometimes. For instance, a Directory Authority is also a
Relay. This adds a "roles" field to a periodic event item object which is used
to know for which roles the event is for.
The next step is to enable the event only if the roles apply. No behavior
change at this commit.
Pars of #25762
Signed-off-by: David Goulet <dgoulet@torproject.org>
No behavior change, just to make it easier to find callbacks and for the sake
of our human brain to parse the list properly.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Consensus method 25 is the oldest one supported by any stable
version of 0.2.9, which is our current most-recent LTS. Thus, by
proposal 290, they should be removed.
This commit does not actually remove the code to implement these
methods: it only makes it so authorities will no longer support
them. I'll remove the backend code for them in later commits.
It tried to pick nodes for which only routerinfo_t items are set,
but without setting UseMicroDescriptors to 0. This won't work any
more, now that we're strict about using the right descriptor types
due to 25691/25692/25213.
In order to fix 25691 and 25692, we need to pass the "direct_conn"
flag to more places -- particularly when choosing single-hop
tunnels. The right way to do this involves having a couple more
functions accept router_crn_flags_t, rather than a big list of
boolean arguments.
This commit also makes sure that choose_good_exit_server_general()
honors the direct_conn flag, to fix 25691 and 25692.
In router_add_running_nodes_to_smartlist(), we had an inline
implementation of the logic from node_has_descriptor(), which should
be changed to node_has_preferred_descriptor().
This patch adds a new node_has_preferred_descriptor() function, and
replaces most users of node_has_descriptor() with it. That's an
important change, since as of d1874b4339 (our fix for #25213),
we are willing to say that a node has _some_ descriptor, but not the
_right_ descriptor for a particular use case.
Part of a fix for 25691 and 25692.
This is done as follows:
* Only one function (find_dl_schedule()) actually returned a
smartlist. Now it returns an int.
* The CSV_INTERVAL type has been altered to ignore everything
after the first comma, and to store the value before the first
comma in an int.
This commit won't compile. It was made with the following perl
scripts:
s/smartlist_t \*(.*)DownloadSchedule;/int $1DownloadInitialDelay;/;
s/\b(\w*)DownloadSchedule\b/$1DownloadInitialDelay/;
When openssl is built with no-deprecated, the TLSv1_1_method()
function isn't visible in the headers. That's sad, because that
method is what we were looking at.
Instead, we now look at SSL_CIPHER_get_id(), which is present in
OpenSSL 1.0.1 and later, which is _not_ deprecated, and which is
also present in LibreSSL.
Fixes ticket 25353. Not a bugfix exactly -- we never really worked
with this configuration.
sizeof(ret) is the size of the pointer, not the size of what it
points to. Fortunately, we already have a function to compare
tor_addr_port_t values for equality.
Bugfix on c2c5b13e5d8a77e; bug not in any released Tor. Found by
clang's scan-build.
Now that we update our buckets on demand before reading or writing,
we no longer need to update them all every TokenBucketRefillInterval
msec.
When a connection runs out of bandwidth, we do need a way to
reenable it, however. We do this by scheduling a timer to reenable
all blocked connections for TokenBucketRefillInterval msec after a
connection becomes blocked.
(If we were using PerConnBWRate more, it might make sense to have a
per-connection timer, rather than a single timeout. But since
PerConnBWRate is currently (mostly) unused, I'm going to go for the
simpler approach here, since usually whenever one connection has
become blocked on bandwidth, most connections are blocked on
bandwidth.)
Implements ticket 25373.
Previously this was done as part of the refill callback, but there's
no real reason to do it like that. Since we're trying to remove the
refill callback completely, we can do this work as part of
record_num_bytes_transferred_impl(), which already does quite a lot
of this.
We used to do this 10x per second in connection_buckets_refill();
instead, we now do it when the bucket becomes empty. This change is
part of the work of making connection_buckets_refill() obsolete.
Closes ticket 25828; bugfix on 0.2.3.5-alpha.
Unfortunately, the units passed to
monotime_coarse_stamp_units_to_approx_msec() was always 0 due to a type
conversion.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Both header and code file had some indentation issues after mass renaming.
No code behavior change.
Signed-off-by: David Goulet <dgoulet@torproject.org>
Really, the uint32_t is only an optimization; any kind of unit
should work fine. Some users might want to use time_t or
monotime_coarse_t or something like that.
Begin by creating a lowest-level triple of the types needed to
implement a token bucket: a configuration, a timestamp, and the raw
bucket itself.
Note that for low-level buckets, the units of the timestamp and the
bucket itself are unspecified: each user can use a different type.
(This patch breaks check-spaces; a later patch will fix it)
This is a simple search-and-replace to rename the token bucket type
to indicate that it contains both a read and a write bucket, bundled
with their configuration. It's preliminary to refactoring the
bucket type.
This test works by having two post-loop events activate one another
in a tight loop. If the "post-loop" mechanism didn't work, this
would be enough to starve all other events.
A linked connection_t is one that gets its I/O, not from the
network, but from another connection_t. When such a connection has
something to write, we want the corresponding connection to run its
read callback ... but not immediately, to avoid infinite recursion
and/or event loop starvation.
Previously we handled this case by activating the read events
outside the event loop. Now we use the "postloop event" logic.
This lets us simplify do_main_loop_once() a little.
We've been labeling some events as happening "outside the event
loop", to avoid Libevent starvation. This patch provides a cleaner
mechanism to avoid that starvation.
For background, the problem here is that Libevent only scans for new
events once it has run all its active callbacks. So if the
callbacks keep activating new callbacks, they could potentially
starve Libevent indefinitely and keep it from ever checking for
timed, socket, or signal events.
To solve this, we add the ability to label some events as
"post-loop". The rule for a "post-loop" event is that any events
_it_ activates can only be run after libevent has re-scanned for new
events at least once.
This differs from our previous token bucket abstraction in a few
ways:
1) It is an abstraction, and not a collection of fields.
2) It is meant to be used with monotonic timestamps, which should
produce better results than calling gettimeofday over and over.
This roughly doubles our test coverage of the bridges.c module.
* ADD new testing module, .../src/test/test_bridges.c.
* CHANGE a few function declarations from `static` to `STATIC`.
* CHANGE one function in transports.c, transport_get_by_name(), to be
mockable.
* CLOSES#25425: https://bugs.torproject.org/25425
* ADD new /src/common/crypto_rand.[ch] module.
* ADD new /src/common/crypto_util.[ch] module (contains the memwipe()
function, since all crypto_* modules need this).
* FIXES part of #24658: https://bugs.torproject.org/24658
This module doesn't actually need to mock the libevent mainloop at
all: it can just use the regular mainloop that the test environment
sets up.
Part of ticket 23750.
This change makes cpuworker and test_workqueue no longer need to
include event2/event.h. Now workqueue.c needs to include it, but
that is at least somewhat logical here.
We are going to stop recommending 0.2.5 so there is no reason to keep the
undef statement anymore.
Fixes#20522.
Signed-off-by: Fernando Fernandez Mancera <ffmancera@riseup.net>
Every node in the network uses that value, it is a general server options, not
a dirauth specific one.
Fixes#25720
Signed-off-by: David Goulet <dgoulet@torproject.org>
Both in geoip_note_client_seen() and options_need_geoip_info(), switch from
accessing the options directly to using the should_record_bridge_info() helper
function.
Fixes#25290
Signed-off-by: David Goulet <dgoulet@torproject.org>
Previously, the limit for MAX_PROTOCOLS_TO_EXPAND was actually being applied
in Rust to the maximum number of version (total, for all subprotocols).
Whereas in C, it was being applied to the number of subprotocols that were
allowed. This changes the Rust to match C's behaviour.
The behaviours still do not match, unsurprisingly, but now we know where a
primary difference is: the Rust is validating version ranges more than the C,
so in the C it's possible to call protover_all_supported on a ridiculous
version range like "Sleen=0-4294967294" because the C uses
MAX_PROTOCOLS_TO_EXPAND to count the number of *subprotocols* whereas the Rust
uses it to count the total number of *versions* of all subprotocols.
There's now no difference in these tests w.r.t. the C or Rust: both
fail miserably (well, Rust fails with nice descriptive errors, and C
gives you a traceback, because, well, C).
The DoS potential is slightly higher in C now due to some differences to the
Rust code, see the C_RUST_DIFFERS tags in src/rust/protover/tests/protover.rs.
Also, the comment about "failing at the splitting stage" in Rust wasn't true,
since when we split, we ignore empty chunks (e.g. "1--1" parses into
"(1,None),(None,1)" and "None" can't be parsed into an integer).
Finally, the comment about "Rust seems to experience an internal error" is only
true in debug mode, where u32s are bounds-checked at runtime. In release mode,
code expressing the equivalent of this test will error with
`Err(ProtoverError::Unparseable)` because 4294967295 is too large.
Previously, if "Link=1-5" was supported, and you asked protover_all_supported()
(or protover::all_supported() in Rust) if it supported "Link=3-999", the C
version would return "Link=3-999" and the Rust would return "Link=6-999". These
both behave the same now, i.e. both return "Link=6-999".
During code review and discussion with Chelsea Komlo, she pointed out
that protover::compute_for_old_tor() was a public function whose
return type was `&'static CStr`. We both agree that C-like parts of
APIs should:
1. not be exposed publicly (to other Rust crates),
2. only be called in the appropriate FFI code,
3. not expose types which are meant for FFI code (e.g. `*mut char`,
`CString`, `*const c_int`, etc.) to the pure-Rust code of other
crates.
4. FFI code (e.g. things in `ffi.rs` modules) should _never_ be called
from pure-Rust, not even from other modules in its own crate
(i.e. do not call `protover::ffi::*` from anywhere in
`protover::protoset::*`, etc).
With that in mind, this commit makes the following changes:
* CHANGE `protover::compute_for_old_tor()` to be
visible only at the `pub(crate)` level.
* RENAME `protover::compute_for_old_tor()` to
`protover::compute_for_old_tor_cstr()` to reflect the last change.
* ADD a new `protover::compute_for_old_tor()` function wrapper which
is public and intended for other Rust code to use, which returns a
`&str`.
It was changed to take borrows instead of taking ownership.
* REFACTOR `protover::ffi::protover_is_supported_here()` to use changed method
signature on `protover::is_supported_here()`.
This includes a subtle difference in behaviour to the previous Rust
implementation, where, for each vote that we're computing over, if a single one
fails to parse, we skip it. This now matches the current behaviour in the C
implementation.
* REFACTOR `protover::ffi::protover_compute_vote()` to use
new types and methods.
This includes a subtle difference in behaviour, as in 4258f1e18, where we return
(matching the C impl's return behaviour) earlier than before if parsing failed,
saving us computation in parsing the versions into a
protover::protoset::ProtoSet.
* REFACTOR `protover::ffi::protover_list_supports_protocol_or_later()` to use
new types and methods.
This includes a subtle difference in behaviour, as in 4258f1e18, where we return
(matching the C impl's return behaviour) earlier than before if parsing failed,
saving us computation in parsing the versions into a
protover::protoset::ProtoSet.
* REFACTOR `protover::ffi::protover_list_supports_protocol()` to use new types
and methods.
This includes differences in behaviour to before, which should now more closely
match the C version:
- If parsing a protover `char*` from C, and the string is not parseable, this
function will return 1 early, which matches the C behaviour when protocols
are unparseable. Previously, we would parse it and its version numbers
simultaneously, i.e. there was no fail early option, causing us to spend more
time unnecessarily parsing versions.
* REFACTOR `protover::ffi::protover_all_supported()` to use new types and
methods.
Previously, the rust implementation of protover considered an empty string to be
a valid ProtoEntry, while the C version did not (it must have a "=" character).
Other differences include that unknown protocols must now be parsed as
`protover::UnknownProtocol`s, and hence their entries as
`protover::UnvalidatedProtoEntry`s, whereas before (nearly) all protoentries
could be parsed regardless of how erroneous they might be considered by the C
version.
My apologies for this somewhat messy and difficult to read commit, if any part
is frustrating to the reviewer, please feel free to ask me to split this into
smaller changes (possibly hard to do, since so much changed), or ask me to
comment on a specific line/change and clarify how/when the behaviours differ.
The tests here should more closely match the behaviours exhibited by the C
implementation, but I do not yet personally guarantee they match precisely.
* REFACTOR unittests in protover::protover.
* ADD new integration tests for previously untested behaviour.
* FIXES part of #24031: https://bugs.torproject.org/24031.
This changes `protover::is_supported_here()` to be aware of new datatypes
(e.g. don't call `.0` on things which are no longer tuple structs) and also
changes the method signature to take borrows, making it faster, threadable, and
easier to read (i.e. the caller can know from reading the function signature
that the function won't mutate values passed into it).
* CHANGE the `protover::is_supported_here()` function to take borrows.
* REFACTOR the `protover::is_supported_here()` function to be aware of new
datatypes.
* FIXES part of #24031: https://bugs.torproject.org/24031
This adds a new type for votes upon `protover::ProtoEntry`s (technically, on
`protover::UnvalidatedProtoEntry`s, because the C code does not validate based
upon currently known protocols when voting, in order to maintain
future-compatibility), and converts several functions which would have operated
on this datatype into methods for ease-of-use and readability.
This also fixes a behavioural differentce to the C version of
protover_compute_vote(). The C version of protover_compute_vote() calls
expand_protocol_list() which checks if there would be too many subprotocols *or*
expanded individual version numbers, i.e. more than MAX_PROTOCOLS_TO_EXPAND, and
does this *per vote* (but only in compute_vote(), everywhere else in the C seems
to only care about the number of subprotocols, not the number of individual
versions). We need to match its behaviour in Rust and ensure we're not allowing
more than it would to get the votes to match.
* ADD new `protover::ProtoverVote` datatype.
* REMOVE the `protover::compute_vote()` function and refactor it into an
equivalent-in-behaviour albeit more memory-efficient voting algorithm based
on the new underlying `protover::protoset::ProtoSet` datatype, as
`ProtoverVote::compute()`.
* REMOVE the `protover::write_vote_to_string()` function, since this
functionality is now generated by the impl_to_string_for_proto_entry!() macro
for both `ProtoEntry` and `UnvalidatedProtoEntry` (the latter of which is the
correct type to return from a voting protocol instance, since the entity
voting may not know of all protocols being voted upon or known about by other
voting parties).
* FIXES part of #24031: https://bugs.torproject.org/24031
rust: Fix a difference in compute_vote() behaviour to C version.
This implements conversions from either a ProtoEntry or an UnvalidatedProtoEntry
into a String, for use in replacing such functions as
`protover::write_vote_to_string()`.
* ADD macro for implementing ToString trait for ProtoEntry and
UnvalidatedProtoEntry.
* FIXES part of #24031: https://bugs.torproject.org/24031
This adds a new protover::UnvalidatedProtoEntry type, which is the
UnknownProtocol variant of a ProtoEntry, and refactors several functions which
should operate on this type into methods.
This also fixes what was previously another difference to the C implementation:
if you asked the C version of protovet_compute_vote() to compute a single vote
containing "Fribble=", it would return NULL. However, the Rust version would
return "Fribble=" since it didn't check if the versions were empty before
constructing the string of differences. ("Fribble=" is technically a valid
protover string.) This is now fixed, and the Rust version in that case will,
analogous to (although safer than) C returning a NULL, return None.
* REMOVE internal `contains_only_supported_protocols()` function.
* REMOVE `all_supported()` function and refactor it into
`UnvalidatedProtoEntry::all_supported()`.
* REMOVE `parse_protocols_from_string_with_no_validation()` and
refactor it into the more rusty implementation of
`impl FromStr for UnvalidatedProtoEntry`.
* REMOVE `protover_string_supports_protocol()` and refactor it into
`UnvalidatedProtoEntry::supports_protocol()`.
* REMOVE `protover_string_supports_protocol_or_later()` and refactor
it into `UnvalidatedProtoEntry::supports_protocol_or_later()`.
* FIXES part of #24031: https://bugs.torproject.org/24031
rust: Fix another C/Rust different in compute_vote().
This fixes the unittest from the prior commit by checking if the versions are
empty before adding a protocol to a vote.
This replaces the `protover::SupportedProtocols` (why would you have a type just
for things which are supported?) with a new, more generic type,
`protover::ProtoEntry`, which utilises the new, more memory-efficient datatype
in protover::protoset.
* REMOVE `get_supported_protocols()` and `SupportedProtocols::tor_supported()`
(since they were never used separately) and collapse their functionality into
a single `ProtoEntry::supported()` method.
* REMOVE `SupportedProtocols::from_proto_entries()` and reimplement its
functionality as the more rusty `impl FromStr for ProtoEntry`.
* REMOVE `get_proto_and_vers()` function and refactor it into the more rusty
`impl FromStr for ProtoEntry`.
* FIXES part of #24031: https://bugs.torproject.org/24031
* ADD new type, protover::UnknownProtocol, so that we have greater type safety
and our protover functionality which works with unsanitised protocol names is
more clearly demarcated.
* REFACTOR protover::Proto, renaming it protover::Protocol to mirror the new
protover::UnknownProtocol type name.
* ADD a utility conversion of `impl From<Protocol> for UnknownProtocol` so that
we can easily with known protocols and unknown protocols simultaneously
(e.g. doing comparisons, checking their version numbers), while not allowing
UnknownProtocols to be accidentally used in functions which should only take
Protocols.
* FIXES part of #24031: https://bugs.torproject.org/24031
* ADD new protover::protoset module.
* ADD new protover::protoset::ProtoSet class for holding protover versions.
* REMOVE protover::Versions type implementation and its method
`from_version_string()`, and instead implement this behaviour in a more
rust-like manner as `impl FromStr for ProtoSet`.
* MOVE the `find_range()` utility function from protover::protover to
protover::protoset since it's only used internally in the
implementation of ProtoSet.
* REMOVE the `contract_protocol_list()` function from protover::protover and
instead refactor it (reusing nearly the entire thing, with minor superficial,
i.e. non-behavioural, changes) into a more rusty
`impl ToString for ProtoSet`.
* REMOVE the `expand_version_range()` function from protover::protover and
instead refactor it into a more rusty implementation of
`impl Into<Vec<Version>> for ProtoSet` using the new error types in
protover::errors.
* FIXES part of #24031: https://bugs.torproject.org/24031.
This will allow us to do actual error handling intra-crate in a more
rusty manner, e.g. propogating errors in match statements, conversion
between error types, logging messages, etc.
* FIXES part of #24031: https://bugs.torproject.org/24031.
When a relay is collecting internal statistics about how many
create cell requests it has seen of each type, accurately count the
requests from relays that temporarily fall out of the consensus.
(To be extra conservative, we were already ignoring requests from clients
in our counts, and we continue ignoring them here.)
Fixes bug 24910; bugfix on 0.2.4.17-rc.
Add a missing lock acquisition around access to queued_control_events
in control_free_all(). Use the reassign-and-unlock strategy as in
queued_events_flush_all(). Fixes bug 25675. Coverity found this bug,
but only after we recently added an access to
flush_queued_event_pending.
These tests handle incoming and outgoing cells on a three-hop
circuit, and make sure that the crypto works end-to-end. They don't
yet test spec conformance, leaky-pipe, or various error cases.
Without this fix, we were just getting SRCDIR=`pwd`, which naturally
was breaking out-of-tree builds.
Bugfix on becae4c943969a4f4f14423cc897d39f41af7773; bug not in any
released Tor.
These declarations need to exist unconditionally, but they were
trapped inside an "#else /* !(defined(HAVE_SYSLOG_H)) */" block.
Fixes a travis regression caused by 23881; bug not in any released tor.
Additionally, this change extracts the functions that created and
freed these elements.
These structures had common "forward&reverse stream&digest"
elements, but they were initialized and freed through cpath objects,
and different parts of the code depended on them. Now all that code
is extacted, and kept in relay_crypto.c
This should help us improve modularity, and should also make it
easier for people to experiment with other relay crypto strategies
down the road.
This commit is pure function movement.
This function is used upon receiving a cell, and only handles the
decrypting part. The encryption part is currently handled inside
circuit_package_relay_cell.
* ADD includes for "torint.h" and "container.h" to crypto_digest.h.
* ADD includes for "crypto_digest.h" to a couple places in which
crypto_digest_t was then missing.
* FIXES part of #24658: https://bugs.torproject.org/24658#comment:30
Folks have found two in the past week or so; we may as well fix the
others.
Found with:
\#!/usr/bin/python3
import re
def findMulti(fname):
includes = set()
with open(fname) as f:
for line in f:
m = re.match(r'^\s*#\s*include\s+["<](\S+)[>"]', line)
if m:
inc = m.group(1)
if inc in includes:
print("{}: {}".format(fname, inc))
includes.add(m.group(1))
import sys
for fname in sys.argv[1:]:
findMulti(fname)
We moved the crypto_pk_obselete_* functions into crypto_rsa.[ch] because they fit
better with the RSA module.
Follows #24658.
Signed-off-by: Fernando Fernandez Mancera <ffmancera@riseup.net>
We moved the crypto_pk_* digest functions into crypto_rsa.[ch] because they fit
better with the RSA module.
Follows #24658.
Signed-off-by: Fernando Fernandez Mancera <ffmancera@riseup.net>
Since 0.2.4, tor uses EWMA circuit policy to prioritize. The previous
algorithm, round-robin, hasn't been used since then but was still used as a
fallback.
Now that EWMA is mandatory, remove that code entirely and enforce a cmux
policy to be set.
This is part of a circuitmux cleanup to improve performance and reduce
complexity in the code. We'll be able to address future optimization with this
work.
Closes#25268
Signed-off-by: David Goulet <dgoulet@torproject.org>
The reason to do so is because these functions haven't been used in years so
since 0.2.4, every callsite is NOP.
In future commits, we'll remove the round robin circuit policy which is mostly
validated within those function.
This simplifies the code greatly and remove dead code for which we never had a
configure option in the first place nor an easy way to use them in production.
Part of #25268
Signed-off-by: David Goulet <dgoulet@torproject.org>
It is rename to something more meaningful that explains what it does exactly
which is sets the EWMA options (currently only one exists). The new name is
cmux_ewma_set_options().
Also, remove a public function from circuitmux_ewma.h that is only used in the
C file. Make it static inline as well.
Signed-off-by: David Goulet <dgoulet@torproject.org>
To achieve this, a default value for the CircuitPriorityHalflife option was
needed. We still look in the options and then the consensus but in case no
value can be found, the default CircuitPriorityHalflifeMsec=30000 is used. It
it the value we've been using since 0.2.4.4-alpha.
This means that EWMA, our only policy, can not be disabled anymore fallbacking
to the round robin algorithm. Unneeded code to control that is removed in this
commit.
Part of #25268
Signed-off-by: David Goulet <dgoulet@torproject.org>
* ADD a new macro, tor_util::string::cstr!() which takes Rust strings,
concatenates them together, appends a NUL byte, and converts it into a
std::ffi::CStr for handing to C.
Since we're making it so that unstable zstd apis can be disabled,
we need to test them. I do this by adding a variant setup/cleanup
function for the tests, and teaching it about a fake compression
method called "x-zstd:nostatic".
Included crypto_digest.[ch] into include.am in order to solve a compiling
issue. Also EOF line in crypto_digest.c added.
Follows #24658.
Signed-off-by: Fernando Fernandez Mancera <ffmancera@riseup.net>
Included crypto_digest.h in some files in order to solve xof+digest module
dependency issues. Removed crypto.h where it isn't needed anymore.
Follows #24658.
Signed-off-by: Fernando Fernandez Mancera <ffmancera@riseup.net>
Add two new files (crypto_digest.c, crypto_digest.h) as new module of
crypto.[ch]. This new module includes all functions and dependencies related
to digest and xof operations. Those have been removed from crypto.[ch].
Follows #24658.
Signed-off-by: Fernando Fernandez Mancera <ffmancera@riseup.net>
Using get_uptime() and reset_uptime() instead of
accessing stats_n_seconds_working directly.
stats_n_seconds_working is not extern anymore.
Ticket #25081
I'm leaving the getsockname code in transproxy alone, since it is
comparatively isolated, rather platform-specific, and hard to test.
Implements 18105.
We'd been using crypto_digest_dup() and crypto_digest_assign() here,
but they aren't necessary. Instead we can just use the stack to
store the previous state of the SHA_CTX and avoid a malloc/free pair.
Closes ticket 24914.
In order to make the OR and dir checking functions in router.c less confusing
we renamed some functions and splitted consider_testing_reachability() into
router_should_check_reachability() and router_do_reachability_checks(). Also we
improved the documentation.
Fixes#18918.
Signed-off-by: Fernando Fernandez Mancera <ffmancera@riseup.net>
## We run `make check` because that's what https://jenkins.torproject.org does.
- if [[ "$DISTCHECK" == "" ]]; then make check; fi
- if [[ "$DISTCHECK" != "" ]]; then make distcheck DISTCHECK_CONFIGURE_FLAGS="$RUST_OPTIONS $COVERAGE_OPTIONS --disable-asciidoc --enable-fatal-warnings --disable-silent-rules --enable-fragile-hardening"; fi
Nick Mathewson
rl1987
Mike Perry
David Goulet
Roger Dingledine
Isis Lovecruft
George Kadianakis
Marcin Cieślak
Taylor Yu
juga0
Alex Xu (Hello71)
Georg Koppen
Neel Chauhan
Alexander Færøy
Deepesh Pathak
Fernando Fernandez Mancera
ArunaMaurya221B
Caio Valente
teor
Chelsea Holland Komlo