Merge branch 'maint-0.2.2' into release-0.2.2

changes/bug1035

@@ -0,0 +1,13 @@
+  o Minor features (authorities)
+    - Take altered router IP addresses and ORPorts into account when
+      determining router stability.  Previously, if a router changed
+      its IP or ORPort, the authorities would not treat it as having
+      any downtime for the purposes of stability calculation, whereas
+      clients would experience downtime since the change could take a
+      while to propagate to them.  Resolves issue 1035.
+  o Minor bugfixes (authorities)
+    - Try to be more robust to hops back in time when calculating
+      router stability.  Previously, if a run of uptime or downtime
+      appeared to be negative, the calculation could give incorrect
+      results.  Bugfix on 0.2.0.6-alpha.
+

changes/bug2629

@@ -0,0 +1,5 @@
+  o Minor bugfixes
+    - Fix a crash bug that could occur occasionally when a client was
+      configured with a large number of bridges.  Fixes bug 2629; bugfix
+      on 0.2.1.2-alpha.  Bugfix by trac user "shitlei".
+

changes/bug2660

@@ -0,0 +1,7 @@
+  o Minor bugfixes:
+    - Fix connect() failures on some platforms (BSD, OS X). Bugfix on
+      0.2.0.3-alpha; fixes first part of bug 2660. Patch by piebeer.
+    - Set target port in get_interface_address6() correctly. Bugfix
+      on 0.1.1.4-alpha and 0.2.0.3-alpha; fixes second part of bug
+      2660.
+

doc/HACKING

@@ -5,17 +5,17 @@ Getting started
 ---------------
 
 For full information on how Tor is supposed to work, look at the files in
-doc/spec/ .
+https://gitweb.torproject.org/torspec.git/tree
 
 For an explanation of how to change Tor's design to work differently, look at
-doc/spec/proposals/001-process.txt .
+https://gitweb.torproject.org/torspec.git/blob_plain/HEAD:/proposals/001-process.txt
 
 For the latest version of the code, get a copy of git, and
 
    git clone git://git.torproject.org/git/tor .
 
-We talk about Tor on the or-talk mailing list.  Design proposals and
-discussion belong on the or-dev mailing list.  We hang around on
+We talk about Tor on the tor-talk mailing list.  Design proposals and
+discussion belong on the tor-dev mailing list.  We hang around on
 irc.oftc.net, with general discussion happening on #tor and development
 happening on #tor-dev.
 
@@ -65,8 +65,8 @@ If at all possible, try to create this file in the same commit where
 you are making the change.  Please give it a distinctive name that no
 other branch will use for the lifetime of your change.
 
-When Roger goes to make a release, he will concatenate all the entries
-in changes to make a draft changelog, and clear the directory.  He'll
+When we go to make a release, we will concatenate all the entries
+in changes to make a draft changelog, and clear the directory. We'll
 then edit the draft changelog into a nice readable format.
 
 What needs a changes file?::
@@ -312,11 +312,11 @@ operation.
 If a library function is currently called such that failure always means ERR,
 then the library function should log WARN and let the caller log ERR.
 
-[XXX Proposed convention: every message of severity INFO or higher should
-either (A) be intelligible to end-users who don't know the Tor source; or (B)
-somehow inform the end-users that they aren't expected to understand the
-message (perhaps with a string like "internal error").  Option (A) is to be
-preferred to option (B). -NM]
+Every message of severity INFO or higher should either (A) be intelligible
+to end-users who don't know the Tor source; or (B) somehow inform the
+end-users that they aren't expected to understand the message (perhaps
+with a string like "internal error"). Option (A) is to be preferred to
+option (B).
 
 Doxygen
 ~~~~~~~~
@@ -405,4 +405,44 @@ function should mention that it does that something in the documentation.  If
 you rely on a function doing something beyond what is in its documentation,
 then you should watch out, or it might do something else later.
 
+Putting out a new release
+-------------------------
+
+Here are the steps Roger takes when putting out a new Tor release:
+
+1) Use it for a while, as a client, as a relay, as a hidden service,
+and as a directory authority. See if it has any obvious bugs, and
+resolve those.
+
+2) Gather the changes/* files into a changelog entry, rewriting many
+of them and reordering to focus on what users and funders would find
+interesting and understandable.
+
+3) Compose a short release blurb to highlight the user-facing
+changes. Insert said release blurb into the ChangeLog stanza. If it's
+a stable release, add it to the ReleaseNotes file too. If we're adding
+to a release-0.2.x branch, manually commit the changelogs to the later
+git branches too.
+
+4) Bump the version number in configure.in and rebuild.
+
+5) Make dist, put the tarball up somewhere, and tell #tor about it. Wait
+a while to see if anybody has problems building it. Try to get Sebastian
+or somebody to try building it on Windows.
+
+6) Get at least two of weasel/arma/karsten to put the new version number
+in their approved versions list.
+
+7) Sign and push the tarball to the website in the dist/ directory. Sign
+and push the git tag.
+
+8) Edit include/versions.wmi to note the new version. Rebuild and push
+the website.
+
+9) Email Erinn and weasel (cc'ing tor-assistants) that a new tarball
+is up. This step should probably change to mailing more packagers.
+
+10) Wait up to a day or two (for a development release), or until most
+packages are up (for a stable release), and mail the release blurb and
+changelog to tor-talk or tor-announce.
 

src/common/address.c

@@ -1025,27 +1025,28 @@ get_interface_address6(int severity, sa_family_t family, tor_addr_t *addr)
 {
   int sock=-1, r=-1;
   struct sockaddr_storage my_addr, target_addr;
-  socklen_t my_addr_len;
+  socklen_t addr_len;
 
   tor_assert(addr);
 
   memset(addr, 0, sizeof(tor_addr_t));
   memset(&target_addr, 0, sizeof(target_addr));
-  my_addr_len = (socklen_t)sizeof(my_addr);
-  /* Use the "discard" service port */
-  ((struct sockaddr_in*)&target_addr)->sin_port = 9;
   /* Don't worry: no packets are sent. We just need to use a real address
    * on the actual Internet. */
   if (family == AF_INET6) {
     struct sockaddr_in6 *sin6 = (struct sockaddr_in6*)&target_addr;
+    /* Use the "discard" service port */
+    sin6->sin6_port = htons(9);
     sock = tor_open_socket(PF_INET6,SOCK_DGRAM,IPPROTO_UDP);
-    my_addr_len = (socklen_t)sizeof(struct sockaddr_in6);
+    addr_len = (socklen_t)sizeof(struct sockaddr_in6);
     sin6->sin6_family = AF_INET6;
     S6_ADDR16(sin6->sin6_addr)[0] = htons(0x2002); /* 2002:: */
   } else if (family == AF_INET) {
     struct sockaddr_in *sin = (struct sockaddr_in*)&target_addr;
+    /* Use the "discard" service port */
+    sin->sin_port = htons(9);
     sock = tor_open_socket(PF_INET,SOCK_DGRAM,IPPROTO_UDP);
-    my_addr_len = (socklen_t)sizeof(struct sockaddr_in);
+    addr_len = (socklen_t)sizeof(struct sockaddr_in);
     sin->sin_family = AF_INET;
     sin->sin_addr.s_addr = htonl(0x12000001); /* 18.0.0.1 */
   } else {
@@ -1058,14 +1059,13 @@ get_interface_address6(int severity, sa_family_t family, tor_addr_t *addr)
     goto err;
   }
 
-  if (connect(sock,(struct sockaddr *)&target_addr,
-              (socklen_t)sizeof(target_addr))<0) {
+  if (connect(sock,(struct sockaddr *)&target_addr, addr_len) < 0) {
     int e = tor_socket_errno(sock);
     log_fn(severity, LD_NET, "connect() failed: %s", tor_socket_strerror(e));
     goto err;
   }
 
-  if (getsockname(sock,(struct sockaddr*)&my_addr, &my_addr_len)) {
+  if (getsockname(sock,(struct sockaddr*)&my_addr, &addr_len)) {
     int e = tor_socket_errno(sock);
     log_fn(severity, LD_NET, "getsockname() to determine interface failed: %s",
            tor_socket_strerror(e));

src/or/circuitbuild.c

@@ -4728,7 +4728,8 @@ any_pending_bridge_descriptor_fetches(void)
         conn->purpose == DIR_PURPOSE_FETCH_SERVERDESC &&
         TO_DIR_CONN(conn)->router_purpose == ROUTER_PURPOSE_BRIDGE &&
         !conn->marked_for_close &&
-        conn->linked && !conn->linked_conn->marked_for_close) {
+        conn->linked &&
+        conn->linked_conn && !conn->linked_conn->marked_for_close) {
       log_debug(LD_DIR, "found one: %s", conn->address);
       return 1;
     }

src/or/connection_or.c

@@ -1098,9 +1098,6 @@ connection_or_check_valid_tls_handshake(or_connection_t *conn,
       as_advertised = 0;
     }
     if (authdir_mode_tests_reachability(options)) {
-      /* We initiated this connection to address:port.  Drop all routers
-       * with the same address:port and a different key.
-       */
       dirserv_orconn_tls_done(conn->_base.address, conn->_base.port,
                               digest_rcvd_out, as_advertised);
     }

src/or/dirserv.c

@@ -3116,19 +3116,27 @@ dirserv_orconn_tls_done(const char *address,
   tor_assert(address);
   tor_assert(digest_rcvd);
 
-  SMARTLIST_FOREACH(rl->routers, routerinfo_t *, ri, {
+  /* XXX023 Doing a loop like this is stupid.  We should just look up the
+   * router by digest_rcvd, and see if address, orport, and as_advertised
+   * match up. -NM */
+  SMARTLIST_FOREACH_BEGIN(rl->routers, routerinfo_t *, ri) {
     if (!strcasecmp(address, ri->address) && or_port == ri->or_port &&
         as_advertised &&
         !memcmp(ri->cache_info.identity_digest, digest_rcvd, DIGEST_LEN)) {
       /* correct digest. mark this router reachable! */
       if (!bridge_auth || ri->purpose == ROUTER_PURPOSE_BRIDGE) {
-        log_info(LD_DIRSERV, "Found router %s to be reachable. Yay.",
-                 ri->nickname);
-        rep_hist_note_router_reachable(digest_rcvd, now);
+        tor_addr_t addr, *addrp=NULL;
+        log_info(LD_DIRSERV, "Found router %s to be reachable at %s:%d. Yay.",
+                 ri->nickname, address, ri->or_port );
+        if (tor_addr_from_str(&addr, ri->address) != -1)
+          addrp = &addr;
+        else
+          log_warn(LD_BUG, "Couldn't parse IP address \"%s\"", ri->address);
+        rep_hist_note_router_reachable(digest_rcvd, addrp, or_port, now);
         ri->last_reachable = now;
       }
     }
-  });
+  } SMARTLIST_FOREACH_END(ri);
   /* FFFF Maybe we should reinstate the code that dumps routers with the same
    * addr/port but with nonmatching keys, but instead of dumping, we should
    * skip testing. */

src/or/rephist.c

@@ -14,6 +14,7 @@
 #include "circuitlist.h"
 #include "circuituse.h"
 #include "config.h"
+#include "networkstatus.h"
 #include "rephist.h"
 #include "router.h"
 #include "routerlist.h"
@@ -73,6 +74,13 @@ typedef struct or_history_t {
   /** If nonzero, we have been unable to connect since this time. */
   time_t down_since;
 
+  /** The address at which we most recently connected to this OR
+   * successfully. */
+  tor_addr_t last_reached_addr;
+
+  /** The port at which we most recently connected to this OR successfully */
+  uint16_t last_reached_port;
+
   /* === For MTBF tracking: */
   /** Weighted sum total of all times that this router has been online.
    */
@@ -119,6 +127,7 @@ get_or_history(const char* id)
     rephist_total_num++;
     hist->link_history_map = digestmap_new();
     hist->since = hist->changed = time(NULL);
+    tor_addr_make_unspec(&hist->last_reached_addr);
     digestmap_set(history_map, id, hist);
   }
   return hist;
@@ -289,13 +298,20 @@ rep_hist_note_connection_died(const char* id, time_t when)
 /** We have just decided that this router with identity digest <b>id</b> is
  * reachable, meaning we will give it a "Running" flag for the next while. */
 void
-rep_hist_note_router_reachable(const char *id, time_t when)
+rep_hist_note_router_reachable(const char *id, const tor_addr_t *at_addr,
+                               const uint16_t at_port, time_t when)
 {
   or_history_t *hist = get_or_history(id);
   int was_in_run = 1;
   char tbuf[ISO_TIME_LEN+1];
+  int addr_changed, port_changed;
 
   tor_assert(hist);
+  tor_assert((!at_addr && !at_port) || (at_addr && at_port));
+
+  addr_changed = at_addr &&
+    tor_addr_compare(at_addr, &hist->last_reached_addr, CMP_EXACT) != 0;
+  port_changed = at_port && at_port != hist->last_reached_port;
 
   if (!started_tracking_stability)
     started_tracking_stability = time(NULL);
@@ -315,6 +331,27 @@ rep_hist_note_router_reachable(const char *id, time_t when)
     down_length = when - hist->start_of_downtime;
     hist->total_weighted_time += down_length;
     hist->start_of_downtime = 0;
+  } else if (addr_changed || port_changed) {
+    /* If we're reachable, but the address changed, treat this as some
+     * downtime. */
+    int penalty = get_options()->TestingTorNetwork ? 240 : 3600;
+    networkstatus_t *ns;
+
+    if ((ns = networkstatus_get_latest_consensus())) {
+      int fresh_interval = (int)(ns->fresh_until - ns->valid_after);
+      int live_interval = (int)(ns->valid_until - ns->valid_after);
+      /* on average, a descriptor addr change takes .5 intervals to make it
+       * into a consensus, and half a liveness period to make it to
+       * clients. */
+      penalty = (int)(fresh_interval + live_interval) / 2;
+    }
+    format_local_iso_time(tbuf, hist->start_of_run);
+    log_info(LD_HIST,"Router %s still seems Running, but its address appears "
+             "to have changed since the last time it was reachable.  I'm "
+             "going to treat it as having been down for %d seconds",
+             hex_str(id, DIGEST_LEN), penalty);
+    rep_hist_note_router_unreachable(id, when-penalty);
+    rep_hist_note_router_reachable(id, NULL, 0, when);
   } else {
     format_local_iso_time(tbuf, hist->start_of_run);
     if (was_in_run)
@@ -324,6 +361,10 @@ rep_hist_note_router_reachable(const char *id, time_t when)
       log_info(LD_HIST,"Router %s is now Running; it was previously untracked",
                hex_str(id, DIGEST_LEN));
   }
+  if (at_addr)
+    tor_addr_copy(&hist->last_reached_addr, at_addr);
+  if (at_port)
+    hist->last_reached_port = at_port;
 }
 
 /** We have just decided that this router is unreachable, meaning
@@ -344,12 +385,20 @@ rep_hist_note_router_unreachable(const char *id, time_t when)
     long run_length = when - hist->start_of_run;
     format_local_iso_time(tbuf, hist->start_of_run);
 
-    hist->weighted_run_length += run_length;
     hist->total_run_weights += 1.0;
     hist->start_of_run = 0;
-    hist->weighted_uptime += run_length;
-    hist->total_weighted_time += run_length;
+    if (run_length < 0) {
+      unsigned long penalty = -run_length;
+#define SUBTRACT_CLAMPED(var, penalty) \
+      do { (var) = (var) < (penalty) ? 0 : (var) - (penalty); } while (0)
 
+      SUBTRACT_CLAMPED(hist->weighted_run_length, penalty);
+      SUBTRACT_CLAMPED(hist->weighted_uptime, penalty);
+    } else {
+      hist->weighted_run_length += run_length;
+      hist->weighted_uptime += run_length;
+      hist->total_weighted_time += run_length;
+    }
     was_running = 1;
     log_info(LD_HIST, "Router %s is now non-Running: it had previously been "
              "Running since %s.  Its total weighted uptime is %lu/%lu.",
@@ -422,7 +471,7 @@ rep_hist_downrate_old_runs(time_t now)
 static double
 get_stability(or_history_t *hist, time_t when)
 {
-  unsigned long total = hist->weighted_run_length;
+  long total = hist->weighted_run_length;
   double total_weights = hist->total_run_weights;
 
   if (hist->start_of_run) {
@@ -458,8 +507,8 @@ get_total_weighted_time(or_history_t *hist, time_t when)
 static double
 get_weighted_fractional_uptime(or_history_t *hist, time_t when)
 {
-  unsigned long total = hist->total_weighted_time;
-  unsigned long up = hist->weighted_uptime;
+  long total = hist->total_weighted_time;
+  long up = hist->weighted_uptime;
 
   if (hist->start_of_run) {
     long run_length = (when - hist->start_of_run);

src/or/rephist.h

@@ -33,7 +33,8 @@ void rep_hist_update_state(or_state_t *state);
 int rep_hist_load_state(or_state_t *state, char **err);
 void rep_history_clean(time_t before);
 
-void rep_hist_note_router_reachable(const char *id, time_t when);
+void rep_hist_note_router_reachable(const char *id, const tor_addr_t *at_addr,
+                                    uint16_t at_port, time_t when);
 void rep_hist_note_router_unreachable(const char *id, time_t when);
 int rep_hist_record_mtbf_data(time_t now, int missing_means_down);
 int rep_hist_load_mtbf_data(time_t now);