[x] persist buffer across sessions
[x] better retransmission mechanism
[x] bin pickling
[ ] fast prioritizer



BUGS

- downloader
  - seen thrice: file is corrupted
    - once, a new, shorter cartel.bash was obtain but the old tail was not
      truncated
    - two times, a .gz file had "trailing garbage ignored" (once on cab18)
  - seen once on cab 35: the last log message was gunzip -c downloader.gz >
    /opt/bin/downloader, but the program was stuck there for a long time. no
    subprocess was running, and the main downloader thread was spinning the
    cpu.
  - seen twice: dies while tar xzmf, but no exit code being logged/written,
    possibly because (for unknown reason) downloader wasn't restarting self
    via really-start-downloader



TODO
- perform code size statistics
- use sql for the code on the cabs

- add the OE, system/ stuff
- in downloader: use md5sum to check files
  - re-download whole thing if necessary
  - server caches a snapshot of the md5sums for each file; if anything
    changes, update
  - downloads are now triggered by md5sums, not timestamps
- patch system



twisted vs af

- pro af
  - af and twisted aren't the same but have substantial overlap
    (i have entirely replaced twisted with af; this is the migration of 10k
     lines of code)
  - af is smaller
  - af is faster
  - af is more amenable to psyco
  - af has some python 2.5 niceties like contextual mutexes (multiplexing,
    pipelined, ...)
  - af doesn't have certain problems in twisted, in particular problems
    that are fundamental to the design of twisted, such as the stack
    explosion bug
  - cooperative threads, coroutines > events, deferreds
- con af
  - sockets > protocols
  - af requires python 2.5
  - af has no support for dealing with os threads; my extensions (afx)
    provide this though, but perhaps not as fast as they could be
  - dev team < 1 maintainer
- twisted has lots of users, support, and ecosystem built around the core

the cafnet problem

- why am I working on this?
  - thought that I was helping Vlad who needed this
  - I'm the only person using this
- interface
  - bind ->
  - schedule ->
  - <- get_msg
  - <- recv_msg
  - cancel ->
- issues (for ICEDB)
  - network-level buffering: very weak and unjustified to start with
    - ICEDB needs to control its own buffers (re-prioritize data)
    - yet it needs to send as soon as the conn is available
    - 2 layers of buffering: one for packet requests (schedules), one
      for actual packets (get_msg)
    - handle details like on-disconnect rescheduling
  - network-level prioritization
    - the reason for buffering, yet defeats purpose of buffering
      (efficiency)
    - mismatch between this and ICEDB's own prioritization
    - global prioritization would require continual aborting of msgs,
      which means lots of extra bookkeeping
  - rpc interface to cafnet
    - support for multiple client apps, but only one (ICEDB)
    - needed just for prioritization across apps
    - complicates the interface, esp. synchronization, error-handling,
      debugging, handling unavailability (unloading, connect-on-bind),
      etc.
    - de-multiplexing among many connections
  - connection notification
    - app needs this: eg updates, global prioritization
    - current solution
      - icedb uses cafnet to propagate updates (downward) and to collect
        data (upward); also for on-connect sync requests
      - added a connection notification callback
      - may change: for on-connect sync requests, use a direct TCP
        connection?
  - reliable transmission
    - requires persistent msg id generator due to re-ordering
    - still should use app-level acks
  - carrying and forwarding
    - a separate node ID layer - complicates setup/admin (assigning
      node IDs), addressing, discovery
    - ctl vs cnl: need to juggle packets between these two (eg
      requests to send, clear to send, pushing back reliable unacked
      packets to the ctl when the connection breaks)
    - lots of client code just for using cafnet
      - imposing ordering
      - multicast reliability
      - bookkeeping of buffered packets
      - synchronization
  - things that are originally easy to solve using soft-state cannot
    be solved that way because of the long delays in delay-tolerant
    networking
  - in general, making sure everything works in the face of
    disconnects is made much more complicated due to these issues
  - frustrations despite the fact that I try very hard to focus just
    on getting things working, not on eg
    optimizations/efficiency/other cleverness
  - cafnet/directly related to cafnet = ~70%-80% of all code



static logging tags

TODO: SHORT TERM

[x] cafnet re-sending a bunch of packets
    - cause: initial (and only) acks were lost
    - fix: ack already-received packets
[x] why isn't stuff appearing in psql?
    - cause: ObjectTransport not already-received objects
    - fix: persistent seqnos
[x] why doesn't gps produce anything? why is gps2db so flaky?
    - cause: gpsd bug
    - fix: wrote my own damned gps-adapter
[ ] make icedb/cafnet send old data (simplify the data model)
    - cause: no persistence of send status
    - fix: *acks*

TODO: LONG TERM (high to low priority)

- decide on the protocol
  - idea that may help in sticky situations with ordered delivery:
    special messages (commands) that instructs nodes to forget all
    past missing messages, and to only consider the subsequent
    messages (this could be periodically issued)
- another possibility
  - just treat the medium as bidirectional and have people send/receive crap on it, potentially very inefficiently
  - eg for bootstrapping ('hello i'm a new node, send me the full log to roll forward')
  - eg on missing packets ('hey i never got packets 4,5,6 but i just got 7-10, what's the dilio')
- make twisted log seconds?
- remove ObjectTransport (where it's not needed; ie all but file xfer)
- remove XMLRPC

tar czf kevin-small.tar.gz !(usr|var|kevin.tar.gz) usr/!(portage|src) var/!(cache) var/cache/!(edb)

- 2/19/07: pretty big (and seemingly small) twisted issue! no
abortConnection (http://twistedmatrix.com/trac/ticket/78) means
loseConnection won't actually reach connectionLost() till much later,
as it's waiting for the outgoing data buffer to be flushed... what
gives?

- 2/19/07: bug: not clearing the unacked buffer. data structure
inconsistency results (non-disjoint sets of metas across the various
sets: metas, outstanding, unacked). times like this when i wish we had
static shape analysis.

- 2/14/07: there's a bug in twisted where if you omit try to lose a
conn while a producer is register, what twisted *should* do is wait
for the producer to finish before actually losing the
connection. however, there seems to be a bug where calling
loseConnection while a producer is registered will send twisted into a
bad state, where we're disconnecting (and don't receive further
packets) but don't actually ever lose the connection (though it may be
possible to trigger the loss by trying to send another packet?)
  - correction: this is not a bug; i just don't think i ever
    unregister the producer

- perf on soekris
  - flood of 10*300 to harvard
    - start 1170835944 mid 1170835953 end 1170836641 mid--end 688
      start--end 697
- perf on harvard
  - tranferring 100*/etc/httpd/conf/httpd.conf
    - rpc:
        3.77user 0.83system 0:27.24elapsed 16%CPU (0avgtext+0avgdata 0maxresident)k
        0inputs+0outputs (0major+5693minor)pagefaults 0swaps
    - xmlrpc:
        real    0m50.499s
        user    0m15.907s
        sys     0m2.991s
  - flood of 1*10301
    - rpc:
        with b=1M,
        start 1170621310 mid 1170621312 end 1170621483 mid--end 171 start--end 173
        with b=10K,
        start 1170621631 mid 1170621633 end 1170621801 mid--end 168 start--end 170
    - xmlrpc:
        with b=1M, doesn't finish
        with b=10K, if it does finish (somtimes doesnt):
        start 1170621981 mid 1170621983 end 1170622365 mid--end 382 start--end 384
  - apparently, for vastly improved performance, keep the cnl buffer
    tiny (10K, default is 1M)
    - hypothesis: this limits the # of concurrent get_msg
      callbacks/'in flight' processing/mem consumption
    - this is characterized by very long pauses in the stream
    - with rpc, we always eventually "make it" and get no
      timeouterrors/unknownerrors
    - not so with xmlrpc

- remove util.py
- reorg the packages/modules into icedb.core so __init__ doesn't load cafnet

- notes:
  - multiple deep systems bugs
    - replies were occurring out of order, since rpc's could return
        deferreds (and thus allowed themselves arbitrary amounts of
        time to complete, enabling OOO returns)
    - at some point, when we're flooding the system with packets,
      cafnet throws "Failure: twisted.internet.error.TimeoutError:
      User timeout caused connection failure."
      - alternatively, we get:
        "Failure: twisted.internet.error.ConnectionLost: Connection to
        the other side was lost in a non-clean fashion."
      - analysis: after extensive debugging, the problem was found to
        have originated in two locations: get_msg and
        ack_receieved. since it only comes up if we're dealing with
        lots of messages, I'm assuming that it's because the
        connection gets messed up either due to too many FDs open on
        either ICEDB/Cafnet, or due to too great a lag for xmlrpc to
        handle. this is not due to xmlrpc per se (which should not
        have timeouts in its rpc's); i believe it is due to the *tcp
        connection*, based on the fact that i have seen this
        elsewhere. (Since the tests sent many messages in just one
        direction, lots of acks were received, but the same problem
        should be present with msg_received if we were getting lots of
        messages.)
      - solution: either have a pool of slots for xmlrpc connections,
        or serialize all of them (pool of size 1). the approach that I
        will take is to use the raw binary protocol which should
        serialize everything since it utilizes only a single tcp
        stream (which should replace XMLRPC anyway for performance
        reasons).
    - when copying a file, and too many deferreds get queued up on a
      deferredlock (via asynchronized), we get. however, this seems to
      only be a problem with cafnet's test.py.
      - solution: break the callback chain by using callLater to
        prevent the recursive stack explosion

- optimize / break up commons? here are some measurements from the soekris box:

loading icedb.bash
loading common.bash
done loading common.bash
loading bash-conf
loading commons 1170416467.33
loading commands 1170416467.96
loading cPickle as pickle 1170416467.99
loading csv 1170416468.09
loading errno 1170416468.47
loading glob 1170416468.48
loading heapq 1170416468.56
loading inspect 1170416468.8
loading itertools 1170416470.13
loading logging 1170416470.15
loading logging.config 1170416471.25
loading logging.handlers 1170416472.33
loading os 1170416472.33
loading os.path 1170416472.34
loading path as paths 1170416472.35
loading re 1170416472.85
loading select 1170416472.86
loading shutil 1170416472.9
loading signal 1170416472.91
loading SimpleXMLRPCServer 1170416472.93
loading SocketServer 1170416475.27
loading string 1170416475.28
loading sys 1170416475.3
loading tempfile 1170416475.3
loading threading 1170416475.32
loading time 1170416475.33
loading urllib 1170416475.34
loading urllib2 1170416476.36
loading UserDict 1170416479.96
loading xmlrpclib 1170416479.98
loading icedb.conf 1170416480.11
loading cafnet 1170416480.65
loading zope.interface import Attribute, Interface, implements 1170416481.2
loading twisted.application import internet, service 1170416482.47
loading twisted.internet import interfaces, protocol, reactor, defer 1170416490.03
loading twisted.python import components, log 1170416492.57
loading twisted.web import resource, server 1170416492.58
loading commons, commons.async 1170416497.83
loading cStringIO, itertools, os, pickle, signal, socket, SocketServer, struct 1170416497.88
done loading cafnet 1170416498.3
loading alpha 1170416498.31
loading beta 1170416498.38
loading gamma 1170416499.24
loading delta 1170416499.64
loading icedb.conf 1170416499.67
loading commons 1170416499.77
loading rest 1170416499.83
done loading bash-conf
done loading icedb.bash

vim:et:sw=2:ts=2
