ISSUE LOG

Tue Jan 15 04:15:53 EST 2008

trying to scale back everything:
- accel rate 100 -> 10 (effectively ~20-30 before anyway)
- disk double buffer switch min thresh 1000 -> 100 chunks
- max tuples 200000 -> 100000 in threp-killer-loop

Mon Jan 14 13:27:33 EST 2008

cab25: slave boots straight into error screen: "PXE-E35: TFTP read timeout."
Jakob has no idea.

cab35: system restarted on "runIO: gunzip -c /dl/downloader.gz >
/opt/bin/downloader" such that /opt/bin/downloader didn't exist (saw at least
two failed starting attempts, i.e. two reboots later). also, be wary of
anything along these lines (eg missing chmod +x)

the startup delay is too long on this system! takes a long time to spill 1000
chunks into the disk double buffer. consider reusing last session's (fully)
persisted results....

Fri Jan 11 22:16:40 EST 2008

all cabs: apparently, one reason why I'm hearing from so few cabs is that they
are handling so much data that it takes hours for them to prioritize the data
- which, unfortunately in the case of (say) thresh, is a lot of up-front work
(as is sorting in general). As a result, we need to cap the amount of data
allowed into the system to about 200,000 tuples. This should also encourage
shorter experiments.

Fri Jan 11 19:00:47 EST 2008

cab32: screen starts the slave, which wasn't running; eventually boots fine.
jakob informs me cartel.start monitors the slave and should've rebooted it
automatically after some time. STILL WAITING ON WHAT HIS VERDICT IS AS TO
WHAT HAPPENED. suspect buggy script, in which case first thing to do is to
find where it's failing by tracing it into a file. this must be done
systematically for all cabs, so we need to use that cab deployment script.

Fri Jan 11 12:08:35 EST 2008

cab16: screen starts the slave, which wasn't running; eventually boots into
oblivion since /tftproot has nothing

Fri Jan  4 15:44:09 EST 2008

cab32: initdb just doesn't complete running (or doesn't run?)

Mon Jan  7 19:58:28 EST 2008

all cabs: downloader doesn't run. this seems to be due to it not starting up
via ip-up.local.

solution: $HOME is unbound when ip-up.local is run

Tue Jan  8 14:19:42 EST 2008

some cabs don't have icedb running

solution: the exit status reports 138 which means sig 10 (sigusr1) killed it;
combined with the logs (only ctl exists), this suggests there is a race
condition at program startup against sigusr issuer

solution: create a separate pid file from within icedb, once it's ready; point
the issuer at this pid file rather than the wrapper



TESTS AND EXAMPLE RUN-THROUGH

I have provided some convenience scripts, setup.bash, for both the device and
server, which will initialize a database and provide some dummy
data/sources/queries.

svr: ./setup.bash
svr: browse to edit-sources.py, edit any sources
svr: browse to edit-queries.py, edit any queries
cli: ./setup.bash
cli: ./source-executor.py &
cli: ./query-executor.py &
cli: ./get-source-updates.py
cli: ./get-query-updates.py
cli: ./put-results.bash
svr: find the results under data/results/

This is basically what the test scripts do. The starting point is in
tools/test.bash, which is capable of running the tests locally (on the dev
machine) or as a deployment on the Soekris.

DEMO DEPLOYMENT

For the demo I modified the following things on the Soekris:

- scanping.pl
  - invokes put-results.bash instead of the dir_post.py.
  - invokes get-query-updates and get-source-updates

- carteld.pl
  - commented out the packer.py script, since it was slowing things down and
    because the results wouldn't be getting uploaded anyway
  - disabled the usb_upload.py script, since it was slowing things down as
    well
  - added code for starting/stopping the query- and source-executors

TODO

( + means high priority )

- re-start the queries after stopping the carteld daemon
- modify the query executor to run queries in CSV format and write them to
  separate files, instead of appending them together in Python format
  - this should actually (hopefully) simplify things a bit
  - the files should be named based on their "query instance" ID, starting from
    0, so 0.csv, 1.csv, etc.
  - once the files are finished writing to, they should be (instantaneously)
    moved to an upload directory
  - the files should be easily read into a DB on the receiver side.
  - how to integrate the query format with the receiver's expecting table
    format? (this seems hard, should be left up to the user?)
  - UPDATE: it is impossible to select into csv format (won't handle fields
    containing , properly) without first selecting into a temp table and then
    copying that table into a csv; since the temp table requires knowing what
    all the field types are, this is probably not the optimal solution. i'll
    stick with the python formatting.

- change the scripts to use base files from cartel, not from the 'base' subdir
- allow source commands to be Debian packages
+ allow for independent and push sources as well as pull sources (vertical
  accessibility)
- add xsd for descriptors

DONE TODO'S

- 01/23/06 fix the start-stop-daemon script
- 12/12/05 modify the test script to additionally do some uploading

ISSUES

- adapter api
- adapter params
    - pushing parameters down to the lowest level
    - max rates/constraints
- design a streaming db for intermittent connectivity
- adapters should only do the simplest thing - reporting values
- irregular data rates: adaptive?
    - queries over different data srces (different rates)
    - interpolation: on the device (reduce data movement)
    - sql: bad for interpolation
- publish/subscribe model?

- larger system architecture
- mobicom, vldb, sensys
    - mobicom: focus on cafnet



START BUILDING SOME OF THESE

3 modes: queue and push
	push has backward params interface

ds descriptor (xml doc)
	name
	type
	p1...pn (name,type) pairs

event clauses
	even tprocessor

priorities/expirations
	network bandwidth, data source bandwidth
	value-based prioritization: map values to priorities
	think of a real life convincing example scenario for this

virtual adapter interface: interpolation
	how do you merge together multiple data sources?



TODO

[ ] table alteration/preservation
[ ] error reporting
[ ] run-once queries (control messages)
[ ] switch to xml-rpc?
[ ] prioritization for sources
[ ] convert all adapters
  [x] gps is highest priority
  [ ] make a shared lib for pushing to qp
[x] straight-uploading
[x] put the results into the db
[ ] make sure that uploads can succeed over wifi as well
[x] patch up the "edit sources" page (bring it up to date, or make it
    more dynamic :)



CARTEL BUGS

- default gateway missing (or goes away intermittently?)



DOCUMENT BUGS

a lot of my implementation plans were just not right/not thought out well

- still unclear what kinds of queries they're interested in (they don't seem
  to have any ideas)
- idle queries: you should still be able to run queries over data that's
  already there, even if the data source is currently unavailable
- data source discovery: data source presence != data source declared
- data source descriptors: seem to be useless in the presence of the central
  management (otherwise, it's unclear how to 'attach' a descriptor to a
  physical data source, eg gps over serial)

- value-based priortization
- query-based prioritization

ISSUES TO DISCUSS

- joins: challenge is in joining unrelated sources on time. (think of 2 data
  sources with unmatching rates; what would the "correct answer" be for a join
  over them?)
  - aurora has resampling operator
- implement sliding window queries?



REASONS FOR PY2.4

- sorting
- decorators



utcdate, utctime, lat, lon, txt_lat, txt_lon ,quality, satcnt, hdil, alt,
geo_sep, diff_age, diff_id, status, grd_speed_knots, track, mag_var, boot_id




def synchronized(lock):
    """ Synchronization decorator. """

    def wrap(f):
        def newFunction(*args, **kw):
            lock.acquire()
            try:
                return f(*args, **kw)
            finally:
                lock.release()
        return newFunction
    return wrap



RESOURCES

http://www.onlamp.com/lpt/a/6435



CHANGES TO ACCOMODATE/CLEAN UP

sources have priorities too



HOMEWORK

write a document outlining the system












SUGGESTIONS

control queries
work through some examples
continuous queries that send coarse synopses and then progressive scan

(# = priority)

1 come up with a query syntax
	relatd ot the adapter api
		queries? push/pulls? etc
	how to multiplex the different sensors/adapters?
prioritization
	we can have more complex prioritizations, eg:
		don't want things we'v eseen before
		want things when there's a whole bunch grouped together in time
	MODEL BASED COMPRESSION?
	do something simple, and then say "more complex things can be done but we don't go into that here"
	in conjunction with the queries
	simple mode: incremental sending of this thing (resolutions)
- lineage/alteration
- configuration of cars
push down into adapters/adapter conifg
	decomposition of queries into things that are sent ot cars and things that remain on the server
		this could work for 
	portal interface: how to present the subset that we do have?
3 query result caching: when to expire?
scenarios: it's a good idea to drive this paper by the things that we ourselves want out of the system
	summarization
	send me unsent data (1/8, 1/4, 1/2, ...): resampling
	2 acks to mark the data as received (ties in with 1)
		also deals with open issue: when to expire data?
		can also use them for acking source updates
		space management issue: if out of space, start overwriting tail
apps run on top of vanilla db; they're responsible for alteration etc


PAPER FOCUS POTENTIALS

motivation: centralized interface for data management on intermittently connected distributed sensors



NEXT STEP

discuss the above issues of interest some more and do write-ups




SCRATCH

ssh root@diversity1 (sk passwd)
ifconfig wlan0 up
iwconfig ath0 essid cartelfoo

main loop, testap_detection, home_detection
	attempt_connection
		upload




CHANGELOG 2006/2/15

updated the deployment script to work with the full cartel system
separated deployment from setup - they're two separate things now, each able to run by itself
updated the main test script to display more useful information
fixed some default-value bugs in the main test script
exporting some more variables to the client test script
abstracted netcat and send_command for bash scripts
added testSshProfile and logging-related variables to the config module
changed the source_updates schema (and all related handling code throughout the system) to use operation codes instead of a boolean (specifying whether the operation was an add)
	updated the testdata as well
integrated with the python standard logging facility
	transitioning toward its more powerful semantics
	logging to multiple destinations, including local files and socket
	TODO: layer a more advanced logging facility on top (to support sending back to server); maybe store log records in the database as well?
	added a server-side app to listen for log records
fixed bugs in the thread interruption mechanisms
fixed bug concerning allow_reuse_address (class var, not instance)
enhanced the socketserver inheritors to support interruption (via 1-hz polling) (TODO: make this more complete?)
made various enhancements to the general lib
fixed up the signal handling code
BUG: when the terminal detaches from the logging facility's stdout logger, thread gets stalled; see http://sourceforge.net/tracker/index.php?func=detail&aid=1431253&group_id=5470&atid=105470
added some output redirection enhancements to the daemonizer bash script
enhanced the daemonizer's killer to retry several times for 5 seconds before force-killing
changed the put-results scripts to simply issue a command to the daemon instead of attempting to do any work itself
moved all the put-result code to the daemon, for reasons of synchronization/safety (probably fixed the synch-bug); modified various parts of the system to support the synchronization
tweaked the exception hierarchy
started adding a query job manager class in order to undo all the messy classmethods
now raises orig exception instead of getsourceupdatesexception; TODO: this should be a temporary fix
added the ability to resume sources and queries when the daemon is (re)-started!
made various enhancements to the client test script
added a second client test that works with the full cartel framework
fixed any localhost/127.0.0.1 issues
started adding both UI for and handling of alter operations; TODO: complete this
added a bunch of sandbox scripts to test logging
modified the carteld files (carteld, scanping, gps2db) to work with QP




IMPORTANT INSTALLED PACKAGES

python2.4
python2.4-pygresql
netcat
python2.4-psyco?

for simulation:
    pynum
    gnuplot-py

CPAN MODULES

Error?



LOG

scanping wasn't starting because scanping_config.pl didn't exist
reinstalled packages
getting the upload triggering to work



ISSUES

>>>
00:00:24:C4:80:40:~# cpan Error
CPAN: Storable loaded ok
LWP not available
CPAN: Net::FTP loaded ok
Fetching with Net::FTP:
  ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz
Going to read /root/.cpan/sources/authors/01mailrc.txt.gz
LWP not available
Fetching with Net::FTP:
  ftp://ftp.perl.org/pub/CPAN/modules/02packages.details.txt.gz
Going to read /root/.cpan/sources/modules/02packages.details.txt.gz
  Database was generated on Mon, 20 Feb 2006 20:18:42 GMT
  HTTP::Date not available

  There's a new CPAN.pm version (v1.86) available!
  [Current version is v1.7601]
  You might want to try
    install Bundle::CPAN
    reload cpan
  without quitting the current session. It should be a seamless upgrade
  while we are running...

LWP not available
Fetching with Net::FTP:
  ftp://ftp.perl.org/pub/CPAN/modules/03modlist.data.gz
Going to read /root/.cpan/sources/modules/03modlist.data.gz
Going to write /root/.cpan/Metadata
Running install for module Error
Running make for U/UA/UARUN/Error-0.15.tar.gz
LWP not available
Fetching with Net::FTP:
  ftp://ftp.perl.org/pub/CPAN/authors/id/U/UA/UARUN/Error-0.15.tar.gz
CPAN: Digest::MD5 loaded ok
LWP not available
Fetching with Net::FTP:
  ftp://ftp.perl.org/pub/CPAN/authors/id/U/UA/UARUN/CHECKSUMS
Checksum for /root/.cpan/sources/authors/id/U/UA/UARUN/Error-0.15.tar.gz ok
Scanning cache /root/.cpan/build for sizes
Error-0.15/
Error-0.15/Error.pm
Error-0.15/ChangeLog
Error-0.15/Error.ppd
Error-0.15/MANIFEST
Error-0.15/Makefile.PL
Error-0.15/README
Error-0.15/example
Error-0.15/t/
Error-0.15/t/02order.t
Error-0.15/t/01throw.t

  CPAN.pm: Going to build U/UA/UARUN/Error-0.15.tar.gz

Checking if your kit is complete...
Looks good
Writing Makefile for Error
    -- NOT OK
Running make test
  Can't test without successful make
Running make install
  make had returned bad status, install seems impossible
>>>











Hi Sam/Hari,

Here are my immediate plans. Please let me know if anything sounds out of sync.

Thanks,

Yang

IMMEDIATE GOALS
- generate random APs from the cdf
- add experiments to simulation
  - multiple "simultaneous" traces
    - NOTE: the following will *not* optimize the global utility because we are
      optimizing for individual query points (we are sending all the points
      within one grid before advancing to the next grid)
    - just use similar traces (ones with the same start/end points)
    - all traces start at random times within an hour
    - establish grids on the map - these should be spaced out at every 30m or
      so along the path (parameterize this)
    - as cars connect, they sync with server (sending their avail grid points,
      recving ordering of those). (should i bother calculating/including the
      overhead here?)
    - the server orders grids based on a score trading off among the following
      data:
      - last time we got data for the grid (how stale the data is): the staler,
        the more we want it
      - # cars with data for this grid (how available the data is): the more
        rare, the more we want it
      - variance in the data we got for the grid (how "interesting" the data
        is): the more interesting, the more we want it
      - how much data we have for this grid (how complete the data is): the
        more incomplete, the more we want it
    - the car runs:
      ===
      for each grid (in the server-specified order):
        send all points for that grid using whatever delivery-order-by scheme
          we are experimenting with (fifo, random, smart)
      ===
    - this illustrates the "hierarchy": two layers, we have grids, and we have
      points within grids















3/6 mtg with sam

TODO
- weigh grids by proximity to query point
- import many more traces
- get speeds from OBD data

DISCUSSION ITEMS
- what are we interested in?
    - so aside from the way devices choose to upload data, we can also choose
      how to let the central server reply to the devices.
    - how much should we weigh what the user is interested in vs. what we're
      collecting? ie what's the balance between:
        - auto-determining query points? eg, based on variance, data etc
            - this is better if our goal is to get complete information. the
              grand plan would be to collect the lowest-resolution data, get
              more resolution about any particular part of that data for
              which we feel more information would be needed, and repeat ad
              infinitum. eg, if we have a few data points about two locations,
              one with high variance, our information about the high-variance
              location is less complete.
          or...
        - letting the user determine what the query points are?
            - this is better for more interactive approaches
- expressivity
    - can imagine that the user wants to use different upload schemes for
      different types of data (different data sources)
      - what's the easiest way (think: declarative lang) that lets the
        user specify how to prioritize the data?
      - any generalizations we can extrapolate from the following schemes?
        - fifo
        - random
        - bisect
          - feedback
            - limited feedback
    - what about server side? can we generalize from following attribs?
      - last time we got data for the grid (how stale the data is): the
        staler, the more we want it
      - # cars with data for this grid (how available the data is): the more
        rare, the more we want it
      - variance in the data we got for the grid (how "interesting" the data
        is): the more interesting, the more we want it
      - how much data we have for this grid (how complete the data is): the
        more incomplete, the more we want it
- how to present these results?
    - we're comparing not only the way devices upload data, but also the
      scheme the central server uses for helping devices prioritize data

notes to self
- TODO: there is a soft bug in the upload-limiting. we're going to exhaust the
  partition by first uploading all the data for one subtrace within that grid.
  ie let's say we have 2 subtraces in a grid. then whichever one the car
  drives through first will have all of its points uploaded first, before the
  algo proceeds to the second subtrace.








-- verify that the speed data makes sense for some randomly chosen boot(s)
select round(cast(lat as numeric),5), round(cast(lon as numeric),5), grd_speed_knots from c_gpslog where boot_id = 3;

-- find the variance for each grid



experiments
	1 car/trace, uniform dist
		vary data size
	1 car/trace, hotspot dist
	multi cars/traces, hotspots
=======


















http://carteldb.csail.mit.edu/sim/view.py?traces=2546&modes=bisect+limit-1-bisect-cut+&data_size=50000&query_point_count=5&conn_point_count=10&image_dims=500+500&start_range=600&do_hot_spots=f&grid_size=.001&seed=0&run=true









ORIGINAL CONTENTS OF svr/testdata.psql

begin;

-- TODO change this to use the ICEDB interface instead of directly inserting testdata.psql

-- TODO change the pull example to something useful (eg top), remove these queries
-- TODO add back in the id's
--insert into QUERIES (string)
--values ('select * from wifi where rssi is not null rate 2 priority 5');
--
--insert into QUERIES (string, is_removed)
--values ('select time, speed from obd rate .25 priority 15', true);
--
--insert into QUERIES (string)
--values ('select * from gpslog rate 1 priority 10');
--
--insert into QUERY_UPDATES (is_add, query_id)
--values (true, 1);
--
--insert into QUERY_UPDATES (is_add, query_id)
--values (true, 2);
--
--insert into QUERY_UPDATES (is_add, query_id)
--values (true, 3);
--
--insert into QUERY_UPDATES (is_add, query_id)
--values (false, 2);

insert into SOURCES (name, type, attributes, query, command, rate, is_removed)
values ('obd',
        0,
        'speed float8, temperature float8',
        '',
        'obd.bash',
        5,
        true);

insert into SOURCES (name, type, attributes, query, command, rate)
values ('gpslog',
        1,
        -- 'lat float8, long float8',
        'utcdate          DATE,
        utctime           TIME,
        lat               DOUBLE PRECISION,
        lon               DOUBLE PRECISION,
        txt_lat           TEXT,
        txt_lon           TEXT,
        quality           SMALLINT,
        satcnt            SMALLINT,
        hdil              DOUBLE PRECISION,
        alt               DOUBLE PRECISION,
        geo_sep           DOUBLE PRECISION,
        diff_age          DOUBLE PRECISION,
        diff_id           INT,
        status            BOOL,
        grd_speed_knots   DOUBLE PRECISION,
        track             DOUBLE PRECISION,
        mag_var           DOUBLE PRECISION,
        boot_id           INT',
        'every 5',
        null,
        null);

insert into SOURCES (name, type, attributes, query, command, rate)
values ('wifi',
        1,
        'ssid varchar(256), rssi float8',
        'every 5',
        null,
        null);

insert into SOURCES (name, type, attributes, query, command, rate)
values ('top',
        0,
        'pid int, usr varchar(256), ni int, virt int, res int, shr int, pcpu real, pmem real, cmd varchar(256)',
        '',
        'top.bash',
        0.3);

--insert into SOURCES (name, type, attributes, command, rate)
--values ('top',
--        0,
--        'pid int,
--        user text,
--        pr int,
--        virt int,
--        res int,
--        shr int,
--        cpu float,
--        mem float,
--        time+ float,
--        cmd text',
--        'top.bash',
--        0.3);

insert into SOURCE_UPDATES (operation, source_id)
values (0, 1);

insert into SOURCE_UPDATES (operation, source_id)
values (0, 2);

insert into SOURCE_UPDATES (operation, source_id)
values (1, 1);

insert into SOURCE_UPDATES (operation, source_id)
values (0, 3);

insert into SOURCE_UPDATES (operation, source_id)
values (0, 4);

-- TODO add this test in
--insert into SOURCE_UPDATES (operation, source_id)
--values (2, 4); -- modify

-- TODO automate the usage of the edit-sources page so that all of the data in this file is inserted the 'normal' way

create table wifi (
        device_id varchar(64) not null,
        time_received timestamp not null,
        rec_id int4 unique not null,
        time timestamp not null,
        ssid varchar(256),
        rssi float8 );

create table gpslog (
        device_id varchar(64) not null,
        time_received timestamp not null,
        rec_id int4 unique not null,
        time timestamp not null,
        utcdate          DATE,
        utctime           TIME,
        lat               DOUBLE PRECISION,
        lon               DOUBLE PRECISION,
        txt_lat           TEXT,
        txt_lon           TEXT,
        quality           SMALLINT,
        satcnt            SMALLINT,
        hdil              DOUBLE PRECISION,
        alt               DOUBLE PRECISION,
        geo_sep           DOUBLE PRECISION,
        diff_age          DOUBLE PRECISION,
        diff_id           INT,
        status            BOOL,
        grd_speed_knots   DOUBLE PRECISION,
        track             DOUBLE PRECISION,
        mag_var           DOUBLE PRECISION,
        boot_id           INT);

create table top (
        device_id varchar(64) not null,
        time_received timestamp not null,
        rec_id int4 unique not null,
        time timestamp not null,
        pid int, usr varchar(256), ni int, virt int, res int, shr int, pcpu real, pmem real, cmd varchar(256)
        );

commit;



CLASS HIERARCHY
cnc (CafNetClient)
  xport (ObjectTransport)
query (QueryJob)
prioritizer (Prioritizer)
  fifo (FifoPrioritizer)
  bisect (BisectPrioritizer)
Seg (Segment)

query.run():
  rows = self.db.query()
  self.prioritizer.buffer(rows): (sync newbuf)
    self.newbuf += rows

prioritizer.prioritize():
  self.splice(): (sync buffer, newbuf)
    self.buffer += newbuf
    self.newbuf = []
    self.yield_count += 1
    IMPLEMENTING
    for rowid in self.rowids
      xport.cancel(objid = rowid): (sync)
        if objid in self.objid2msgs
          for msg in self.objid2msgs[objid]
            del self.msg2objid[msg]
            self.cnc.cancel(msg): (sync)
              hash = self.msg2hash[msg]
              del self.msg2hash[msg]
              del self.hash2msg[hash]
              self.xrc.cancel(hash)
          del self.objid2msgs[objid]
          del self.objid2callback[objid]
    count = 0
  # XXX how does garbage collection for generators?
  self.yielder = self.start_yielder()
  self.start_stream(yielder):
    first_rows = self.start_stream_1(yielder): (sync buffer)
      while self.count < thresh
        index = yielder.next()
        row = self.buffer[index]
        first_rows.append()
        self.count += 1
    first_rowids = self.start_stream_2(first_rows):
      first_rowids = []
      for row in first_rows
        rowid = self.xport.send(obj = row, callback = self.gotmsg)
        first_rowids.append(rowid)
    self.start_stream_3(first_rowids): (sync buffer)
      for rowid in first_rowids
        self.rowids.add(rowid)

fifo.start_yielder():
  yielder_count = self.yielder_count
  for i in xrange(len(buffer))
    yield i
    if self.yielder_count > yielder_count
      break

bisect.init():
  self.segs = heap()

bisect.start_yielder():
  yielder_count = self.yielder_count
  if len(self.segs) == 0
    for i,seg in enumerate(self.segs)
      if seg is last_seg:
        seg = Seg(seg.start,len(buffer)-1)
        self.segs[i] = last_seg # reinsert
  else
    seg = Seg(0,len(buffer)-1)
    self.segs.push(seg)
  for seg in self.segs
    yield seg.mid
    if self.yielder_count > yielder_count
      break
    left, right = seg.split()
    self.segs.push(left)
    self.segs.push(right)

IMPLEMENTING
xport.send(obj, callback = None): (sync)
  msgs = self.serialize(obj)
  for msg in msgs
    self.msg2objid[msg] = objid
    self.objid2msgs[objid] = msg
    self.objid2callback[objid] = callback
    self.cnc.sched(msg): (sync)
      hash = self.xrc.hash(msg)
      self.msg2hash[msg] = hash
      self.hash2msg[hash] = msg
      self.xrc.sched(msg, hash)
  self.objid += 1
  return self.objid - 1

xport.getmsg(hash): (sync)
  msg = self.cnc.getmsg(hash): (sync)
    try
      msg = self.hash2msg[hash]
      del self.hash2msg[hash]
      del self.msg2hash[msg]
    except KeyError
      msg = 'already sent'
    return msg
  if msg != 'already sent'
    objid = self.msg2objid[msg]
    callback = self.objid2callback[msg]
    callback():
      self.gotmsg(objid): (sync buffer)
        index = self.rowids[objid]
        del self.rowids(objid)
        self.buffer[index] = None
        try
          index = self.yielder.next()
          row = self.buffer[index]
          self.xport.send(row, callback = self.gotmsg)
        except StopIteration
          pass
    del self.msg2objid[msg]
    try del self.objid2msgs[objid]
    try del self.objid2callback[objid]
  return msg










































CAFNET PSEUDO CODE OUTLINE

toSend :: LinkedHashTable(String: Meta)
pendingSends :: ArrayList(PendingSend)
	maps CTL's to PendingSend objects (only one per CTL allowed at any time)
	think of as hash table
	(check this concept; not sure)

ctl.sched(msg):
	sync toSend
		wasEmpty = toSend is empty
		toSend[hash] = msg
		sorted.add(msg)
		if wasEmpty
			thread cnl.rts(size)
cnl.rts(size):
	sync pendingSends
		pendingSends[ctl].size = size
	clearPending():
		sync bytesAvail
			sync pendingSends
				# TODO pendingPointer never actually gets modified; what's the point of this?
				# TODO seems like you meant to make a round robin mechanism
				...
				some funky loop on pendingSends
					if pendingSend.size <= bytesAvail
						pendingSends.remove(pendingPointer)
						thread ctl.cts(size = pendingSend.size)
ctl.cts(size): # TODO size unused
	sync toSend
		msg = sorted.pop()
		del toSend[msg.hash]
		if msg != null
			if msg is end2end
				sync unacked
					unacked[msg.hash] = msg
			cnl.send(msg):
				if msg.dst != null
					set msg.src
					adjust msg.dst
					sync msgQueue
						prev = msgQueue[msg.id]
						msgQueue[msg.id] = msg
						sorted.add(msg)
						if prev == null
							sync bytesAvail
								bytesAvail -= msg.size
					sendAll():
						sync actionInProgress
							if actionInProgress
								actionNeeded = true
							else
								actionInProgress = true
								actionNeeded = false
								thread sendAllThread()
			nextmsg = sorted.peek()
			if nextmsg == null
				thread cnl.rts(size = nextmsg.size)
sendAllThread():
	sync neighbors
		for neighborEntry in neighbors.entries
			mal = neighborEntry.key
			malNeighbors = neighborEntry.value
			for node in malNeighbors:
				if node.iface != null
					sync msgQueue
						for msg in sorted
							if msg.dst == node.remoteCNL
								succeeded = mal.send(dst = node.iface, msg = msg):
									if msg == null
										return false
									try
										node = dst.addr
										if node == null or node.connection == null
											return false
										out = RateLimitedOutputStream(node.conn.outstream, bytes)
										sync out
											out.write(msg):
												# TODO this is a horrible, wasteful implementation
											out.flush()
										return true
									catch
										return false
								if succeeded
									sorted.remove(msg)
									del msgQueue[msg.id]
									sync bytesAvail
										bytesAvail += msg.size
								else
									malNeighbors.remove(node)



xport.send_lock:
  objid
  msg2objid
  objid2msgs

cnc.send_lock:
  
  

schedule(row):
  rowid = xport.send(obj = row):
    msgs = self.serialize(obj)
    sync self.send_lock
      for msg in msgs
        self.add_outstanding_msg(msg):
          self.msg2objid[msg] = self.objid
          self.objid2msgs[self.objid] = set()
          self.objid2msgs[self.objid].add(msg)
        self.cnc.sched(msg):
          hash = self.xrc.hash(msg)
          self.add_outstanding(hash, msg):
            sync self.send_lock
            self.msg2hash[msg] = hash
            self.hash2msg[hash] = msg
          self.xrc.sched(msg, hash)
      self.objid += 1
      return self.objid - 1
  sync rowids
    self.rowids.add(rowid)
remove(query):
  query_jobs[query].stop():
    sync rowids
      for rowid in self.rowids
        xport.cancel(objid = rowid):
          sync self.send_lock
            if objid in self.objid2msgs
              for msg in self.objid2msgs[objid]
                del self.msg2objid[msg]
                self.cnc.cancel(msg):
                  self.remove_hash(msg)
                    sync self.send_lock
                      hash = self.msg2hash[msg]
                      del self.msg2hash[msg]
                      del self.hash2msg[hash]
                  self.xrc.cancel(hash)
              del self.objid2msgs[objid]
  del query_jobs[query]




sudo minicom -o



rate limit 1000

priority 50

select lat, lon, essid, bssid from adapter_gpswifi

select lat, lon, essid, bssid from adapter_gpswifi where lat <
42.45183466943919 and lat > 42.39810802339276 and lon > -71.158447265625 and
lon < -71.07330322265625



[x] drop rate limiting
[ ] cancel_all
[ ] delete all queries on startup





PROBLEMS WHEN SETTING UP ON CARTEL12 (A NEW BOX)

- root passwd suposed to be "1d00" but it didn't work, vlad had to set to
  "test"
- icedb passwd unknown
- icedb's home dir missing
- /opt/cartel not owned by icedb (only /opt/cartel/icedb)
- old processes (scanping, heartbeat, etc) were all running
  - had to be killed; /etc/init.d/cartel stop didn't work
  - (make sure scanping (and packer) are commented out from carteld.pl)
- some system-wide postgres instance was running (/etc/init.d/postgresql stop)
- localhost is removed from /etc/hosts (maybe others are removed too, not sure)
- cannot resolve domains of other csail machines after taking down scanping
- carteld.pl starts icedb right away
- PATH doesn't include:
  - /opt/cartel/bin/
    - pg_ctl
    - psql
    - java
- icedb's default shell is sh, not bash (had to chsh)



TASKS (FIRST TO LAST)

- get current version going on soekris!
  X update findutils on golf
  X set up following packages:
    X assorted
    X twisted
    X zope
    X commons
    X path
  X test cafnet on golf
  X write pull deployment script for assorted
  X write push deployment script for assorted
  X write push deployment script for twisted + zope + cafnet + commons + path
  X test deployment scripts
  X learn soekris setup (nfs)
  X update findutils on soekris
  X copy to soekris
  X test cafnet on soekris

- integrate cafnet and icedb
  - verify synchronous semantics (icedb doesn't use twisted)
  X update ObjectTransport to interface with the new cafnet
  X set up new pg db (old one got blown away by fc5 up2date)
  - update configs to target new (flattened) /opt tree
  X update tests/setups
  - test tests and icedb
- deploy icedb on soekris
  - install pg, pygresql on golf
  - discover other dependencies; resolve them
  - deploy to golf
  - set up pg db on golf
  - update deployment tools to target new (flattened) /opt tree
  - install pg, pygresql and other deps on soekris
  - deploy to soekris
  - run tests

- miscellany
  - set up icedb requirements on golf
    - maintain/update documentation, instructions, requirements, etc
  - test icedb on golf
    - fix deployment scripts
  - set up icedb requirements on cartel-gentoo
  - test icedb on cartel-gentoo
  - add dependencies (on path/others) to commons' pypi
- clean up/reorganize the code
  - factor out commons
  - release commons
  - migrate to more sane file hierarchy
    - ultimately group bin's together
    - move out data, config, etc
  - write scripts to prettify code
  - write scripts to handle modelines
  - remove unused config vars
  - package up cafnet
  - TODO add more here
