Ipqbdb concurrent access analysis

Ipqbdb uses the Concurrent Data Store (CDB) model, that supports 
deadlock-free, multiple-readers/single-writer semantics, not record or 
region locking. Updates occur in two ways:
*   non-locked: as read-change-write cycles, which can overlap, and
*   locked: as lock-read-change-write, using write cursors.

In the non-locked mode, multiple programs read a given record and 
concurrently write an update. When this happens, internal bdb locking 
ensures that the relevant .db file will not be corrupted. However, all 
updates except the last one will be "lost". How does this affect the 
program's logic?

Here we analyze the possible outcome of concurrent writes, limiting to 
two programs making concurrent writes, on a case by case basis.


block.db
========

The block.db table is written by ibd-ban and ibd-parse using locked 
mode. The global lock ensures they don't interfere with one another. 
However, ibd-judge also writes updated records, and it currently uses 
non-locked mode. In addition, ibd-del may delete those records. That
can result in some data loss, which we deem acceptable.  There are nine
cases:

1. Two instances of ibd-ban or ibd-parse write the same record.
   The second one will have read the data written by the first, thus
    correctly counting it how many times it has been caught.

2. Two instances of ibd-judge (two queues) write the same record.
   In non-locked mode, the second can overlap, so that the number of 
   packets blocked (block_cnt) may be not accurate. However, the 
   rehabilitate function works correctly.

3. Two instances of ibd-del delete the same record.  The second one
   will fail (and report the error).

4. One of ibd-ban or ibd-parse writes a record before ibd-judge updates 
   it. ibd-judge data can be older than the update, and the verdict is 
   issued before the record is written by ibd-ban or ibd-parse. 
   However, if the record is updated by ibd-judge, then the result of 
   having caught the IP is lost. This state of affairs is helped by the 
   fact that ibd-judge only updates records in two circumstances:

    4.1 UPDATE_TICK_TIME (default 20) seconds have elapsed since the
        previous time the record has been updated, and rehabilitation 
        occurs. It is a nuisance to allow rehabilitation instead of 
        doubling the blocking probability, but the rather large 
        UPDATE_TICK_TIME makes for this circumstance being quite 
        fortuitous.

    4.2 The packet is being blocked, hence it has a relatively high
        blocking probability already. If packets are blocked, they
        cannot be caught.

   To resolve this issue it could possible to acquire a lock when the 
   packet is retrieved from the queue by ibd-judge, then read it, 
   possibly update it and release the lock. The performance degradation 
   implied by such behavior would have to be evaluated.

5. ibd-del decides a record is old enough to deserve deletion, given
   the need to free disk space.  Then one of ibd-ban or ibd-parse 
   updates the record just before ibd-del deletes it.  The update is 
   lost.

6. ibd-judge updates a record before one of ibd-ban or ibd-parse writes
   it.  No issue, the latter two ones always read the updated data.

7. ibd-judge updates a record before ibd-del deletes it.

8. ibd-del deletes a record before ibd-ban or ibd-parse writes it.

9. ibd-del deletes a record before ibd-judge updates it.


descr.db
========
This table is being accessed safely.

white.db
========
This is almost a read-only db. You may want to set it up with a handful of
particular IP addresses and never mind about it again. ibd-white does not
do any locking and automated updates to this table may result in unreliable
settings.


