various bugfixes and improvments

1. only generate tokens for valid utf8
1. fix the count in shard.ReadState.Total
1. properly order dupi.Query.Next instead
of only round-robin.

update tutorial

fix problems with incremental indexing

there were several things whose state was not properly
re-established in subsequent runs of appending to an index:
- the last created docid in dmd/adder
- the current doc in internal/shard/poster

this commit keeps track of them and also ensures
that shatterers run with the right docid.

readme-status

readme-status

use stats to determine how much to extract

killer app command for the moment

dupi index <stuff>
dupi extract | awk '{print $1}' | xargs dupi unblot

added -all to unblot

added inspect verb

fix monotonicity of doc ids

the concurrent shatter had left open the
possibility of having non-monotonically increasing
document ids.  Fix with a sync.Cond

make dupenron work with unblot

fix imports, doh.