Re: [sup-devel] Invalid meta data error in OklahomaMixer

[Horacio Sanson <hsanson@gmail.com>]

I managed to get the store.tch working again by installing
tokyocabinet-bin and running this command on the file:

  tchmgr optimize -nl store.tch   # without -nl nothing works...

at the end of this command I got a write error but the file got back
to a usable state. I am afraid this command simply truncated the file
causing some missing emails but not really sure. Is there a way to
check if the index and the messages are in sync? That is all indexed
messages have a corresponding email message in the store.tch?

Here is a good blog post with tips for using Tokyo Cabinet. It
recommends enabling logging and running optimize from time to time.
Logging would allow us to see what caused the metadata corruption:

  http://www.bigdbahead.com/?p=700

Some posts of people with the same invalid metadata problem:

  http://twitter.com/#!/ono_matope/statuses/17650765308
  http://groups.google.com/group/tokyocabinet-users/browse_thread/thread/f114f078848e20ed

Tokyo Cabinet is pretty good and is actually used for the largest
social network in Japan
(http://alpha.mixi.co.jp/blog/?s=tchdb&paged=2) so I would not change
it. Even big databases (e.g. MySQL, Postgres and Oracle) are not free
of table corruption. The difference is that they provide tools to
detect and recover from such corruption state.

regards,
Horacio

On Tue, Jun 14, 2011 at 12:30 PM, William Morgan
<wmorgan-sup@masanjin.net> wrote:
> Reformatted excerpts from Horacio Sanson's message of 2011-06-14:
>> Not even close... my store.tch is 76MB only. This metadata corruption
>> in Tokyo Cabinet seems to be a common occurrence and what is scary
>> about this is that there seems to be no way to recover from it (as far
>> as google can tell). The only option I see is delete my store/index
>> and restart sync again.
>
> Can you point me to some links about this type of corruption?
>
> I'm far from wedded to Oklahoma Mixer / Tokyo Cabinet. It was just the
> easiest thing to get started with. Any kind of mutable persistent
> key-value store will do. Other alternatives to consider: another
> TokyoCabinet gem besides OH, KyotoCabinet, BDB... I kinda don't want to
> move to separate processes like Redis though. If only there were a
> libredis (well there is, but it's not what you want.)
> --
> William <wmorgan-sup@masanjin.net>
> _______________________________________________
> Sup-devel mailing list
> Sup-devel@rubyforge.org
> http://rubyforge.org/mailman/listinfo/sup-devel
>
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel