Re: [sup-devel] after second heliotrope import, new messages not appearing

So, it appears heliotrope tells you your mails are indexed, yet they don't appear. All I see is `MetaIndex#add_message` returning in the test that checks if the mail is already in the store. Problem is, the exact same check is already done before that moment, and if the mail really was already present, it should be marked as "seen" and not "indexed".

Another possibility could be `MetaIndex#gen_new_docid` which, for an unknown reason, returns a wrong value, so it would always overwrite an old message.

I'm not really good at debugging efficiently, but I would re-launch the import after having placed a few "puts" in this `add_message` method, see if the existence test returns true or not. If not, print the docid, and see what the message with that docid is. If the message with that docid is the one you just imported, then there is a problem with how we generate them (because it was already given to another message).

On Tue, Jan 1, 2013 at 7:33 PM, Hamish D <dmishd@gmail.com> wrote:

> Your problem may have multiple origins, and I have no idea where you could
> start :
>
> - Did you use the correct directories for messages and index for _both_ the
> import and the server ? (sorry if this sounds stupid)

Always good to check the basics, but yes, I do use the same directory.
I have scripts for the import and the server and both use the -d
argument to ensure they use the same directory.

> - Did the index size in heliotrope change ? (as given by a GET to
> /size.json)

No - it remained the same. (And is about half of the count that sup reports).

> - Did the messages file size change ?

It did on one import, but not the most recent.

> - Are the messages absent in inbox only or from the whole turnsole ?

All searches that I do stop in May.

> - As a last resort, could you send the mbox file so that I can try this at
> home, if your mails aren't too personal ?

It's my work email, with a total of over 100,000 messages. So I'd
prefer not to send it over.

Any logs that might be worth looking at? While doing the import, I do
get various lines along the line of:

; scanned 71, indexed 71, skipped 0 bad and 0 seen messages in
22.1s = 3.2 m/s

I do use the state argument (-t) to save reimporting old mail. I have
also tried doing a reindex (as I import from multiple mboxes) but to
no avail.

I could just try another import of the whole lot into a clean directory I guess.

I'm also happy to type stuff into heliotrope-console if that will help
diagnose stuff.

Thanks for your suggestions so far
Hamish

> On Thu, Dec 27, 2012 at 5:19 PM, Hamish D <dmishd@gmail.com> wrote:
>>
>> Hello
>>
>> I've just been trying to carry on from where I left off with migrating
>> my work email into heliotrope (which I last worked on in May).
>>
>> So I've rsynced the mbox files across to where I will run heliotrope
>> and tried running heliotrope-import to put all the new messages in the
>> index. It appeared to work properly, reporting that messages were
>> being added to the index and the index files themselves have had their
>> timestamps updated. But when I start turnsole, I can't find any email
>> after May.
>>
>> Any ideas on where to start looking to work out what is going on?
>>
>> Hamish
_______________________________________________
Sup-devel mailing list
Sup-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-devel

--
Matthieu RAKOTOJAONA