[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [sup-devel] Experimental Gmail Source



Hello all,

I am still trying to implement sync-back functionality to my GMail source but not going anywhere.
The problem I have is that the Sup index keeps the messages ids provided by RMail (e.g. 20cf301af801a2aa8b04dc6e9931@google.com) and not the id I get from the source that is the X-MSG-ID provided by Gmail (e.g. 1434541737393941768). I need a way to query the index using the Gmail provided id and get a message back or at least it's labels so I can compare and update them on the server side if required.

This is what I have found so far:

  - There is no way to query the index for a message using the source (Gmail) provided id.
  - The only places I can see where the source message ids are stored is in a locations array
     that keeps the source own id and the source message id.
  - Sup index provides methods to query messages by id (build_message, contains_id?, etc)
    but the id they accept is the message id provided my RMail, not the id of the source.

Questions:

  - What is the purpose of the locations array? To allow a single message to exists on
    multiple sources?
  - Is there an easy way to query the index to return a message/messages using the 
    source provided id?
  - Is a good idea to add a term to the index that keeps the source id along with the message id?

regards,
Horacio




On Sun, May 12, 2013 at 3:18 AM, Horacio Sanson <hsanson@gmail.com> wrote:



On Sat, May 11, 2013 at 7:15 PM, Gaute Hope <eg@gaute.vetsj.com> wrote:


On 09. mai 2013 11:28, Horacio Sanson wrote:
> I am trying to implement a new source for Gmail accounts. This is
> copied from my efforts to do the same in Heliotrope.
>
> Here is an experimental implementation that can read the email from
> Gmail and add it to the Sup index:
>
> https://github.com/hsanson/sup/tree/gmail_source
>
> To use:
>
> - Install leveldb gem "sudo gem install leveldb-ruby" - Add a gmail
> source:   sup-add gmail://username@gmail.com - Start sup and see how
> it syncs your emails.
>
> Warnings:
>
> - This is experimental - This always syncs only the All mailbox so
> make sure to use an account with not too many emails for testing. -
> All email data and headers are stored in a LevelDB database at:
> ~/.sup/gmail/account
>

> - For some reason I get duplicate "Inbox" and "Sent" labels in the
> list of labels and I am not sure why.
>  - I still have no clue on how  to handle sync-back. That is how to
propagate
> changes made in sup
> back to Gmail. Any tips on how the maildir source does it would be
> appreciated.

Hi Horacio,

nice work. Working directly with GMail labels is probably a good idea
(the other option is to move messages between IMAP folders). I have a
design question though:

Should remote sources be part of regular sup? Or should rather the
fetching and syncing be put in a separate script which creates a LevelDB
setup like you have it with a Gmail source in sup working directly on it?


I don't like the current two step sync setup that Sup uses now (IMAP -> Maildir -> Sup). It requires external programs (offlineimap) and synchronization is one way only. I know about the sync-back branch but AFAIK it is still limited to flags only. Also Maildir is an old storage format that doesn't work well with current email workflows. For example I have tons of duplicate emails on several folders due to the use of mailboxes rather than labels.

I would prefer Sup to take care of the mail storage/indexing (as heliotrope does) and the sources be in charge of syncing the Sup storage/index with the remote servers. I understand that this can be difficult due to the difference in paradigm between IMAP/POP and Sup but GMail offers extensions that map directly to Sup workflow. GMail gives each email a unique 64bit ascending indentifier to all mail messages and adds labels them. I don't even need the LevelDB database as I could easily implement the source so it fetches the mail headers and body from the Gmail server directly when requested. I only added the LevelDB storage as a cache to speed up the message lookups and for offline use. 
 
Recall that the IMAP source was removed in 52e29ba [1] (discussion
probably on the mailinglist somewhere).


Yes I know this and reading the commits and source code of this source I can tell that William hated the IMAP protocol. This is understandable as I myself have dealt with this protocol and know first hand how broken it is. But again with the extensions supported by GMail servers the implementation is far easier to do.
 
I like this approach for GMail, but I would like to see it for regular
IMAP sources as well with folders as labels.. I briefly experimented
with a maildir-root folder approach [2] which treats all underlying
maildirs as sources which correspond to a label (do not use,
incomplete). It of course presents a plethora of questions on how to
sync messages between labels, but implementing it is is probably
relatively straight forward.

 
IMAP is a horrible protocol and implementing it requires herculean effort. Still once the Gmail source is finished it can become a starting point for a more complete IMAP source. 
 
> Help:
>
> - How do I stop the source poll when I quit sup? If I have a large
> amount of emails when polling is running and I quit sup the process
> hangs there.

This normally runs in a separate thread, I don't think maildir really
stops the polling - so I sometimes get an error if I quit sup while the
polling is running and various stuff just disappears underneath the poller.


I see that sup simply kills the threads... would be better if each source had a stop method that Sup could invoke to stop the polling. I will try to look into this issue as it is problematic for remote sources with large amounts of emails.

I don't have the chance to get into your other questions at the moment.


Thanks for answering my inquiries. If you have a chance I would really like to know how to get the labels for a specific email from the index. I need this to implement the sync-back part of the Gmail source.

regards,
Horacio