Re: [sup-devel] Bug: UTF-8 error when sending messages

[Gaute Hope <eg@gaute.vetsj.com>]

Excerpts from Adeel Ahmad Khan's message of 2011-01-31 19:24:42 +0100:
> 
> Tero Tilus <tero@tilus.net>:
> > Adeel Ahmad Khan, 2011-01-29 05:11:
> > >     invalid byte sequence in UTF-8
> > ...
> > > I am using a nearly fresh installation of Sup 0.12.1 with Ruby
> > > 1.9.2p136.  I have LOCALE="en_US.UTF-8".

What editor are you using ? Could you attach some buggy text (make sure
it is transferred in binary - ie upload an archive somewhere like
http://paste.xinu.at/)?

> > Both the errors were from regex matches against message body.  Somehow
> > your editor doesn't know your locale or is not obeying it.  As a
> > result non-utf8 stuff gets saved to disk and sup gets confused.
>
> You were right.  It turned out to be an issue with editor.

I'm not so sure. I think it is a problem with Ruby (1.9) degrading the
the text it reads to US-ASCII (8bit). And when you later try to do UTF-8
stuff with it (append, or regexp with UTF-8) it fails. Perhaps even when
trying to access chars in the string (the body) that are not really
US-ASCII [2].

What my patch does is tell Ruby that the string is an UTF-8 string, no
matter what it deduced from reading the file in the first place - or
what might have happened throughout Sup's processing of the text.

Try this patch, it forces encoding on the entire body: http://ix.io/1rO

(these patches are workarounds; not to be applied to source-tree)

The same happened with labels.txt (or with contacts). Whenever a label
string could be degraded to US-ASCII, Ruby did so (how should it know it
was UTF-8 anyway?), then when trying to append, match or work with the
US-ASCII string towards UTF-8 input it failed.

What I think must be done (as an alternative to supporting different
encodings all the way) - is to _always_ read all files in UTF-8 [1] (or
transcode to UTF-8), and perhaps most difficultly _keep_ the strings
UTF-8 throught the entire Sup processes.

[1] http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings
[2] http://www.ruby-forum.com/topic/194493

Best regards,
Gaute