Re: [sup-talk] Ferret to Xapian conversion

[Kevin Riggle <kevinr@free-dissociation.com>]

Excerpts from William Morgan's message of Sun Jan 03 10:18:53 -0500 2010:
> If you run this script, please report your experience, since I'd like to
> include it in the 0.10 release coming soon.
> 
When I run the script, I get a number of lines complaining about being unable
to convert various encodings -- is this expected behavior?

eg.:

## read 15375m (about 19%) @ 8.1m/s. 0:31:44 elapsed, about 2:17:28 remaining
[Sun Jan 03 14:53:33 -0500 2010] WARNING: couldn't transcode text from UTF-8 (utf-8) to UTF-8) ("Summer time is popul"...) (got "\223assisting\224 with"... (Iconv::IllegalSequence))
...
[Sun Jan 03 15:31:45 -0500 2010] WARNING: couldn't transcode text from ANSI (ANSI) to UTF-8) ("Dear Librarian, \n\nBr"...) (got invalid encoding ("UTF-8", "ANSI") (Iconv::InvalidEncoding))
## read 33992m (about 41%) @ 8.1m/s. 1:10:13 elapsed, about 1:39:07 remaining
[Sun Jan 03 15:31:55 -0500 2010] WARNING: couldn't transcode text from X-UNKNOWN (ASCII) to UTF-8) ("I agree with Jacky's"...) (got "\240 \240 \240I actually "... (Iconv::IllegalSequence))

Also, the conversion appeared to terminate prematurely, so I didn't bother 
running Sup against the resulting database.  The last few lines were:

## read 34123m (about 42%) @ 8.1m/s. 1:10:28 elapsed, about 1:38:49 remaining
## read 34259m (about 42%) @ 8.1m/s. 1:10:43 elapsed, about 1:38:30 remaining
## read 34388m (about 42%) @ 8.1m/s. 1:10:58 elapsed, about 1:38:12 remaining
kevinr@black-opal:~/src/sup$ 

- Kevin
-- 
Kevin Riggle (kevinr@free-dissociation.com) 
MIT Class of 2010, Course VI-3 (Computer Science)
http://free-dissociation.com
_______________________________________________
sup-talk mailing list
sup-talk@rubyforge.org
http://rubyforge.org/mailman/listinfo/sup-talk