Quantcast

Migrate from twiki 5.1.4 with utf-8.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Migrate from twiki 5.1.4 with utf-8.

Виктор
  I have Twiki 5.1.4 with my self patch for utf-8. It is hardly patched
include CGi.pm from CPAN ))). I thinking this is not the right way.I do
not quite understand any changes made in foswiki for utf-8.  can i get a
litle bit more  info about engine work and utf-8 ?

I try to migrate fromTWiki because it not still anounced going to utf-8  
and 6.0.1 have big refactoring code and  some my utf-8 patch not work now.

Victor.

---
Это сообщение проверено на вирусы антивирусом Avast.
http://www.avast.com


------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Migrate from twiki 5.1.4 with utf-8.

George Clark-2
Hello Victor,

Foswiki 2.0 now uses UNICODE strings internally, and only converts to
utf-8 at the "edges".  The old {Site}{CharSet} setting is *hardcoded* to
utf-8,  and there is a new {Store}{Encoding} that is used when reading
topics from and writing topics to the Store (RCS or the new PlainFile
store).   The end result is that UNICODE "just works".   What this means
for extensions is that they have to remove their code that does
encoding/decoding of utf-8.  The only time anything needs to be encoded
is when writing to external files.   We've added a global
$Foswiki::UNICODE flag that extensions can use to determine if they are
running on a Foswiki 2.0 core.

Other extension changes needed are any regular expressions that use A-Z
style notation need to be changed to character classes.

Probably the most difficult part of upgrades to Foswiki 2.0 is the need
to convert the stored topics to utf-8.   We've found that because users
paste in characters from windows applications, the existing store is
often a mixture of encodings.  iso-8859-1 being the default., plus
cp-1252 pasted in from windows.   So the conversion to utf-8 can be
challenging. If you have mixed encodings, then the
CharsetConverterContrib is able to handle that, though it needs some
manual tuning.

As far as CGI and perl goes, Foswiki went along with the recommended
changes from the CGI maintainers.
 - All use of qw(-any) has been scrubbed
 - The CGI::multi_param has been implemented as
Foswiki::Request::multi_param(), and all calls to ::param() in list
contexts have been appropriately changed to either use multi_param, or
declare as scalar.
 - We've tested on perl up through 5.23.2,  fixing all of the uses of
deprecated syntax   - unescaped { braces in regexes,  qw() without
enclosing (),  etc.

Foswiki no longer ships any CPAN in the release.  It's expected that
CPAN modules should be installed using whatever pkg mechanism is
appropriate for your system.  (Some non-default extensions don't follow
this, but core Foswiki is clean.)

You can find more UTF8 information at:
 - http://foswiki.org/Support/Utf8MigrationConsiderations
 - http://foswiki.org/Support/WhatIsUtf8AllAbout

I've been looking through development documents, and don't see anything
else that's more informative about the core changes. A lot of the
unicode development documentation pre-date the Foswiki utf-8
conversion.  Maybe someone else will have some pointers.

George


On 11/30/2015 12:30 PM, Виктор wrote:

>   I have Twiki 5.1.4 with my self patch for utf-8. It is hardly patched
> include CGi.pm from CPAN ))). I thinking this is not the right way.I do
> not quite understand any changes made in foswiki for utf-8.  can i get a
> litle bit more  info about engine work and utf-8 ?
>
> I try to migrate fromTWiki because it not still anounced going to utf-8  
> and 6.0.1 have big refactoring code and  some my utf-8 patch not work now.
>
> Victor.
>
> ---
> Это сообщение проверено на вирусы антивирусом Avast.
> http://www.avast.com
>
>
> ------------------------------------------------------------------------------
> Go from Idea to Many App Stores Faster with Intel(R) XDK
> Give your users amazing mobile app experiences with Intel(R) XDK.
> Use one codebase in this all-in-one HTML5 development environment.
> Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
> http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss



------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Migrate from twiki 5.1.4 with utf-8.

Crawford Currie
In reply to this post by Виктор
Hi Victor,

We have been able to support UTF-8 data in Foswiki for a long time.
However the support was never very good.

In Foswiki 2.0 we decided to move the core code to use *Unicode* for
characters. This is slightly different to supporting *UTF-8* (which is
just one possible way of encoding unicode characters). By supporting
unicode, we are able to support *any* character encoding. At the same
time we have standardised on using UTF-8 for the database, as it is a
standard for web services and most 3rd party tools support it.

Unicode characters may be any size, so we had to find and rewrite all
code that assumed that character data (e.g. topic contents) was 8-bit.
This impacted in a *lot* of areas, such as regular expressions,
encoding/decoding parameter data, HTTP request handling, database
encoding.... the list goes on. We had been preparing the code for this
change for several years, so it wasn't as bad as it could have been, but
it was still a lot of work.

The result of this work is that Foswiki 2.x supports international
characters in topic content, topic names, attachment names,
metadata...... everywhere, in fact.

The biggest problem facing someone making the move to Foswiki 2.x is
normalising the encoding used in the database. It is possible for a
TWiki/Foswiki<2 database to contain topics with characters that are
encoded using a number of different encodings - for example, a single
topic might include text encoded using windows-1252, iso-8859-1 and
UTF-8. TWiki/Foswiki<2 tolerates this because it treats all topic data
as streams of bytes. However Foswiki 2.x requires the database to be
encoded consistently in UTF8, so you have to find all the bad encodings
in topics and fix them.

To support this, we developed a range of powerful tools that help you
find and fix the encodings.
http://foswiki.org/Extensions/CharsetConverterContrib is an excellent
place to start.

Regards,

C.


On 30/11/15 17:30, Виктор wrote:

>    I have Twiki 5.1.4 with my self patch for utf-8. It is hardly patched
> include CGi.pm from CPAN ))). I thinking this is not the right way.I do
> not quite understand any changes made in foswiki for utf-8.  can i get a
> litle bit more  info about engine work and utf-8 ?
>
> I try to migrate fromTWiki because it not still anounced going to utf-8
> and 6.0.1 have big refactoring code and  some my utf-8 patch not work now.
>
> Victor.
>
> ---
> ��� ��������� ��������� �� ������ ����������� Avast.
> http://www.avast.com
>
>
> ------------------------------------------------------------------------------
> Go from Idea to Many App Stores Faster with Intel(R) XDK
> Give your users amazing mobile app experiences with Intel(R) XDK.
> Use one codebase in this all-in-one HTML5 development environment.
> Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
> http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss


--
Crawford Currie - Owner, C-Dot Consultants - landline: +44-1606-330-242
- mobile: +44-7837-877-956 - skype: cdot-uk - public key
http://pgp.mit.edu/pks/lookup?op=vindex&search=0x0CD6BAE648697B60

------------------------------------------------------------------------------
Go from Idea to Many App Stores Faster with Intel(R) XDK
Give your users amazing mobile app experiences with Intel(R) XDK.
Use one codebase in this all-in-one HTML5 development environment.
Design, debug & build mobile apps & 2D/3D high-impact games for multiple OSs.
http://pubads.g.doubleclick.net/gampad/clk?id=254741911&iu=/4140
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Loading...