Need for working Import/Export option

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Need for working Import/Export option

Lynnwood Brown
Greetings Fellow Foswiki-zens,
I discovered yesterday that ExcelImportExportPlugin has been non-working since around release 1.1.2. This prompts me to raise a call for some kind of working import/export solution for Foswiki. While ExcelImportExportPlugin has had problems for...well, forever, it at least provided a solution for batch creation of records. I have often needed this over the years for various reasons such as importing information from some external application or recreating a batch of topics that had some problem. My most recent need was recreating some 100 user topics that had been lost.

It's not clear to me that this particular plugin was or is the best option for addressing this need. I've often wished that similar functionality was offered for more generic file format such as csv. Some of the issues with ExcelImportExportPlugin had to do with the peculiarities and changes over the years to the M$ proprietary file format which don't really add anything to the essential functionality of the plugin.

Another option I've considered over the years is to leverage the functionality that's already coded into the bulk user registration to allow creating sets of topics from a table.

In an brief irc conversation yesterday Michael Daum also pointed out that some bulk-data-generating needs might be better addressed using MetaDataPlugin rather than creating separate topics for each record. I completely agree but I still occasionally need the ability to bulk-generate topics as well.

As an aside, I have even searched repeatedly for some kind of application that would batch-generate sets of text files - something like mail-merge but outputting text files - thinking that I would generate the topic files off-line and then upload them to a Foswiki installation. Rather to my surprise, I've never found any simple solution for this approach either.

Exporting is not so much of an issue as it's easy enough to create a formatted search that outputs tab-delineated values inside of a textarea field - allowing copy-and-paste export. Still, it would be nice to have the ability to generate an attached csv file.

In summary, if we were creating something from scratch, here would be my wish-list of core features:
   * Ability to bulk-generate sets of topics and populate the DataForm fields.
   * Allow import from a simple comma or tab delineated file (and perhaps also from an xml file). Saving to Excel file is less important to me personally but may be important to many users who depend of that program.
   * Optionally allow generating topics from a Foswiki table (similar to bulk registration).
   * Allow for 3 different output formats: as a table within a topic, as MetaDataPlugin records, or as separate topic-records.
   * Allow for creating a downloadable (or attachment) csv file from a search-generated table.

I have a couple of projects down the road where I'll need this functionality and while I don't have a sponsor currently for this work, I may be able to find one. Or else, I may consider sponsoring it myself since I feel it's a vitally important general feature for Foswiki.

Thanks for any feedback on this idea and for everyone's contribution to Foswiki!
Lynnwood
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Need for working Import/Export option

Aaron Fuleki
+1

One suggestion: I very much like the idea of the core functionality for this making it's way into a a set of core API methods, which could be unit tested/developed with a CLI first, then maybe wired up to a plugin that provides a GUI.

I've used a variety of import/export tools for CMS's, CRM's, etc., and the user experience is a huge variable.  That could easily become a giant time sink.  Developing in phases (API->CLI->GUI) might help avoid that.

If the eventual plugin exposed a RESTful interface, I'd be happy to help code a GUI.

-Aaron

-------------------------------------------
Aaron Fuleki
Business Intelligence Analyst
Denison University
740.587.5752
-------------------------------------------


On Tue, Oct 22, 2013 at 9:55 AM, Lynnwood Brown <[hidden email]> wrote:
Greetings Fellow Foswiki-zens,
I discovered yesterday that ExcelImportExportPlugin has been non-working since around release 1.1.2. This prompts me to raise a call for some kind of working import/export solution for Foswiki. While ExcelImportExportPlugin has had problems for...well, forever, it at least provided a solution for batch creation of records. I have often needed this over the years for various reasons such as importing information from some external application or recreating a batch of topics that had some problem. My most recent need was recreating some 100 user topics that had been lost.

It's not clear to me that this particular plugin was or is the best option for addressing this need. I've often wished that similar functionality was offered for more generic file format such as csv. Some of the issues with ExcelImportExportPlugin had to do with the peculiarities and changes over the years to the M$ proprietary file format which don't really add anything to the essential functionality of the plugin.

Another option I've considered over the years is to leverage the functionality that's already coded into the bulk user registration to allow creating sets of topics from a table.

In an brief irc conversation yesterday Michael Daum also pointed out that some bulk-data-generating needs might be better addressed using MetaDataPlugin rather than creating separate topics for each record. I completely agree but I still occasionally need the ability to bulk-generate topics as well.

As an aside, I have even searched repeatedly for some kind of application that would batch-generate sets of text files - something like mail-merge but outputting text files - thinking that I would generate the topic files off-line and then upload them to a Foswiki installation. Rather to my surprise, I've never found any simple solution for this approach either.

Exporting is not so much of an issue as it's easy enough to create a formatted search that outputs tab-delineated values inside of a textarea field - allowing copy-and-paste export. Still, it would be nice to have the ability to generate an attached csv file.

In summary, if we were creating something from scratch, here would be my wish-list of core features:
   * Ability to bulk-generate sets of topics and populate the DataForm fields.
   * Allow import from a simple comma or tab delineated file (and perhaps also from an xml file). Saving to Excel file is less important to me personally but may be important to many users who depend of that program.
   * Optionally allow generating topics from a Foswiki table (similar to bulk registration).
   * Allow for 3 different output formats: as a table within a topic, as MetaDataPlugin records, or as separate topic-records.
   * Allow for creating a downloadable (or attachment) csv file from a search-generated table.

I have a couple of projects down the road where I'll need this functionality and while I don't have a sponsor currently for this work, I may be able to find one. Or else, I may consider sponsoring it myself since I feel it's a vitally important general feature for Foswiki.

Thanks for any feedback on this idea and for everyone's contribution to Foswiki!
Lynnwood
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss


------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Need for working Import/Export option

Julian Levens
In reply to this post by Lynnwood Brown
Lynwood

I have written a plugin for local use with some of those features.

It's actually a rest handler that scans a workarea via a cron job then  
creates topics based on the tsv files found in that directory.

The input tsv is expected to have two headers rows:

WebName, TopicPrefix, InitialState [,FormName] [,WorkFlowName]
Field/Column Names as per common practice

and of course data in following lines matching the field names.

Topic names are usually formed as $TopicPrefix . "AUTOINC000001" and the  
topic built
using the data provided.

If the 1st column has the name _topic then this is used instead as the  
topic name.

It is expected that a WorkFlow from WorkFlowPlugin is used and an  
InitialState is also set.

If an explicit FormName or WorkFlowName are not given then they default to
$TopicPrefix ."FormName" and $TopicPrefix . "FlowName" respectively.

It may not be everything you want but it should give you a start.

It's not that much code see http://pastebin.com/AGupD6MG to pick up the  
essence of it.

One omission is the ability to have a column called _text that would be  
used to create topic content, but that's easy to fix.

A caveat is that this was written for FW 1.0.8 so it may need tweaking for  
FW 1.1+, but I think that's minor.

Hope that helps

--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Need for working Import/Export option

Crawford Currie
In reply to this post by Lynnwood Brown
On 22/10/13 14:55, Lynnwood Brown wrote:
As an aside, I have even searched repeatedly for some kind of application that would batch-generate sets of text files - something like mail-merge but outputting text files - thinking that I would generate the topic files off-line and then upload them to a Foswiki installation. Rather to my surprise, I've never found any simple solution for this approach either.
There is no simple solution to this problem because it is a very simple problem , where the devil is in the detail. I usually write custom perl scripts to do this sort of transformation (though I could equally use AWK or some other text processing language), and usually without calling the Foswiki APIs. The reasons are various:
  1. It is rare that such a transformation is a simple 1:1 conversion - more often than not some processing is require en route,
  2. "Simple" tools often require complex guidance to achieve their simple jobs - for example, writing XSL rules used to transform XML can be quite complex,
  3. Input data often requires calling custom APIs to extract it - e.g. from databases used by other tools,
  4. Given a framework of such a transformation script, it is usually easy to customise it for the specific task in hand.
Of course writing such a perl script is easy for me because I know the store inside out, so can anticipate most problems before they occur.

Perhaps what is required is not so much a "tool" to do this job, and more a set of examples and a tutorial for using the existing tools i.e. documentation. Of course this is just as much work than writing YADATA (Yet Another Data Transform Appliance) but my feeling is that it would be longer lived and easier for non-programmers to maintain and extend. The knowledge gained in writing the tutorial would also be useful in steering future developments of the APIs and data formats.
In summary, if we were creating something from scratch, here would be my wish-list of core features:
   * Ability to bulk-generate sets of topics and populate the DataForm fields.
   * Allow import from a simple comma or tab delineated file (and perhaps also from an xml file). Saving to Excel file is less important to me personally but may be important to many users who depend of that program.
   * Optionally allow generating topics from a Foswiki table (similar to bulk registration).
   * Allow for 3 different output formats: as a table within a topic, as MetaDataPlugin records, or as separate topic-records.
   * Allow for creating a downloadable (or attachment) csv file from a search-generated table.
This list could act as the index for such a tutorial.

C.

--
Crawford Currie - Owner, C-Dot Consultants - landline: +44-1606-330-242 - mobile: +44-7837-877-956 - skype: cdot-uk - public key http://keyserver.veridis.com:11371/search?q=Crawford+Currie

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135991&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss