Quantcast

Which plugin to use for classification of topics?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Which plugin to use for classification of topics?

BramVanOosterhout
Hi,

I am looking for a plugin to help me with the following application.

I have multiple email archives. Some are saved in proprietary formats. I am
able to extract these formats and create a foswiki site that has a page for
each email with attachments.

I am looking for a way to get easier access to this archive and have been
thinking about:
   * tag clouds by email names/topics/date
   * Hierarchies of year, month, day
   * Hierarchies by subject/date

Other slices as I think of them

Before I start experimenting, is there any advice you can offer re the merits
of the following plugins as a starting point:
   * ClassificationPlugin
   * TopicClassificationAddOn
   * TagCloudPlugin, TagMePlugin, TagsPlugin

I have read the documentation, and believe that all these plugins require
user interaction to classify/tag a topic.

I intend to classify/tag each topic as it is created programmatically.
Have I misunderstood the documentation?
Is there a better way? Another more useful plugin?

Thanks for your time.

--
Bram van Oosterhout


------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of topics?

Lynnwood Brown
Hey Bram,
The key to setting up your email archive within Foswiki the way you describe would be to populate Foswiki data forms with the email metadata. Assuming you figured out a way to create Foswiki topics from the individual emails, perhaps you already figured out how to do this or could modify your workflow to add this.

Once you had all you topics created with key info entered in the data forms, you could fairly easily add navigation and search filters using Classification Plugin which includes all the infrastructure you need for managing tags.

Among the plugins you listed, I would recommend ClassificationPlugin. I created TopicClassificationAddOn and still use it on some sites but if I’m creating a new site and want full-featured content organization with categories and tags, I use ClassificationPlugin. I do not recommend TagMePlugin for a couple of reasons. One is that it stores the tags in non-standard ways which means that they can not be searched using normal Foswiki searches. Also, that plugin has not been updated in several years and I was noticing yesterday that it has a number of outstanding bug reports.

I’d be very interested in hearing more about how you converted your emails into Foswiki topics. If there was any possibility of your method being generalized, I think a lot of folks would be interested in hearing about it and possibly even creating a Foswiki contrib for this purpose.

Good luck with your project!
Lynnwood Brown

> On May 30, 2015, at 8:26 AM, Bram van Oosterhout <[hidden email]> wrote:
>
> Hi,
>
> I am looking for a plugin to help me with the following application.
>
> I have multiple email archives. Some are saved in proprietary formats. I am
> able to extract these formats and create a foswiki site that has a page for
> each email with attachments.
>
> I am looking for a way to get easier access to this archive and have been
> thinking about:
>   * tag clouds by email names/topics/date
>   * Hierarchies of year, month, day
>   * Hierarchies by subject/date
>
> Other slices as I think of them
>
> Before I start experimenting, is there any advice you can offer re the merits
> of the following plugins as a starting point:
>   * ClassificationPlugin
>   * TopicClassificationAddOn
>   * TagCloudPlugin, TagMePlugin, TagsPlugin
>
> I have read the documentation, and believe that all these plugins require
> user interaction to classify/tag a topic.
>
> I intend to classify/tag each topic as it is created programmatically.
> Have I misunderstood the documentation?
> Is there a better way? Another more useful plugin?
>
> Thanks for your time.
>
> --
> Bram van Oosterhout
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss


------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of topics?

Raymond Lutz
Bram:
I have been working on a similar project in that it includes the need
for extraction of email metadata and population of datafields in topics.
However, the goal in my case is to allow users to send email directly to
the foswiki and then create topics based on what is in the email, the
email metadata, and content of the body. Thus, it works not in
processing a bunch of emails that already exist, but for emails that are
being sent in. These two projects are almost the same.

There exists a contrib package called MailInContrib which does most of
the email part, and was originally designed mostly to allow people to
extend current topics. For that it works pretty well, allowing the
format of the body of the email message to be formatted using a
template, and there is a "ContentProcessor" concept to allow that the
email body to be additionally processed.

This contrib is great if you have a lot of attachments to add to a
topic. Manual method is one at a time, whereas with MailInContrib, all
the attachments of the email are attached in one action. Only drawback
is that the description of each attachment is just "sent from email".
For my purposes, I need to extend the MailInContrib so it will allow the
email metadata (to, from, date, etc) as well as content from the email
that is tagged according to some scheme (like "fieldname: fielddata",
ex: URL: http...) to be adopted from the body of the email. I was hoping
to make this facility somewhat arbitrary so that a regexp would be
applied to the bold and the result qualified, and then placed into a
data field.

What is missing from MailInContrib is much like what you need.
1. It is missing the facility to construct AUTOINC topic names as the
emails are processed. This should be pretty simple and I believe that it
may amount to a single line of code placed in exactly the right place.
You will need this, most likely, as you construct topics for each email.
As the topics are created, they are created with a topic template and an
associated Form.
2. It is missing the facility to insert values into data fields in the
new topics.
3. It already parses the email and extracts the meta data but these are
not available in a standards way for #2 above.
4. It already can manipulate the body but it does not have a means to
pass extracted data parameters back to #2.
5. MailInContrib can operate differently based on what mailbox it
accesses data from, so it can already generate a new topic (but not with
AUTOINC)

I think there is substantial overlap. Except for the fact that the
MailInContrib goes out to the POP2 or IMAP mail server to get the email,
from there on out, I think the facility you need is also what I am
needing for my purposes.

So, to be clear, in my application, I have a dataset with a form
specifying Media, say news articles. People now email these to
listservs. They are used to putting the URL in the first line (or so) of
the email, and pasting in a plain text or HTML formatted body. In my
application, the MailInContrib first receives the email. This exists.
The parsing of the email header is done, but not easily available, that
can be fixed. Then, I want the ContentProcessor facility to extract
fields (or could scan the email and generate tags, in your use). I need
a new topic to be generated with the specified topic template and form
akin to the URL directive:

http://www.copswiki.org/w/bin/edit/Common/MAUTOINC0?templatetopic=Common.MediaTopicTemplate&topicparent=Common.NewsMedia&formtemplate=Common.MediaForm

This generates a new topic with name Mnnnn where n is the next available
number, using MediaTopicTemplate and form MediaForm. I need to then
update the form fields as a part of the ContentProcessor hooks that
already exist.

Now, I think that most of this is usable in your application, and that
the MailInContrib could be modified to simply look into a folder and
process all files there as if they are in an inbox. With that small
additional extension, then all the things I need for what I have been
toying with map to what you are doing.

Crawford Currie is the developer who wrote the MailInContrib and it does
a lot of things you will need, such as dealing with various formats of
body, attachments. etc.  He has been very cooperative so far in
directing me in using the MailInContrib as a start. I would encourage
you to install the MailInContrib. I will note that to use it the crontab
directive they mention is incorrect (put &1>2 thingy at the end, not the
middle (task added for doc correction)) and there is another package or
two that are missing.  I can get you that information from my notes if
you proceed with this.

The MailInContrib is envoked by a cron job and uses a configuration file
to run. You should be able to use this to parse all your emails and
establish topics.

Although it is pretty close but there a few things missing. I have been
around foswiki and TWiki for many years but have not done a lot of
development at this level on this beast. I looked but did not find Func
calls that will establish data fields in topics. Either I missed it or
it does not exist at this time. For example, I wanted to get a list of
all the Form fields and that may, at this time, require parsing an HTML
data block, the one that is used when the user interacts with the form
-- but again, I only just started to research how to do it.

Bottom line is that your application is almost identical to what I was
toying with in many respects. I would like to collaborate with you on this.

--Ray Lutz










On 2015-05-30 6:40 AM, Lynnwood Brown wrote:

> Hey Bram,
> The key to setting up your email archive within Foswiki the way you describe would be to populate Foswiki data forms with the email metadata. Assuming you figured out a way to create Foswiki topics from the individual emails, perhaps you already figured out how to do this or could modify your workflow to add this.
>
> Once you had all you topics created with key info entered in the data forms, you could fairly easily add navigation and search filters using Classification Plugin which includes all the infrastructure you need for managing tags.
>
> Among the plugins you listed, I would recommend ClassificationPlugin. I created TopicClassificationAddOn and still use it on some sites but if I’m creating a new site and want full-featured content organization with categories and tags, I use ClassificationPlugin. I do not recommend TagMePlugin for a couple of reasons. One is that it stores the tags in non-standard ways which means that they can not be searched using normal Foswiki searches. Also, that plugin has not been updated in several years and I was noticing yesterday that it has a number of outstanding bug reports.
>
> I’d be very interested in hearing more about how you converted your emails into Foswiki topics. If there was any possibility of your method being generalized, I think a lot of folks would be interested in hearing about it and possibly even creating a Foswiki contrib for this purpose.
>
> Good luck with your project!
> Lynnwood Brown
>
>> On May 30, 2015, at 8:26 AM, Bram van Oosterhout <[hidden email]> wrote:
>>
>> Hi,
>>
>> I am looking for a plugin to help me with the following application.
>>
>> I have multiple email archives. Some are saved in proprietary formats. I am
>> able to extract these formats and create a foswiki site that has a page for
>> each email with attachments.
>>
>> I am looking for a way to get easier access to this archive and have been
>> thinking about:
>>    * tag clouds by email names/topics/date
>>    * Hierarchies of year, month, day
>>    * Hierarchies by subject/date
>>
>> Other slices as I think of them
>>
>> Before I start experimenting, is there any advice you can offer re the merits
>> of the following plugins as a starting point:
>>    * ClassificationPlugin
>>    * TopicClassificationAddOn
>>    * TagCloudPlugin, TagMePlugin, TagsPlugin
>>
>> I have read the documentation, and believe that all these plugins require
>> user interaction to classify/tag a topic.
>>
>> I intend to classify/tag each topic as it is created programmatically.
>> Have I misunderstood the documentation?
>> Is there a better way? Another more useful plugin?
>>
>> Thanks for your time.
>>
>> --
>> Bram van Oosterhout
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Foswiki-discuss mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss


------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

BramVanOosterhout
Hi Ray,
Thanks for the extensive response. I was dimly aware of the MailInContrib.
But it never occurred to me to investigate it. I have done all the email
parsing using CPAN Email::MIME and other modules. It has taken me a few weeks
to come to grips with them, but I now feel fairly confident I understand them.

Reading the documentation of MailInContrib I could have saved myself some
time by looking at Crawford's code. I have dug through his code before (the
PublishPlugin) which is a superb piece of work. Crawford does write a lot
better code than I do, and I find it sometimes hard to follow. He does write
working code, which is more than I can say for my own efforts of the past
weeks.

I will have a look at MailInContrib as a replacement for my own efforts. From
the documentation I gather that I should be able to do what I want using a
*content processor* to interface to the ClassificationPlugin suggested by
Lynwood in response to my question.

I plan to start my investigation of the ClassificationPlugin in June. I don't
have much practice with Forms. I have never liked the look of them and have
stuck to editing wiki pages in the text editor. Don't like the WYSIWIG thing
either :-).

Once I have worked out how I can put the email data in the form I will get
back to the content processor for MailInContrib.

I'll let you know when I make progress.

Thanks for your response and pointers.

--
Bram van Oosterhout


------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

BramVanOosterhout
In reply to this post by Lynnwood Brown
Hi Lynwood,
Thanks for your response I will pursue the implementation of my project using
the ClassificationPlugin. Ray Lutz pointed out that I should consider
MailInContrib to do the Email parsing. An interface between the two may be a
practical implementation of the whole project.

If I understand the documentation of ClassificationPlugin correctly, then it
will add the classification data to the page as meta data:
%META:someformname{field1="value1" field2=value2"" field3="value3" and so
on ...}%

I should be able to write this %META<type>{..}% line containing the attribute
values of the page and the rest should be managed by the plugin functionality.

This is the first time I will be meddling with Forms. Never liked them. The
documentation is opaque, although I have found some pages that are helpful:
   * http://foswiki.org/Support.WhatIsStructuredDataInFoswiki
   * http://foswiki.org/System/MetaData

I think that once I have the ClassificationPlugin installed, I can experiment
and work out the details.

I'll report back if/when I have some result. It may be a few weeks!

Thanks for the suggestion.

--
Bram van Oosterhout


------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

Raymond Lutz
In reply to this post by BramVanOosterhout
Bram:
I too do not usually use wysiwyg but it is useful at times esp. to paste
in formatted web pages, for example. But data forms are a key element of
the power of foswiki.
I strongly suggest that you do not manipulate the form data directly in
the topic file (such as by fiddling with the META markup) as that does
not easily port to other storage mechanisms, such as a mysql backend.
This does get important as the number of data items grows, which may be
the case for you.
I do think that you should be able to do what you need if all the hooks
that I was looking for are there, with the addition of the one
additional that you want to run the MailInCron script and have it
operate not on incoming mail, but on a directory tree of mail items.

This part of integrating with rest of the foswiki machine elegantly is
where we need to be careful. I also went through the process of
manipulating mail and had a lot of that working prior to finding
MailInContrib. So we are sort of in the same boat...
--Ray

On 2015-05-31 2:12 AM, Bram van Oosterhout wrote:

> Hi Ray,
> Thanks for the extensive response. I was dimly aware of the MailInContrib.
> But it never occurred to me to investigate it. I have done all the email
> parsing using CPAN Email::MIME and other modules. It has taken me a few weeks
> to come to grips with them, but I now feel fairly confident I understand them.
>
> Reading the documentation of MailInContrib I could have saved myself some
> time by looking at Crawford's code. I have dug through his code before (the
> PublishPlugin) which is a superb piece of work. Crawford does write a lot
> better code than I do, and I find it sometimes hard to follow. He does write
> working code, which is more than I can say for my own efforts of the past
> weeks.
>
> I will have a look at MailInContrib as a replacement for my own efforts. From
> the documentation I gather that I should be able to do what I want using a
> *content processor* to interface to the ClassificationPlugin suggested by
> Lynwood in response to my question.
>
> I plan to start my investigation of the ClassificationPlugin in June. I don't
> have much practice with Forms. I have never liked the look of them and have
> stuck to editing wiki pages in the text editor. Don't like the WYSIWIG thing
> either :-).
>
> Once I have worked out how I can put the email data in the form I will get
> back to the content processor for MailInContrib.
>
> I'll let you know when I make progress.
>
> Thanks for your response and pointers.
>
> --
> Bram van Oosterhout
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>


------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

Chris Hoefler
Bram:
I suggest looking at the NatEditPlugin, which is integrated in the upcoming 1.2.0 I think. It is a great editor for wiki pages. The DataForms are primarily a storage mechanism. How they are displayed and edited is mostly up to you. With custom templates you can make them appear any way you choose. The power is in the way you can retrieve data, search, and otherwise make use of the data because it is structured.

Ray:
I think you are looking for Foswiki::Meta. Most of the functions for manipulating topic meta data are in that class. It is not too difficult to use. Many of the plugins on foswiki.org make use of them, so you should be able to find examples.



> On May 31, 2015, at 10:42 AM, Ray Lutz <[hidden email]> wrote:
>
> Bram:
> I too do not usually use wysiwyg but it is useful at times esp. to paste
> in formatted web pages, for example. But data forms are a key element of
> the power of foswiki.
> I strongly suggest that you do not manipulate the form data directly in
> the topic file (such as by fiddling with the META markup) as that does
> not easily port to other storage mechanisms, such as a mysql backend.
> This does get important as the number of data items grows, which may be
> the case for you.
> I do think that you should be able to do what you need if all the hooks
> that I was looking for are there, with the addition of the one
> additional that you want to run the MailInCron script and have it
> operate not on incoming mail, but on a directory tree of mail items.
>
> This part of integrating with rest of the foswiki machine elegantly is
> where we need to be careful. I also went through the process of
> manipulating mail and had a lot of that working prior to finding
> MailInContrib. So we are sort of in the same boat...
> --Ray
>
>> On 2015-05-31 2:12 AM, Bram van Oosterhout wrote:
>> Hi Ray,
>> Thanks for the extensive response. I was dimly aware of the MailInContrib.
>> But it never occurred to me to investigate it. I have done all the email
>> parsing using CPAN Email::MIME and other modules. It has taken me a few weeks
>> to come to grips with them, but I now feel fairly confident I understand them.
>>
>> Reading the documentation of MailInContrib I could have saved myself some
>> time by looking at Crawford's code. I have dug through his code before (the
>> PublishPlugin) which is a superb piece of work. Crawford does write a lot
>> better code than I do, and I find it sometimes hard to follow. He does write
>> working code, which is more than I can say for my own efforts of the past
>> weeks.
>>
>> I will have a look at MailInContrib as a replacement for my own efforts. From
>> the documentation I gather that I should be able to do what I want using a
>> *content processor* to interface to the ClassificationPlugin suggested by
>> Lynwood in response to my question.
>>
>> I plan to start my investigation of the ClassificationPlugin in June. I don't
>> have much practice with Forms. I have never liked the look of them and have
>> stuck to editing wiki pages in the text editor. Don't like the WYSIWIG thing
>> either :-).
>>
>> Once I have worked out how I can put the email data in the form I will get
>> back to the content processor for MailInContrib.
>>
>> I'll let you know when I make progress.
>>
>> Thanks for your response and pointers.
>>
>> --
>> Bram van Oosterhout
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Foswiki-discuss mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss

------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

Raymond Lutz
Yes, I agree, Chris. I want to use the ::Meta or maybe ::Form interface
and avoid touching the actual files. This is very important to me as I
want to move to a mysql backend soon as the number of files is getting
large and I believe searches will be much faster.

On 2015-05-31 10:19 AM, Chris Hoefler wrote:

> Bram:
> I suggest looking at the NatEditPlugin, which is integrated in the upcoming 1.2.0 I think. It is a great editor for wiki pages. The DataForms are primarily a storage mechanism. How they are displayed and edited is mostly up to you. With custom templates you can make them appear any way you choose. The power is in the way you can retrieve data, search, and otherwise make use of the data because it is structured.
>
> Ray:
> I think you are looking for Foswiki::Meta. Most of the functions for manipulating topic meta data are in that class. It is not too difficult to use. Many of the plugins on foswiki.org make use of them, so you should be able to find examples.
>
>
>
>> On May 31, 2015, at 10:42 AM, Ray Lutz <[hidden email]> wrote:
>>
>> Bram:
>> I too do not usually use wysiwyg but it is useful at times esp. to paste
>> in formatted web pages, for example. But data forms are a key element of
>> the power of foswiki.
>> I strongly suggest that you do not manipulate the form data directly in
>> the topic file (such as by fiddling with the META markup) as that does
>> not easily port to other storage mechanisms, such as a mysql backend.
>> This does get important as the number of data items grows, which may be
>> the case for you.
>> I do think that you should be able to do what you need if all the hooks
>> that I was looking for are there, with the addition of the one
>> additional that you want to run the MailInCron script and have it
>> operate not on incoming mail, but on a directory tree of mail items.
>>
>> This part of integrating with rest of the foswiki machine elegantly is
>> where we need to be careful. I also went through the process of
>> manipulating mail and had a lot of that working prior to finding
>> MailInContrib. So we are sort of in the same boat...
>> --Ray
>>
>>> On 2015-05-31 2:12 AM, Bram van Oosterhout wrote:
>>> Hi Ray,
>>> Thanks for the extensive response. I was dimly aware of the MailInContrib.
>>> But it never occurred to me to investigate it. I have done all the email
>>> parsing using CPAN Email::MIME and other modules. It has taken me a few weeks
>>> to come to grips with them, but I now feel fairly confident I understand them.
>>>
>>> Reading the documentation of MailInContrib I could have saved myself some
>>> time by looking at Crawford's code. I have dug through his code before (the
>>> PublishPlugin) which is a superb piece of work. Crawford does write a lot
>>> better code than I do, and I find it sometimes hard to follow. He does write
>>> working code, which is more than I can say for my own efforts of the past
>>> weeks.
>>>
>>> I will have a look at MailInContrib as a replacement for my own efforts. From
>>> the documentation I gather that I should be able to do what I want using a
>>> *content processor* to interface to the ClassificationPlugin suggested by
>>> Lynwood in response to my question.
>>>
>>> I plan to start my investigation of the ClassificationPlugin in June. I don't
>>> have much practice with Forms. I have never liked the look of them and have
>>> stuck to editing wiki pages in the text editor. Don't like the WYSIWIG thing
>>> either :-).
>>>
>>> Once I have worked out how I can put the email data in the form I will get
>>> back to the content processor for MailInContrib.
>>>
>>> I'll let you know when I make progress.
>>>
>>> Thanks for your response and pointers.
>>>
>>> --
>>> Bram van Oosterhout
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Foswiki-discuss mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Foswiki-discuss mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
> ------------------------------------------------------------------------------
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>


------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

Chris Hoefler
One thing I've found with respect to searches is that with a good index (ex: dbcachecontrib or solrplugin) complicated searches of even thousands of topics can be really fast. At least in our case, performance was not limited by the search, but by all of the processing of the search results. Since SolrPlugin provides a rest interface, I was able to implement searching and search result display asynchronously, which really improved the page load times. There are definitely use cases for database backends, but I don't think searching is one of them. I would suggest looking into some of the solr stuff if you are facing this issue. JQGridPlugin is a way to get Ajax-y  search results without doing a lot of work, and it supports both solr and dbcachecontrib.



> On May 31, 2015, at 4:44 PM, Ray Lutz <[hidden email]> wrote:
>
> Yes, I agree, Chris. I want to use the ::Meta or maybe ::Form interface
> and avoid touching the actual files. This is very important to me as I
> want to move to a mysql backend soon as the number of files is getting
> large and I believe searches will be much faster.
>
>> On 2015-05-31 10:19 AM, Chris Hoefler wrote:
>> Bram:
>> I suggest looking at the NatEditPlugin, which is integrated in the upcoming 1.2.0 I think. It is a great editor for wiki pages. The DataForms are primarily a storage mechanism. How they are displayed and edited is mostly up to you. With custom templates you can make them appear any way you choose. The power is in the way you can retrieve data, search, and otherwise make use of the data because it is structured.
>>
>> Ray:
>> I think you are looking for Foswiki::Meta. Most of the functions for manipulating topic meta data are in that class. It is not too difficult to use. Many of the plugins on foswiki.org make use of them, so you should be able to find examples.
>>
>>
>>
>>> On May 31, 2015, at 10:42 AM, Ray Lutz <[hidden email]> wrote:
>>>
>>> Bram:
>>> I too do not usually use wysiwyg but it is useful at times esp. to paste
>>> in formatted web pages, for example. But data forms are a key element of
>>> the power of foswiki.
>>> I strongly suggest that you do not manipulate the form data directly in
>>> the topic file (such as by fiddling with the META markup) as that does
>>> not easily port to other storage mechanisms, such as a mysql backend.
>>> This does get important as the number of data items grows, which may be
>>> the case for you.
>>> I do think that you should be able to do what you need if all the hooks
>>> that I was looking for are there, with the addition of the one
>>> additional that you want to run the MailInCron script and have it
>>> operate not on incoming mail, but on a directory tree of mail items.
>>>
>>> This part of integrating with rest of the foswiki machine elegantly is
>>> where we need to be careful. I also went through the process of
>>> manipulating mail and had a lot of that working prior to finding
>>> MailInContrib. So we are sort of in the same boat...
>>> --Ray
>>>
>>>> On 2015-05-31 2:12 AM, Bram van Oosterhout wrote:
>>>> Hi Ray,
>>>> Thanks for the extensive response. I was dimly aware of the MailInContrib.
>>>> But it never occurred to me to investigate it. I have done all the email
>>>> parsing using CPAN Email::MIME and other modules. It has taken me a few weeks
>>>> to come to grips with them, but I now feel fairly confident I understand them.
>>>>
>>>> Reading the documentation of MailInContrib I could have saved myself some
>>>> time by looking at Crawford's code. I have dug through his code before (the
>>>> PublishPlugin) which is a superb piece of work. Crawford does write a lot
>>>> better code than I do, and I find it sometimes hard to follow. He does write
>>>> working code, which is more than I can say for my own efforts of the past
>>>> weeks.
>>>>
>>>> I will have a look at MailInContrib as a replacement for my own efforts. From
>>>> the documentation I gather that I should be able to do what I want using a
>>>> *content processor* to interface to the ClassificationPlugin suggested by
>>>> Lynwood in response to my question.
>>>>
>>>> I plan to start my investigation of the ClassificationPlugin in June. I don't
>>>> have much practice with Forms. I have never liked the look of them and have
>>>> stuck to editing wiki pages in the text editor. Don't like the WYSIWIG thing
>>>> either :-).
>>>>
>>>> Once I have worked out how I can put the email data in the form I will get
>>>> back to the content processor for MailInContrib.
>>>>
>>>> I'll let you know when I make progress.
>>>>
>>>> Thanks for your response and pointers.
>>>>
>>>> --
>>>> Bram van Oosterhout
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Foswiki-discuss mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Foswiki-discuss mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Foswiki-discuss mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss

------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

Crawford Currie
If you want to add forms (and fields) to new topics using MailInContrib,
there are two approaches you can take.

If you are writing Perl and have access to Func (as implied by Ray),
then you can access the Foswiki::Meta API. This API will let you add the
form and the fields to the Meta object. However this approach requires
good knowledge of perl, and the internals of Foswiki.

The second approach is to embed topic meta-data in the text of the topic
before it is saved. For example, you might add this to the end of the
text of the topic:

%META:FORM{name="ExistingForm"}%
%META:FIELD{ name="Summary" attributes="M" title="Summary" value="Das es
la zummary"}%

Regards,

C.

On 01/06/15 03:29, Chris Hoefler wrote:

> One thing I've found with respect to searches is that with a good index (ex: dbcachecontrib or solrplugin) complicated searches of even thousands of topics can be really fast. At least in our case, performance was not limited by the search, but by all of the processing of the search results. Since SolrPlugin provides a rest interface, I was able to implement searching and search result display asynchronously, which really improved the page load times. There are definitely use cases for database backends, but I don't think searching is one of them. I would suggest looking into some of the solr stuff if you are facing this issue. JQGridPlugin is a way to get Ajax-y  search results without doing a lot of work, and it supports both solr and dbcachecontrib.
>
>
>
>> On May 31, 2015, at 4:44 PM, Ray Lutz <[hidden email]> wrote:
>>
>> Yes, I agree, Chris. I want to use the ::Meta or maybe ::Form interface
>> and avoid touching the actual files. This is very important to me as I
>> want to move to a mysql backend soon as the number of files is getting
>> large and I believe searches will be much faster.
>>
>>> On 2015-05-31 10:19 AM, Chris Hoefler wrote:
>>> Bram:
>>> I suggest looking at the NatEditPlugin, which is integrated in the upcoming 1.2.0 I think. It is a great editor for wiki pages. The DataForms are primarily a storage mechanism. How they are displayed and edited is mostly up to you. With custom templates you can make them appear any way you choose. The power is in the way you can retrieve data, search, and otherwise make use of the data because it is structured.
>>>
>>> Ray:
>>> I think you are looking for Foswiki::Meta. Most of the functions for manipulating topic meta data are in that class. It is not too difficult to use. Many of the plugins on foswiki.org make use of them, so you should be able to find examples.
>>>
>>>
>>>
>>>> On May 31, 2015, at 10:42 AM, Ray Lutz <[hidden email]> wrote:
>>>>
>>>> Bram:
>>>> I too do not usually use wysiwyg but it is useful at times esp. to paste
>>>> in formatted web pages, for example. But data forms are a key element of
>>>> the power of foswiki.
>>>> I strongly suggest that you do not manipulate the form data directly in
>>>> the topic file (such as by fiddling with the META markup) as that does
>>>> not easily port to other storage mechanisms, such as a mysql backend.
>>>> This does get important as the number of data items grows, which may be
>>>> the case for you.
>>>> I do think that you should be able to do what you need if all the hooks
>>>> that I was looking for are there, with the addition of the one
>>>> additional that you want to run the MailInCron script and have it
>>>> operate not on incoming mail, but on a directory tree of mail items.
>>>>
>>>> This part of integrating with rest of the foswiki machine elegantly is
>>>> where we need to be careful. I also went through the process of
>>>> manipulating mail and had a lot of that working prior to finding
>>>> MailInContrib. So we are sort of in the same boat...
>>>> --Ray
>>>>
>>>>> On 2015-05-31 2:12 AM, Bram van Oosterhout wrote:
>>>>> Hi Ray,
>>>>> Thanks for the extensive response. I was dimly aware of the MailInContrib.
>>>>> But it never occurred to me to investigate it. I have done all the email
>>>>> parsing using CPAN Email::MIME and other modules. It has taken me a few weeks
>>>>> to come to grips with them, but I now feel fairly confident I understand them.
>>>>>
>>>>> Reading the documentation of MailInContrib I could have saved myself some
>>>>> time by looking at Crawford's code. I have dug through his code before (the
>>>>> PublishPlugin) which is a superb piece of work. Crawford does write a lot
>>>>> better code than I do, and I find it sometimes hard to follow. He does write
>>>>> working code, which is more than I can say for my own efforts of the past
>>>>> weeks.
>>>>>
>>>>> I will have a look at MailInContrib as a replacement for my own efforts. From
>>>>> the documentation I gather that I should be able to do what I want using a
>>>>> *content processor* to interface to the ClassificationPlugin suggested by
>>>>> Lynwood in response to my question.
>>>>>
>>>>> I plan to start my investigation of the ClassificationPlugin in June. I don't
>>>>> have much practice with Forms. I have never liked the look of them and have
>>>>> stuck to editing wiki pages in the text editor. Don't like the WYSIWIG thing
>>>>> either :-).
>>>>>
>>>>> Once I have worked out how I can put the email data in the form I will get
>>>>> back to the content processor for MailInContrib.
>>>>>
>>>>> I'll let you know when I make progress.
>>>>>
>>>>> Thanks for your response and pointers.
>>>>>
>>>>> --
>>>>> Bram van Oosterhout
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> _______________________________________________
>>>>> Foswiki-discuss mailing list
>>>>> [hidden email]
>>>>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>>>> ------------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Foswiki-discuss mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Foswiki-discuss mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Foswiki-discuss mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
> ------------------------------------------------------------------------------
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>


--
Crawford Currie - Owner, C-Dot Consultants - landline: +44-1606-330-242
- mobile: +44-7837-877-956 - skype: cdot-uk - public key
http://pgp.mit.edu/pks/lookup?op=vindex&search=0x0CD6BAE648697B60

------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

BramVanOosterhout
Thanks Crawford,
I will consider the options after I have come to grips with the
classification plugin.

I have completed the installation. Now for the hard part.
How do I use it in my context. From the documentation I understand I am
embarking on the creation of a wiki application. And I have found the wiki
app installer plugin. Thanks for that.

But there is quite a way to go!

I appreciate all suggestions I have gotten in response to my request for
assistance.

Thanks all.


--
Bram van Oosterhout


------------------------------------------------------------------------------
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Which plugin to use for classification of

BramVanOosterhout
In reply to this post by Lynnwood Brown
Hi Lynwood,

some time ago I asked for advice on this forum re the creation of a Mail archive application. I have made some progress and thought I could report my successes and ask some questions here. The solution adopted uses parts of the ClassificationApp, which did help me somewhat as an example. You can see the full implementation as it stands today at http://foswiki.org/Support/Question1628.

In summary the solution was the following:

  • Just do it. Create the meta fields as you see fit and add them to the topic.txt
  • Update the metadata definition in Applications/ClassificationApp.ClassifiedTopic
    • And make To, From and CC of type tag
  • Create a page called ListOfTags and copy the DBSTATS statements from Applications/ClassificationApp.RenderKnownTags
    • make the field= a variable TAGFIELD, which is the tag we collect stats for.
    • use the list of tags to create a reference to a search for the tag: the SearchTaggedTopics page
  • The SearchTaggedTopics page contains one line:
    • %SEARCH{"From~'*!*'" type="query" format="| $topic | $formfield(Date) | $formfield(Subject) |"}%
I have a few questions that will help me improve on this solution. The essence of my questions is: I do not fully understand how Foswiki applications are constructed and used. In detail my questions are as follows:
  1. I extended the form definition in the ClassificationTopic to include the fields: From, To, CC, Date, Subject. I expected to do that using the TopicType creation link in the ClassificationApp, inheriting from ClassifiedTopic but I have been unable to work out how to proceed from there.
    • For instance, what are the implications of the inheritance. I expected the form to be copied, but that does not appear to be the case
  2. I copied and modified the Applications/ClassificationApp.RenderKnownTags page to get what I wanted. But the ClassificationApp has other Render... pages that would be nice to use. I do not understand the application well enough to decide whether
    • I should continue to use the ClassificationApp as an example and modify, OR
    • should change my approach regarding the meta fields so that I can use the Tag field for all the things I want to do
  3. Considering the possibility that I create and submit this functionality as an Foswiki application, should I:
    • Create a new application (MailApp), separate from the ClassificationApp, OR
    • Create additional pages in the ClassificationApp and repackage that application.


      On Sat, 30 May 2015 09:40:09 -0400, Lynnwood Brown wrote
      > Hey Bram,
      > The key to setting up your email archive within Foswiki the way you
      > describe would be to populate Foswiki data forms with the email
      > metadata. Assuming you figured out a way to create Foswiki topics
      > from the individual emails, perhaps you already figured out how to
      > do this or could modify your workflow to add this.
      >
      > Once you had all you topics created with key info entered in the
      > data forms, you could fairly easily add navigation and search
      > filters using Classification Plugin which includes all the
      > infrastructure you need for managing tags.
      >
      > Among the plugins you listed, I would recommend
      > ClassificationPlugin. I created TopicClassificationAddOn and still
      > use it on some sites but if [UTF-8?]I’m creating a new site and want full-
      > featured content organization with categories and tags, I use
      > ClassificationPlugin. I do not recommend TagMePlugin for a couple of
      > reasons. One is that it stores the tags in non-standard ways which
      > means that they can not be searched using normal Foswiki searches.
      > Also, that plugin has not been updated in several years and I was
      > noticing yesterday that it has a number of outstanding bug reports.
      >
      > [UTF-8?]I’d be very interested in hearing more about how you converted
      > your emails into Foswiki topics. If there was any possibility of
      > your method being generalized, I think a lot of folks would be
      > interested in hearing about it and possibly even creating a Foswiki
      > contrib for this purpose.
      >
      > Good luck with your project!
      > Lynnwood Brown
      >
      > > On May 30, 2015, at 8:26 AM, Bram van Oosterhout <bram@van-
      oosterhout.org> wrote:
      > >
      > > Hi,
      > >
      > > I am looking for a plugin to help me with the following application.
      > >
      > > I have multiple email archives. Some are saved in proprietary formats. I
      am
      > > able to extract these formats and create a foswiki site that has a page
      for
      > > each email with attachments.
      > >
      > > I am looking for a way to get easier access to this archive and have been
      > > thinking about:
      > >   * tag clouds by email names/topics/date
      > >   * Hierarchies of year, month, day
      > >   * Hierarchies by subject/date
      > >
      > > Other slices as I think of them
      > >
      > > Before I start experimenting, is there any advice you can offer re the
      merits
      > > of the following plugins as a starting point:
      > >   * ClassificationPlugin
      > >   * TopicClassificationAddOn
      > >   * TagCloudPlugin, TagMePlugin, TagsPlugin
      > >
      > > I have read the documentation, and believe that all these plugins require
      > > user interaction to classify/tag a topic.
      > >
      > > I intend to classify/tag each topic as it is created programmatically.
      > > Have I misunderstood the documentation?
      > > Is there a better way? Another more useful plugin?
      > >
      > > Thanks for your time.
      > >
      > > --
      > > Bram van Oosterhout
      > >
      > >
      > > --------------------------------------------------------------------------
      ----
      > > _______________________________________________
      > > Foswiki-discuss mailing list
      > > [hidden email]
      > > https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
      >
      > ----------------------------------------------------------------------------
      --
      > _______________________________________________
      > Foswiki-discuss mailing list
      > [hidden email]
      > https://lists.sourceforge.net/lists/listinfo/foswiki-discuss


      --
      Bram van Oosterhout
  • Reply | Threaded
    Open this post in threaded view
    |  
    Report Content as Inappropriate

    The start of a Mail archive application

    BramVanOosterhout
    In reply to this post by Crawford Currie
    Hi All,
    Some time ago I posted here about a mail archive application.

    I have a very rudimentary implementation together and the code is available
    in the support web. http://foswiki.org/Support/Question1628

    There are 2 parts:
       * a !MailArchiveContrib.tgz. A Foswiki application that displays
    correspondence to and/or from correspondents.
       * A mbx2wiki.tgz. A perl script and make file that take a pst or mbx
    format mail file and populate the mail archive.

    They constitute a work in progress and are by no means production ready. But
    as I will be travelling the east and west coast of the USA for the next three
    months, consider this a progress update.

    In the MailArchiveContrib.tgz file you find 5 pages of notes that describe my
    journey in developing the app. I learned a lot of new things and they may be
    helpful to others that want to develop Foswiki apps. There is a lot of good
    thinking in the WikiWorkbench. But the tool needs more expectation setting as
    part of the documentation. The 5 pages may help doing that. Although I
    believe an edit, if not a rewrite is essential. I will get back to that, but
    playing with the MailArchive, I get more ideas on making that more functional
    and approachable. The edit too, will have to wait until after my trip.

    Hope this is useful to you. And if not; wait for the next instalment!


    Bram van Oosterhout.

    ------------------------------------------------------------------------------
    _______________________________________________
    Foswiki-discuss mailing list
    [hidden email]
    https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
    Loading...