MailerContrib WebNotify emails have a broken diff

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

MailerContrib WebNotify emails have a broken diff

Heath Raftery-2
Foswikians,

I've been trying to track down an issue we've had with WebNotify emails
since day dot. The emails get sent just fine and most of the content is
fine, but the diff summary is not.

It always seems to be <del>Web.Topic</del><ins>Web.Topic ... The entire
current revision of the topic</ins>.

Maybe as if it was diff'ing it with a very old rev?

Based on assistance from #foswiki, I've confirmed that the .changes file
is correct and that nothing strange is happening to work_areas. If I
click on the diff link in the email, the browser rendered diff is fine
(after logging in).

I've manually traced the program flow from WebNotify to MailerContrib to
mailnotify.tmpl to mailnotify.pattern.tmpl to MailerContrib.pm to
Change.pm to Func.pm to Meta.pm, but I don't have a debug environment so
I'm not sure where in that chain things are going wrong. Given that the
rest of the email is correct I assume it's somewhere within
Func::summariseChanges.

Foswiki version v1.1.9 under mod_perl/2.0.8, Plugin API version 2.2.
MailerContribPlugin (2.60, 2.60)

Any ideas?

Heath

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: MailerContrib WebNotify emails have a broken diff

Crawford Currie
Hi Heath,

So many possibilities.

First, Foswiki uses the CPAN module Algorithm::Diff to compare the text
of the versions, so check that you have the latest Algorithm::Diff
installed.

Second, check your configuration and the content of the topic. What
charset are you using? Is the topic correctly encoded?

Third, check that old revs are being retrieved correctly. Make sure that
when you look at old revs of the topic in the browser, you see what you
expect to see (the old revs of the actual topic that MailerContrib
should be reporting on).

Permissions next. Does the user who runs the notify cron have read
access to the topics it's trying to compare? All versions of them? In
Foswiki? On the file system?

All good so far? OK, so the finger of shame is slowly rotating to point
at the store. What store are you using? On what platform?

BTW "a debug environment" is nothing special. Most of us debug using
"print STDERR" statements added to the code. The output is printed to
the Apache log file. What you are interested in is
Foswiki::Meta::summariseChanges, where it should be called with $orev !=
$nrev ($orev is the previous revision number and $nrev the new one).
Within that function, compare $ostring and $nstring once they have both
been retrieved.

Regards,

C.

On 27/11/14 23:14, Heath Raftery wrote:

> Foswikians,
>
> I've been trying to track down an issue we've had with WebNotify emails
> since day dot. The emails get sent just fine and most of the content is
> fine, but the diff summary is not.
>
> It always seems to be <del>Web.Topic</del><ins>Web.Topic ... The entire
> current revision of the topic</ins>.
>
> Maybe as if it was diff'ing it with a very old rev?
>
> Based on assistance from #foswiki, I've confirmed that the .changes file
> is correct and that nothing strange is happening to work_areas. If I
> click on the diff link in the email, the browser rendered diff is fine
> (after logging in).
>
> I've manually traced the program flow from WebNotify to MailerContrib to
> mailnotify.tmpl to mailnotify.pattern.tmpl to MailerContrib.pm to
> Change.pm to Func.pm to Meta.pm, but I don't have a debug environment so
> I'm not sure where in that chain things are going wrong. Given that the
> rest of the email is correct I assume it's somewhere within
> Func::summariseChanges.
>
> Foswiki version v1.1.9 under mod_perl/2.0.8, Plugin API version 2.2.
> MailerContribPlugin (2.60, 2.60)
>
> Any ideas?
>
> Heath
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Foswiki-discuss mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
>


--
Crawford Currie - Owner, C-Dot Consultants - landline: +44-1606-330-242
- mobile: +44-7837-877-956 - skype: cdot-uk - public key
http://pgp.mit.edu/pks/lookup?op=vindex&search=0x0CD6BAE648697B60


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: MailerContrib WebNotify emails have a broken diff

Heath Raftery-2
Thank you for the response. Please see my responses inline below.

On 28/11/2014 6:05 PM, Crawford Currie wrote:
> So many possibilities.

I was hoping the fact that the "Compare revisions" function in each
page's history works fine would eliminate some of the possibilities. I
guess the chain is different enough that that's not really the case.

> First, Foswiki uses the CPAN module Algorithm::Diff to compare the text
> of the versions, so check that you have the latest Algorithm::Diff
> installed.

Installed: 1.1902. 1.1903 exists but has no Ubuntu package and only
consists of spelling fixes anyway.

> Second, check your configuration and the content of the topic. What
> charset are you using? Is the topic correctly encoded?

$Foswiki::cfg{Site}{CharSet} = 'iso-8859-1';

The txt files for each topic in /data look like normal ASCII text files.

> Third, check that old revs are being retrieved correctly. Make sure that
> when you look at old revs of the topic in the browser, you see what you
> expect to see (the old revs of the actual topic that MailerContrib
> should be reporting on).

Looks fine.

> Permissions next. Does the user who runs the notify cron have read
> access to the topics it's trying to compare? All versions of them? In
> Foswiki? On the file system?

It's root's crontab which has this entry:

0 0 * * * cd /mnt/system/wiki_html && perl -I bin tools/mailnotify -q

All the topic files are owned by www-data with permissions: rwxr-xr-x,
so readable by all. How do I see what Foswiki user is used?

> All good so far? OK, so the finger of shame is slowly rotating to point
> at the store. What store are you using? On what platform?

$Foswiki::cfg{Store}{Implementation} = 'Foswiki::Store::RcsLite';

Ubuntu 14.04.1 LTS.

> BTW "a debug environment" is nothing special. Most of us debug using
> "print STDERR" statements added to the code. The output is printed to
> the Apache log file.

Ah, good to know. In fact when debugging mailnotify it's even easier -
print statements go to the command line.

> What you are interested in is
> Foswiki::Meta::summariseChanges, where it should be called with $orev !=
> $nrev ($orev is the previous revision number and $nrev the new one).
> Within that function, compare $ostring and $nstring once they have both
> been retrieved.

Got it! In my Sandbox and Main webs all works perfectly. In our primary
web, $orev, $nrev and $nstring are all good, but $ostring is just
"WebName.WebHome" where WebName is the name of our primary web. And for
some reason summariseChanges is called twice with the same output...

So far I've discovered $ostring is invalid because
$oldTopicObject->haveAccess('VIEW') is false. I'll continue to debug to
find out why but thank you for your help so far.

Regards,
Heath


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: MailerContrib WebNotify emails have a broken diff

George Clark-2
Heath,

One thing that might cause you troubles later...   mailnotify does write
some files, in working/work_areas,  and by setting your crontab under
root, you'll possibly change ownership in those files.   Same for any
logs, etc.

It's best to run foswiki cron based tasks under the www-data user,  ( or
whatever user  the webserver / cgi uses to read/write foswiki files).

George

On 11/30/2014 07:50 PM, Heath Raftery wrote:
> It's root's crontab which has this entry:
>
> 0 0 * * * cd /mnt/system/wiki_html && perl -I bin tools/mailnotify -q
>
> All the topic files are owned by www-data with permissions: rwxr-xr-x,
> so readable by all. How do I see what Foswiki user is used?
>
>> >


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: MailerContrib WebNotify emails have a broken diff

Heath Raftery-2
Thank you George. I've moved the cron entry to /etc/cron.d/wiki so I can
specify www-data as the user. I decided this was better than using "sudo
-u www-data crontab -e" since I'd probably forget I put it there.

Can I suggest the documentation at

http://foswiki.org/Extensions/MailerContrib#Setting_up_a_cron_job_40s_41

be updated? There's no advice on what user to use for the cron job.

On 1/12/2014 12:07 PM, George Clark wrote:

> Heath,
>
> One thing that might cause you troubles later...   mailnotify does write
> some files, in working/work_areas,  and by setting your crontab under
> root, you'll possibly change ownership in those files.   Same for any
> logs, etc.
>
> It's best to run foswiki cron based tasks under the www-data user,  ( or
> whatever user  the webserver / cgi uses to read/write foswiki files).
>
> George
>
> On 11/30/2014 07:50 PM, Heath Raftery wrote:
>> It's root's crontab which has this entry:
>>
>> 0 0 * * * cd /mnt/system/wiki_html && perl -I bin tools/mailnotify -q
>>
>> All the topic files are owned by www-data with permissions: rwxr-xr-x,
>> so readable by all. How do I see what Foswiki user is used?
>>


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: MailerContrib WebNotify emails have a broken diff

Heath Raftery-2
In reply to this post by Heath Raftery-2
I've completed my investigation and found the heart of the problem. I've
only implemented a workaround though because I can't see how it's
supposed to work. Perhaps someone can suggest a proper fix:

In Meta.pm line 3384, it calls:

$oldTopicObject->haveAccess('VIEW')

This returns false because at this point $session->{user} is
BaseUserMapping_666, and because I have a ALLOWWEBVIEW set for this web,
it doesn't find that user listed there.

Thus, the oldTopicObject is blanked and the diff continues by comparing
the current rev to a blank version.

My workaround is to remove that blanking code. Alternatively, adding
"BaseUserMapping_666" to ALLOWWEBVIEW also worked.

I can't understand why the check is done here in the first place -
access to the current rev is also checked a few lines earlier, which
also fails, but then it goes and grabs the topic text anyway.

More strangely, much earlier in WebNotify.pm:260, checkAccessPermission
is called using the actual subscriber user and the topic, so it seems to
me that permissions have already been checked.

My guess is that Meta is designed to be called when a user is logged in
through the browser, so $session->{user} would normally be the current
user. Even so, it seems strange to retrieve the topic text, and then
check whether VIEW access is available.

In summary: in Foswiki 1.1.9 and MailerContrib 2.60, setting a
ALLOWWEBVIEW in WebPreferences breaks the diff section of the mailnotify
emails because Meta::summariseChanges calls Meta::haveAccess which
returns false when BaseUserMapping_666 is not found in the ALLOWWEBVIEW
list.

Regards,
Heath

On 1/12/2014 11:50 AM, Heath Raftery wrote:

> Thank you for the response. Please see my responses inline below.
>
> On 28/11/2014 6:05 PM, Crawford Currie wrote:
>> So many possibilities.
>
> I was hoping the fact that the "Compare revisions" function in each
> page's history works fine would eliminate some of the possibilities. I
> guess the chain is different enough that that's not really the case.
>
>> First, Foswiki uses the CPAN module Algorithm::Diff to compare the text
>> of the versions, so check that you have the latest Algorithm::Diff
>> installed.
>
> Installed: 1.1902. 1.1903 exists but has no Ubuntu package and only
> consists of spelling fixes anyway.
>
>> Second, check your configuration and the content of the topic. What
>> charset are you using? Is the topic correctly encoded?
>
> $Foswiki::cfg{Site}{CharSet} = 'iso-8859-1';
>
> The txt files for each topic in /data look like normal ASCII text files.
>
>> Third, check that old revs are being retrieved correctly. Make sure that
>> when you look at old revs of the topic in the browser, you see what you
>> expect to see (the old revs of the actual topic that MailerContrib
>> should be reporting on).
>
> Looks fine.
>
>> Permissions next. Does the user who runs the notify cron have read
>> access to the topics it's trying to compare? All versions of them? In
>> Foswiki? On the file system?
>
> It's root's crontab which has this entry:
>
> 0 0 * * * cd /mnt/system/wiki_html && perl -I bin tools/mailnotify -q
>
> All the topic files are owned by www-data with permissions: rwxr-xr-x,
> so readable by all. How do I see what Foswiki user is used?
>
>> All good so far? OK, so the finger of shame is slowly rotating to point
>> at the store. What store are you using? On what platform?
>
> $Foswiki::cfg{Store}{Implementation} = 'Foswiki::Store::RcsLite';
>
> Ubuntu 14.04.1 LTS.
>
>> BTW "a debug environment" is nothing special. Most of us debug using
>> "print STDERR" statements added to the code. The output is printed to
>> the Apache log file.
>
> Ah, good to know. In fact when debugging mailnotify it's even easier -
> print statements go to the command line.
>
>> What you are interested in is
>> Foswiki::Meta::summariseChanges, where it should be called with $orev !=
>> $nrev ($orev is the previous revision number and $nrev the new one).
>> Within that function, compare $ostring and $nstring once they have both
>> been retrieved.
>
> Got it! In my Sandbox and Main webs all works perfectly. In our primary
> web, $orev, $nrev and $nstring are all good, but $ostring is just
> "WebName.WebHome" where WebName is the name of our primary web. And for
> some reason summariseChanges is called twice with the same output...
>
> So far I've discovered $ostring is invalid because
> $oldTopicObject->haveAccess('VIEW') is false. I'll continue to debug to
> find out why but thank you for your help so far.
>
> Regards,
> Heath


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: MailerContrib WebNotify emails have a broken diff

George Clark-2
I can confirm that I've recreated this locally. Task http://foswiki.org/Tasks/Item13126 opened.

 1) Add user to WebNotify topic of two webs,  one without restrictions, one with ALLOWWEBVIEW = SomeUser
 2) Create two identical topics.  One in web without view restriction,  one in web with ALLOWWEBVIEW = SomeUser.
 3) run mailnotify to flush the notifications
 4) Edit topics, make a minor change, and force save as a new rev.
 5) Run mailnotify.

The Diff of the topic in the web with view restrictions is wrong. 

Restricted web diff
Restricted.SomeNewTopic
Restricted.SomeNewTopic
Test topic

asdf


Unrestricted web diff
asdf
asdf
asdf somechange  asdf
asdf
wq34
agd BAR BAR




On 11/30/2014 10:12 PM, Heath Raftery wrote:
I've completed my investigation and found the heart of the problem. I've 
only implemented a workaround though because I can't see how it's 
supposed to work. Perhaps someone can suggest a proper fix:

In Meta.pm line 3384, it calls:

$oldTopicObject->haveAccess('VIEW')

This returns false because at this point $session->{user} is 
BaseUserMapping_666, and because I have a ALLOWWEBVIEW set for this web, 
it doesn't find that user listed there.

Thus, the oldTopicObject is blanked and the diff continues by comparing 
the current rev to a blank version.

My workaround is to remove that blanking code. Alternatively, adding 
"BaseUserMapping_666" to ALLOWWEBVIEW also worked.

I can't understand why the check is done here in the first place - 
access to the current rev is also checked a few lines earlier, which 
also fails, but then it goes and grabs the topic text anyway.

More strangely, much earlier in WebNotify.pm:260, checkAccessPermission 
is called using the actual subscriber user and the topic, so it seems to 
me that permissions have already been checked.

My guess is that Meta is designed to be called when a user is logged in 
through the browser, so $session->{user} would normally be the current 
user. Even so, it seems strange to retrieve the topic text, and then 
check whether VIEW access is available.

In summary: in Foswiki 1.1.9 and MailerContrib 2.60, setting a 
ALLOWWEBVIEW in WebPreferences breaks the diff section of the mailnotify 
emails because Meta::summariseChanges calls Meta::haveAccess which 
returns false when BaseUserMapping_666 is not found in the ALLOWWEBVIEW 
list.

Regards,
Heath

On 1/12/2014 11:50 AM, Heath Raftery wrote:
Thank you for the response. Please see my responses inline below.

On 28/11/2014 6:05 PM, Crawford Currie wrote:
So many possibilities.
I was hoping the fact that the "Compare revisions" function in each
page's history works fine would eliminate some of the possibilities. I
guess the chain is different enough that that's not really the case.

First, Foswiki uses the CPAN module Algorithm::Diff to compare the text
of the versions, so check that you have the latest Algorithm::Diff
installed.
Installed: 1.1902. 1.1903 exists but has no Ubuntu package and only
consists of spelling fixes anyway.

Second, check your configuration and the content of the topic. What
charset are you using? Is the topic correctly encoded?
$Foswiki::cfg{Site}{CharSet} = 'iso-8859-1';

The txt files for each topic in /data look like normal ASCII text files.

Third, check that old revs are being retrieved correctly. Make sure that
when you look at old revs of the topic in the browser, you see what you
expect to see (the old revs of the actual topic that MailerContrib
should be reporting on).
Looks fine.

Permissions next. Does the user who runs the notify cron have read
access to the topics it's trying to compare? All versions of them? In
Foswiki? On the file system?
It's root's crontab which has this entry:

0 0 * * * cd /mnt/system/wiki_html && perl -I bin tools/mailnotify -q

All the topic files are owned by www-data with permissions: rwxr-xr-x,
so readable by all. How do I see what Foswiki user is used?

All good so far? OK, so the finger of shame is slowly rotating to point
at the store. What store are you using? On what platform?
$Foswiki::cfg{Store}{Implementation} = 'Foswiki::Store::RcsLite';

Ubuntu 14.04.1 LTS.

BTW "a debug environment" is nothing special. Most of us debug using
"print STDERR" statements added to the code. The output is printed to
the Apache log file.
Ah, good to know. In fact when debugging mailnotify it's even easier -
print statements go to the command line.

What you are interested in is
Foswiki::Meta::summariseChanges, where it should be called with $orev !=
$nrev ($orev is the previous revision number and $nrev the new one).
Within that function, compare $ostring and $nstring once they have both
been retrieved.
Got it! In my Sandbox and Main webs all works perfectly. In our primary
web, $orev, $nrev and $nstring are all good, but $ostring is just
"WebName.WebHome" where WebName is the name of our primary web. And for
some reason summariseChanges is called twice with the same output...

So far I've discovered $ostring is invalid because
$oldTopicObject->haveAccess('VIEW') is false. I'll continue to debug to
find out why but thank you for your help so far.

Regards,
Heath

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: MailerContrib WebNotify emails have a broken diff

Crawford Currie
In reply to this post by Heath Raftery-2
Thanks Heath for your research, high quality bug reports like this are
greatly appreciated.

On 01/12/14 03:12, Heath Raftery wrote:
> More strangely, much earlier in WebNotify.pm:260, checkAccessPermission
> is called using the actual subscriber user and the topic, so it seems to
> me that permissions have already been checked.
*sigh* several revs of MailerContrib ago, we used to 'spoof' the current
user so that all lower-level access control checks operated as the
mailing user. There were other access control issues that made that a
really bad idea, so I changed it - obviously without working through all
the possible implications.

The solution is for the call to summariseChanges to accept an optional
parameter of the CUID of the calling user (defaulting it to the default
user). That way we maintain the API, but support checking for a
different user.

C.

--
Crawford Currie - Owner, C-Dot Consultants - landline: +44-1606-330-242
- mobile: +44-7837-877-956 - skype: cdot-uk - public key
http://pgp.mit.edu/pks/lookup?op=vindex&search=0x0CD6BAE648697B60


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss