Modtimes on topic files can screw up SEARCH...

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Modtimes on topic files can screw up SEARCH...

Raymond Lutz
I just ran across a wrinkle as I worked to update a foswiki install with
1.0.5.

If you use SEARCH with order="modified" then Search.pm relies on the
idea that the modification time of your topic files actually matches the
metadata in the file. Of course it is a lot easier to sort on filesystem
metadata than to have to parse into each topic file to do the sort, and
that is why the code still does it that way, apparently.

Anywho, I *ucked up and overwrote my data files with some with bad
dates, i.e. all today. So that screws up the search, and I was using
that exact search to find recently modified topics.

I documented this here:
http://www.foswiki.org/bin/view/Tasks/Item1669

and I included a small utility (attached to that topic) that will update
all the modification dates in a foswiki data directory. That settles
that. You may want to run it on your install to correct the dates. I'm
wondering, first of all, does the system automatically correct these
modification dates to match the foswiki metadata, (ouch, that means I
wasted my time) and if not, then perhaps we should couple a script like
the one I provided with a cron job and provide it with the install.

--Raymond

--
---------------------------------------
Raymond Lutz
Cognisys, Inc.                
1010 Old Chase Ave., Bldg B            
El Cajon (San Diego Cty), CA 92020 USA
Voice 619-447-3246
http//www.cognisys.com


------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Modtimes on topic files can screw up SEARCH...

Martin Cleaver
Raymond:

If you are interested in cron type things, there may be a case for 

Initial implementation of SchedulerContribhttp://develop.twiki.org/~develop/cgi-bin/view/Bugs/Item2070
  
As per http://twiki.org/cgi-bin/view/Codev/CronCentre 

Regards, 

   Martin
--
[hidden email]
http://twitter.com/mrjcleaver
+1 416-786-6752 (GMT-5)



On Thu, May 28, 2009 at 8:11 PM, Raymond Lutz <[hidden email]> wrote:
I just ran across a wrinkle as I worked to update a foswiki install with
1.0.5.

If you use SEARCH with order="modified" then Search.pm relies on the
idea that the modification time of your topic files actually matches the
metadata in the file. Of course it is a lot easier to sort on filesystem
metadata than to have to parse into each topic file to do the sort, and
that is why the code still does it that way, apparently.

Anywho, I *ucked up and overwrote my data files with some with bad
dates, i.e. all today. So that screws up the search, and I was using
that exact search to find recently modified topics.

I documented this here:
http://www.foswiki.org/bin/view/Tasks/Item1669

and I included a small utility (attached to that topic) that will update
all the modification dates in a foswiki data directory. That settles
that. You may want to run it on your install to correct the dates. I'm
wondering, first of all, does the system automatically correct these
modification dates to match the foswiki metadata, (ouch, that means I
wasted my time) and if not, then perhaps we should couple a script like
the one I provided with a cron job and provide it with the install.

--Raymond

--
---------------------------------------
Raymond Lutz
Cognisys, Inc.
1010 Old Chase Ave., Bldg B
El Cajon (San Diego Cty), CA 92020 USA
Voice 619-447-3246
http//www.cognisys.com


------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss


------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Modtimes on topic files can screw up SEARCH...

Michael Daum-2
In reply to this post by Raymond Lutz
On Friday 29 May 2009, Raymond Lutz wrote:
> If you use SEARCH with order="modified" then Search.pm relies on the
> idea that the modification time of your topic files actually matches the
> metadata in the file. Of course it is a lot easier to sort on filesystem
> metadata than to have to parse into each topic file to do the sort, and
> that is why the code still does it that way, apparently.

Have a look at the DBCache* extention family. These do cache metadata
extracted from topic texts in a way it can be used efficiently - i.e. sorting.

        Michael.

--
-- Michael Daum Consulting
-- Your Wiki Software Partner
-- WikiRing Consultant
-- http://michaeldaumconsulting.com
-- Tel: +49 (0)40 21 99 27 51

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Modtimes on topic files can screw up SEARCH...

Raymond Lutz
In reply to this post by Martin Cleaver
This sort of function could either be part of an administrators control panel, or perhaps activated on an occasional basis, such as once a month, just to clean up the site. Since no one suggested that the fixup of the modification dates is currently being done, the simple script I wrote could be provided as a tool in the admin tool box, or perhaps done only upon install of an upgrade (hopefully in a must more robust upgrade facility).

THX

--Raymond

Martin Cleaver wrote:
Raymond:

If you are interested in cron type things, there may be a case for 

Initial implementation of SchedulerContribhttp://develop.twiki.org/~develop/cgi-bin/view/Bugs/Item2070
  
As per http://twiki.org/cgi-bin/view/Codev/CronCentre 

Regards, 

   Martin
--
[hidden email]
http://twitter.com/mrjcleaver
+1 416-786-6752 (GMT-5)



On Thu, May 28, 2009 at 8:11 PM, Raymond Lutz <[hidden email]> wrote:
I just ran across a wrinkle as I worked to update a foswiki install with
1.0.5.

If you use SEARCH with order="modified" then Search.pm relies on the
idea that the modification time of your topic files actually matches the
metadata in the file. Of course it is a lot easier to sort on filesystem
metadata than to have to parse into each topic file to do the sort, and
that is why the code still does it that way, apparently.

Anywho, I *ucked up and overwrote my data files with some with bad
dates, i.e. all today. So that screws up the search, and I was using
that exact search to find recently modified topics.

I documented this here:
http://www.foswiki.org/bin/view/Tasks/Item1669

and I included a small utility (attached to that topic) that will update
all the modification dates in a foswiki data directory. That settles
that. You may want to run it on your install to correct the dates. I'm
wondering, first of all, does the system automatically correct these
modification dates to match the foswiki metadata, (ouch, that means I
wasted my time) and if not, then perhaps we should couple a script like
the one I provided with a cron job and provide it with the install.

--Raymond

--
---------------------------------------
Raymond Lutz
Cognisys, Inc.
1010 Old Chase Ave., Bldg B
El Cajon (San Diego Cty), CA 92020 USA
Voice 619-447-3246
http//www.cognisys.com


------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss


------------------------------------------------------------------------------ Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com

_______________________________________________ Foswiki-discuss mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/foswiki-discuss

-- 
---------------------------------------
Raymond Lutz
Cognisys, Inc.                
1010 Old Chase Ave., Bldg B             
El Cajon (San Diego Cty), CA 92020 USA
Voice 619-447-3246
http//www.cognisys.com

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Modtimes on topic files can screw up SEARCH...

Crawford Currie
Hi Raymond,

Raymond Lutz wrote:
This sort of function could either be part of an administrators control panel, or perhaps activated on an occasional basis, such as once a month, just to clean up the site. Since no one suggested that the fixup of the modification dates is currently being done, the simple script I wrote could be provided as a tool in the admin tool box, or perhaps done only upon install of an upgrade (hopefully in a must more robust upgrade facility).
By all means add it to the 'tools' subdirectory, along with the other admin scripts provided to ease the life of admins. But please, do us a favour, and make sure it is thoroughly documented! System.CommandAndCgiScripts is the conventional place for such documentation.

Note that as we work more on abstracting the store layer and supporting different store implementations, this sort of fixup script should become less of a requirement. But we're not there yet :-(

Regards,

C.

-- 
Crawford Currie
- C-Dot Consultants http://c-dot.co.uk
- A WikiRing Partner http://wikiring.com
- landline: +44-1606-330-242
- mobile: +44-7837-877-956
- public key http://keyserver.veridis.com:11371/search?q=Crawford+Currie

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss
Reply | Threaded
Open this post in threaded view
|

Re: Modtimes on topic files can screw up SEARCH...

Raymond Lutz
Crawford:

I will certainly document this and fix it up so it can be a good tool for the tools directory (now that I know it has not already been done!!) I also will add the functionality of to (optionally) delete stale semaphores, which is a documented bug in RCS.

A catastrophe (e.g. a system crash) can cause RCS to leave behind a semaphore file that causes later invocations of RCS to claim that the RCS file is in use. To fix this, remove the semaphore file. A semaphore file's name typically begins with , or ends with _.
See http://foswiki.org/Development/VarCachePlugin when I encountered this problem and thought it was part of VarCachePlugin. Since the tool already explores all of the data directory tree, it will be an easy to make that an option as well to at least provide a means to deal with this known RCS bug. (If I knew more about the operation of RCS, I might propose that this be added to the RCSwrap-per, i.e. if an error is received that the file is already in use, then it may be appropriate to silently delete the semaphore file.)

As a result, I will title the utility more generally, so we may include once-in-a-while maintenance functions such as these.

--Raymond

Crawford Currie wrote:
Hi Raymond,

Raymond Lutz wrote:
This sort of function could either be part of an administrators control panel, or perhaps activated on an occasional basis, such as once a month, just to clean up the site. Since no one suggested that the fixup of the modification dates is currently being done, the simple script I wrote could be provided as a tool in the admin tool box, or perhaps done only upon install of an upgrade (hopefully in a must more robust upgrade facility).
By all means add it to the 'tools' subdirectory, along with the other admin scripts provided to ease the life of admins. But please, do us a favour, and make sure it is thoroughly documented! System.CommandAndCgiScripts is the conventional place for such documentation.

Note that as we work more on abstracting the store layer and supporting different store implementations, this sort of fixup script should become less of a requirement. But we're not there yet :-(

Regards,

C.

-- 
Crawford Currie
- C-Dot Consultants http://c-dot.co.uk
- A WikiRing Partner http://wikiring.com
- landline: +44-1606-330-242
- mobile: +44-7837-877-956
- public key http://keyserver.veridis.com:11371/search?q=Crawford+Currie
  

------------------------------------------------------------------------------ Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT is a gathering of tech-side developers & brand creativity professionals. Meet the minds behind Google Creative Lab, Visual Complexity, Processing, & iPhoneDevCamp as they present alongside digital heavyweights like Barbarian Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com

_______________________________________________ Foswiki-discuss mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/foswiki-discuss

-- 
---------------------------------------
Raymond Lutz
Cognisys, Inc.                
1010 Old Chase Ave., Bldg B             
El Cajon (San Diego Cty), CA 92020 USA
Voice 619-447-3246
http//www.cognisys.com

------------------------------------------------------------------------------
Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
is a gathering of tech-side developers & brand creativity professionals. Meet
the minds behind Google Creative Lab, Visual Complexity, Processing, &
iPhoneDevCamp as they present alongside digital heavyweights like Barbarian
Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com 
_______________________________________________
Foswiki-discuss mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/foswiki-discuss