Feature Proposal: extender.pl should create a backup archive for extensions.

Motivation

• The current strategy of renaming files to the .bak suffix scatters backup files throughout the file system, leaving the admin to have to clean them up later.
• If the installer is run twice, the backup is destroyed
• Recovery is manual and difficult especially for extensions with large numbers of files.
• rcs files are not backed up
• On trunk, bin/configure doesn't make backups.

Description and Documentation

As part of Tasks.Item8640, tools/extender.pl and Configure/UIs/EXTEND.pm are being refactored to put common functions in Configure/Util.pm. It will also save the package data from the installer file into the working directory for future use. A new function - Util::backupExtension - will create a backup of the to-be-installed Extension. if Archive::Tar or Archive::Zip are available, add each file to an archive. For the data directory, also add the .txt,v file to the backup if it exists. If no archive functions are available on the system, mirror the extension files into a sub-directory of the backup location.

Add the following file structure under working

• working
  • configure
    • pkgdata - Contains copy of manifest/dependencies file extracted from the _installer package for each installed extension
      • File: [Extension].[version].pgkdata (contains extracted DATA segment from installer.)
    • backups - Contains backup files for work done by configure
      • File: [Extension]_backup_timestamp.[tgz | zip]
      • Directory: [Extension]_backup_timestamp (Only created if archive software is unavailable)

The backup will be made of the current "state" of the installed extension, including:

• All files installed by the extension.
• All rcs files created locally for the extension files, so that a restore will restore the state of any local modifications.
• The manifest of the new extension will be used to build the backup.

I do not plan to implement a "restore" utility. The administrator should be able to recover a failed Extension with the following steps:

• cd [root of foswiki installation]
• tar -zxf working/configure/backups/[Extension]_backup_[timestamp]
• restart / reload web server if needed for persistent perl.

Extension backup will not copy the current LocalSite.cfg file. I recall someone talking about adding LocalSite.cfg backup to bin/configure however I haven't found the proposal for that.

This proposal does not change rcs file handling or checkin processing There are separate unrelated issues with rcs handling mentioned in the discussion below but they describe an unrelated bug.

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: GeorgeClark - 27 Feb 2010

Discussion

I've also noticed that on trunk, the bin/configure interface no longer makes the .bak files, so no backup is made of Extensions. So this capability needs to be restored either using a new tar/zip backup, or by restoring the .bak naming.

-- GeorgeClark - 27 Feb 2010

instead of leaving more cruft around in the foswiki root directory, i would instead store the zipped up version in working/work_areas/Configure/InstalledExtensions (or something like that, but definitely under working/work_areas).

if an archiver is not available, i would not create .bak files as we do now; instead, simply move the existing files to a directory tree under what would have been the zip filename.

Tasks.Item1629 can disappear if/when this proposal is realised.

is configure going to know about existing "backups" and how to restore those versions or are admins expected to perform restores using the command line?

i think that the installer will have to save MANIFEST. consider the case where a newer release no longer contains a file that used to be distributed (as recently happened with JHotDrawPlugin).

-- WillNorris - 27 Feb 2010

working/work_areas/Configure/ works for me. Maybe ExtensionsBackup as the directory? Should the backup location be configurable in LocalSite.cfg?

Copying to a tree named with the Extension_backup_timestamp sounds reasonable if archivers are not found.

I was not planning on implementing a restore. That gets too deep into the internals of FindExtensions.

If the installer saves the prior manifest, should the backup try to use the old manifest? Installer can copy the MANIFEST file into working/work_areas/Configure/ExtensionManifests?

I would not have the backup process actually remove the old files. leaving them around is cleaner in the event that the install fails. If an installer wants to clean up old files, we might want to document a sample POSTINSTALL routine for cleanup. I was also thinking that it would be useful if the manifest were expanded to include a Digest (MD5 or SHA) of each file. filename, permissions, digest, description This would potentially allow the installer to detect local modifications prior to installation, and to also report on which files will change. But that is beyond the scope of what I wanted to do here.

-- GeorgeClark - 27 Feb 2010

> I was also thinking that it would be useful if the manifest were expanded to include a Digest (MD5 or SHA) of each file.

George - take a look at http://twiki.org/cgi-bin/view/Plugins/TWikiReleaseTrackerPlugin - it does that, and let's you see how exactly how an admininstrator has changed everything in an install. See http://foswiki.org/Tasks/Item437

It has not been updated for Foswiki but was a lot of work and is pretty good code.

-- MartinCleaver - 06 Mar 2010

Additionally, if you are wondering how to make backups of the Plugin .pm, gif, .txt files?, why not just check them into RCS?

This would make Foswiki more orthogonal

-- MartinCleaver - 09 Mar 2010

Thanks Martin - I'll check out the ReleaseTrackerPlugin.

I was looking for a backup approach that would allow a simple recovery from a Extension installation by a simple untar / unzip of the old extension. RCS would still involve individual updates to revert changes. Also we don't currently have rcs data on the modules in the /lib directory, etc. so it would be a big change to start doing revision control on these other directories.

As far as checkin - that's a fairly high overhead operation. Currently the shell installer does do checkins, but the web installer does not. it's a bit of a mixed bag. I'm trying to fix up all this as Tasks.Item8640, and make sure that Shell and Web installers give identical results. My thoughts are to handle checkin as follows:

• If a file does not exist, don't do a checkin, just move/copy - lowest overhead.
• If a file exists along with a rcs *,v file, always do checkin - might suggest local changes exist.
• If a file exists without rcs *,v file, follow the noci setting in the manifest -
  • If set to noci - just move the file over the existing without a checkin
  • If checkin requested - do the checkin.

Once this proposal is approved, I'll add the backup as part of Tasks.Item8640

-- GeorgeClark - 10 Mar 2010

I was just going to set this to approved.

BUT. I lack an updated SPEC of the proposal. The discussion has not shown any discontent with the IDEA, but there are some implementation details that are now not clear.

• Where do the back-up tar files go?
• What files are stored in them? (MANIFEST of the current plugin version is NOT available today)
• How is the admin actually going to use this?

George can you wrap up the spec at the top of the proposal?

From a personal point of view I look forward to not having 300-500 .bak files created when I upgrade TinyMCEPlugin and other plugins with 100s of files in the pub dir. But I would not want them suddenly having ,v files. I have no use of 300-500 ,v files in the pub dir of an extension. I would never in practical use be able to get back to previous version.

The only files I see reasonable to bake back-ups of is the .txt files of extensions. And only for extensions that have settings in the .txt file.

If the .bak hell is removed in trunk then I welcome this and would not want it back.

-- KennethLavrsen - 12 Mar 2010

I've updated the Description above with implementation details to answer your questions.

The backup will be of all files that might be changed by a new version of the extension. I considered doing partial backups, but given that we currently don't make it easy to install a previous version of an extension, I believe that the backup should provide a simple way for the admin to step back to exactly what was there prior to starting the install of the extension. I think that this also clarifies the question of should the backup use the previous manifest (if available) or the current manifest. Since it is possible that the extension might overlay existing files that were previously not contained in the extension, the backup should be of any files installed by the new extension. (If we add the ability for an extension to remove files, then this decision should be reconsidered, or the manifest needs a new "type" of file so request that it be removed - which might actually be the cleanest solution.)

Regarding comment on the ,v files, The "shell" installer today creates a ,v file for each data or pub file unless the file is flagged as noci in the manifest. The bin/configure installer today does not checkin any files. My changes to the installer as part of Tasks.Item8640 will give consistent results regardless of the installation method. It will cause the web installer to create some ,v files, but will reduce the number created by the shell installer.

• New file, no existing history - no checkin. (shell would have created a checkin)
• Existing file, no history - checkin unless flagged noci - manifest in control. (web would have missed the checkin)
  • This allows the user to view the changes installed by an extension
• Existing file with existing ,v file, Always checkin (web would have resulted in out-of-sync .txt and .txt,v )
  • Very important to avoid auto-revert caused by the new Store implementation which trusts the rcs file over the txt file.
• Attachments in pub will be treated under the same rules.

I didn't put any of the above comments on the rcs files into a proposal because I considered the inconsistent behavior between shell and web installers a bug. Plus the new store behavior on trunk could cause major issues if .txt file is changed without updating the existing .txt,v file.

-- GeorgeClark - 12 Mar 2010

It was the uncertainty around RCS of files in the pub dir that made me raise concern.

This is now resolved. I lift my concern. And since we are past the 14 days it means passed with consensus

-- KennethLavrsen - 15 Mar 2010