Foswiki Upgrade Guide

This guide covers upgrading from a previous version of Foswiki to Foswiki 2.x.

This upgrade procedure is used to upgrade a to a new major version of Foswiki (1.x to 2.x) or from TWiki. Generally, upgrades of a minor Foswiki version (2.0 to 2.1), or Foswiki patch releases (2.1.3 to 2.1.4) can be done by using the -upgrade- version of the Foswiki package, and follow the Release Notes for the new release.

Overview

Foswiki is a fork from TWiki 4.2.3. (TWiki^® is a trademark of Peter Thoeny.) Note that newer versions of Foswiki have not directly tracked newer versions of TWiki, and some divergence has occurred. Foswiki 2.0 diverges significantly in some areas.

If you are upgrading from TWiki to Foswiki, please refer to Foswiki:Support.UpgradingFromOlderTWikiReleases.

Upgrade requirements

• Please review the Foswiki:System.AdminSkillsAssumptions before you upgrade your site.
• Carefully review Foswiki:System.SystemRequirements. Foswiki no longer ships with CPAN libraries. CPAN dependencies must be installed prior to upgrade.
• Before upgrading, a backup of your topics is strongly recommended.
• Once the upgrade has been applied, an existing earlier installation will still be able to read all the topics, but should not be used to write.

Upgrading to a new patch or minor release

To upgrade to a new patch release — for example, from Foswiki 1.1.0 to 1.1.2 — an upgrade package can be used that will not overwrite typical customizations. Unless otherwise stated in the release notes, we do not recommend upgrading between major or minor versions using the patch (For ex. 1.1.9 to 2.0). A re-installation is recommended.

For patch releases you will find a brief upgrade procedure on the download page for the release. Follow this procedure to upgrade to the patch release. It may contain important steps that are unique to each patch release (for example, some configure settings may need to be changed).

If you use the Foswiki PageCaching feature, be sure to refresh the cache after upgrading to a new Foswiki version. Visit your site with the Query parameter ?refresh=all

Upgrade procedure: upgrading to a new major version

The following is a high level view of the upgrade procedure:

1. Before the upgrade, plan for client cache management.
2. Prepare for all upgrade steps.
3. Install the new Foswiki version and configure it with the same settings as the old version.
   
   • Windows server users: Don't forget to rerun tools/rewriteshebang.pl to fix up the Perl locations
4. Install any additional extensions (Plugins) used by your old installation. Make sure to use the latest Foswiki versions.
5. Convert all the non-default webs from the old installation to the new one. (Encoding and Store changes)
6. Convert the users, groups, and site customizations from the old installation to the Main web in the new installation, including all user topics.
7. Apply preferences from the old installation.
8. Apply your site customizations: skin, logos, menu bars, forms for personal information, and so forth.
9. Validate your Wiki applications and other key functionality.
10. Switch your production site from the old installation to the new installation.

More details for each step appear in the following sections. The steps may need to be modified or otherwise tailored with specifics for your installation. In particular, you must take care to preserve any special configuration or customizations you have made, especially if you have modified any of the default software files or system topics that are contained within the installation package.

For purposes of discussion, the following conventions are used:

• <oldwiki> refers to the directory in which the old installation is located
• <newwiki> refers to the directory in which the new installation is located; it is assumed to be immediately below the root directory of your web server
• <old_users_web> refers to the web in which the user topics are located in the old installation. The default value is the Main web. The web is specified in the Store settings pane of the configure page, in the {UsersWebName} setting (visible when Expert mode is enabled).
• <old_system_web> refers to the web used for documentation and default preferences in the old installation. In Foswiki, the default value is the System web. The web is specified in the Store settings pane of the configure page, in the {SystemWebName} setting (visible when Expert mode is enabled).

After the upgrade, in the new installation, the Main web is used for user topics and site preferences, and the System web is used to hold documentation and default preferences.

The configure page mentioned in this document is accessible via your web browser at http://yourdomain/<newwiki>/bin/configure .

Before the upgrade

Managing caches of static .js, .css files:

If you are using Expires tags, (you should be!) it is very important to take the longest expiration time into consideration. Clients will locally cache JavaScript and CSS until the time expires, unless they clear their cache. There are significant changes to the JavaScript and CSS files shipped with Foswiki 2.0. Clients using locally cached data will not operate correctly.

• Prior to the upgrade, reduce the Expires tags to a short duration, for example 1 hour.
  • Examine your Apache configuration for statements like: ExpiresDefault "access plus 11 days", change it to ExpiresDefault "access plus 1 hour"
• Defer the upgrade until the longest expiration time has passed. If the longest time was 2 weeks, delay the upgrade for 2 weeks.
• Complete the upgrade.
• Once confident that further upgrades, or fallback are not required, restore the original far future expiration.

If these steps are not done, users will have to clear their browser cache to get the most recent CSS and JavaScript..

Prepare for all upgrade steps

Download the Foswiki distribution from the following location: https://foswiki.org/Download — if you are installing your extensions manually, also download them from the repository where they are stored. (Default extensions are included in the Foswiki distribution).

Review Foswiki:System.SystemRequirements and Install missing Perl modules using your local package manager or CPAN. Foswiki 2.0 no longer ships with CPAN modules. If you have access to the command line on the server, you can test for missing dependencies by running: perl tools/dependencies

Review the Release Notes and learn about the differences between your old installation and the new release to which you are upgrading. Take note of any areas that affect your site and what special steps you may need to take.

• Foswiki:System.ReleaseNotes02x01 current release notes
• Foswiki:System.ReleaseNotes02x00 if upgrading from a release prior to Foswiki 2.0.3
• Foswiki:System.ReleaseNotes01x01 if upgrading from a release prior to Foswiki 1.1.9, and
• Foswiki:System.ReleaseNotes01x00 if upgrading from an older release of Foswiki 1.0.x

Check that all the extensions (plugins, contribs, skins) used by your old installation are available with the new release. Familiarize yourself with any new behaviour that you will have to adapt to or any configuration changes you will have to perform.

• The EditTablePlugin has been deprecated and is not enabled by default on Foswiki 2.0. It is replaced by the EditRowPlugin. Only one of EditTablePlugin or EditRowPlugin should be enabled.
• Review the deprecated jQuery javascript plugins. The JQueryPlugin has several changes in available jQuery JavaScript plugins. Determine if any of these will impact your JavaScript enabled topics.

Choose the character encoding to be used in your site.

• Previous versions of Foswiki defaulted to iso-8859-1 encoding (The "Latin Alphabet 1, intended for US and Western European languages).
• Foswiki 2.0 defaults to UTF-8 encoding, which provides better support for international character sets.

Note that the 1.x versions of Foswiki make no attempt to enforce the character encoding. Topics contain whatever the user may have pasted into them. If you have windows users, it is highly likely that your old installation is using cp-1252 instead of iso-8859-1. cp-1252 contains windows specific high-ASCII characters including typographic "smart" quotes, alternative bullets and the emdash character. For more information see Foswiki:Support.Utf8MigrationConsiderations.

You must match the prior encoding, or convert old data to the new encoding if you intend to use topics created on an older version of Foswiki. There are two common use cases:

• Case 1: Your existing site is already using utf-8 encoding. Character set conversion is not needed. Proceed to chosing your store.
• Case 2: Your existing site uses the default iso-8859-1 or any other common encoding. All topics are consistent with this encoding. You have three options:
  1. Install CharsetConverterContrib and convert topics in-place on your 1.1 system. (The store implementation cannot be changed using this method.) or
  2. Use the bulk_copy.pl script to migrate your existing 1.1.x store over to Foswiki 2.0. Each topic will be converted from the 1.1.x {Site}{CharSet} encoding to the 2.0 {Store}{Encoding}. We recommend you leave 2.0 {Store}{Encoding} as undefined (utf-8). or
  3. Set the 2.0 {Store}{Encoding} to match your 1.1.x {Site}{Encoding} and copy the data into Foswiki 2.0 unmodified.
• Case 3: Your site contains a mix of encodings. This can happen if users manually paste in encoded data into topics, or topics are created / modified external to Foswiki.
  • In this case, any topics with unusual encodings will display corrupted. Use Foswiki:Extensions.CharsetConverterContrib to modify the character encoding of your Foswiki 1.1.x system in place. Changes in character encoding must be done using the RCS based store. When this tool is run with the -r (repair) option, the tool attempts to detect the encoding and can convert individual topics based upon their content. This is rather unpredictable and may require manual intervention.
  • We strongly recommend that a backup be taken before attempting to use the CharsetConverterContrib. As it modifies topics and attachments in-place, it can cause damage and data loss. The bulk_copy.pl script does not modify existing topics but is unable to handle some legacy configuratino..
  • Note that it's possible that some data cannot be cleanly converted, for ex, if the Charset encoding was changed, so that different topic revisions use different encoding. In this case you may need to remove the topic history.

Choose your desired Store. Foswiki ships with two native stores.

• RcsStoreContrib is compatible with topics created in prior versions of Foswiki.
• PlainFileStoreContrib requires that topic histories be converted to a new history format. This can be done at the same time you convert the character set. perl -I lib tools/bulk_copy.pl --help for more information on conversion.
  • Note: If your old data is on TWiki, then bulk_copy.pl is not compatible. You must migrate to Foswiki before converting the Store.

If you are using authentication, prepare a test plan to verify that your authentication mechanism is working correctly. Make sure you are able to test logins by a sufficient sample of users to cover all categories of users of your site. For example, users of various groups may need to be tested. In particular, ensure you test that non-admin users cannot access topics restricted to admins.

• Empty DENYTOPICxxxx rules are deprecated They are disabled by default. We recommend converting any existing rules into    * Set ALLOWTOPICxxxx = * wildcard allow rules. Use perl tools/convertTopicSettings.pl -help for further information on the conversion process.

Identify all essential Wiki topics and Wiki applications that must be fully functional upon completion of the upgrade. Prepare a test plan to verify their functionality. If you are using access controls, ensure that the test plan will adequately test all categories and groups of users of your site.

If your testing will require a test environment to be set up, ensure that it is ready, with any required support infrastructure (for example, testbed authentication servers). If you need to be able to login with different users in different categories and groups, ensure that you have the required login information ready, or you have testers from those groups available to perform the required test cases.

Installation

Follow the installation instructions in INSTALL.html, located in the root of the new installation, or online at Foswiki:System.InstallationGuide. Install the new release in a new directory. Do not install on top of the old release.

• For public or otherwise sensitive installations, ensure that your web server configuration is set to deny access to the new Foswiki installation for anyone except you.
• Configure Foswiki using the configure page.
  • (Not recommended!) If you are upgrading from an older Foswiki release, first copy your <oldwiki>/lib/LocalSite.cfg file to <newwiki>/lib/LocalSite.cfg in order to preserve your existing configuration settings (Not recommended). Alternatively, you can reconfigure the new installation from scratch (you can use your old LocalSite.cfg file as a reference).
  • Verify all of the configuration settings on the configure page, including any new settings added in the new version. Save the configuration after you have completed your changes.
  • To wipe out all your settings and start configuring from a fresh installation, just delete the <newwiki>/lib/LocalSite.cfg file and visit your default view URL. From there follow the link to configure.
• Additional resources
  • Foswiki:System.InstallationGuide
  • Foswiki:Support.InstallingOnSpecificPlatforms
  • Foswiki:Support.ApacheConfigGenerator
  • Foswiki:Support.SettingFileAccessRightsLinuxUnix
  • Foswiki:Support.UpgradingFromOlderTWikiReleases - upgrading TWiki from older TWiki releases

Test your newly-installed Foswiki site and ensure that its basic functionality works: viewing and editing topics (you can try creating and editing a topic in the Sandbox web).

Caution: If you intend to copy data from an older installation without using bulk_copy.pl to change Stores, you should select the RcsStoreContrib in the configuration. Once topic history has been created with the wrong store, it has to either be removed, or old data should be migrated with bulk_copy.pl. If Foswiki encounters mixed RCS and PlainFile topic history, it will "die" to prevent topic history corruption.

To make it easier to follow the subsequent steps, you can view this upgrade guide using your new Foswiki site by entering System.UpgradeGuide into the "Jump" text box on the top right of any topic. By doing this instead of using the UpgradeGuide.html file from the distribution, you will be able to use the embedded hyperlinks to jump directly to the referenced pages.

Install extensions

Install all of the extensions that were installed in your old site. In particular, start with any extensions required for the authentication and authorization methods you use (if any). You can use the Install, Update or Remove extensions tab in the Extensions section of the configure page to review installed extensions, search for extensions or all available extensions and configure extensions from the Foswiki:Extensions repository. You can also install extensions manually; see the instructions on the extension's web page from where you obtained the extension (for Foswiki extensions, on foswiki.org).

Check the plugin topics from your old Foswiki installation and transfer the plugin settings to the Main.SitePreferences topic in your new Foswiki site, prefixing each setting with the name of the plugin in uppercase followed by an underscore. For example, to copy over the DEFAULT_TYPE setting from the CommentPlugin topic in the old site to the new site, copy the value to a COMMENTPLUGIN_DEFAULT_TYPE setting in the Main.SitePreferences topic in the new site.

Commonly-customized plugin settings include the following:

• CommentPlugin - DEFAULT_TYPE
• EditTablePlugin Deprecated! Replaced with EditRowPlugin - CHANGEROWS, QUIETSAVE, EDITBUTTON
• InterwikiPlugin - RULESTOPIC
• InterWikis - If you added your own rules, make sure you copy over the rules to the new installation. Use of a local rules topic is the preferred way to customize the links.
• SlideShowPlugin - If you changed the embedded 'Default Slide Template', then copy your customized template to the topic in the new installation. You should prefer creating your own slide show template in a separate topic, so you will not have to take special steps over upgrades to preserve your modifications to the default slide template.
• SmiliesPlugin - If you added your own smileys, make sure you copy over your customizations to the topic in the new installatin.
• TablePlugin - TABLEATTRIBUTES

Activate, and if required, configure the installed extensions in configure.

Copy parts of the working directory

The working directory contains some critical information for some extensions, found below the foswiki/working/work_areas directory. Extensions use it to store persistent information critical to operation. For example, the MailerContrib directory contains the timestamps of the last notification email run per web. If not copied, the next mailnotify run will notify all recorded changes. This is the most common data that should be copied. Review other non-default extensions to determine if anything else should be copied.

Migrate .htpasswd file

If used, the .htpasswd file may require conversion. If users have registered using "high ascii" characters in their WikiNames (e.g. Ä, ä, Ë, ë, Ï, ï, Ö, ö, Ü, ü) then conversion will be required. Foswiki does not provide a tool to convert .htpasswd. Consider command line tools such as iconv or recode. See this StackOverflow article for more details.

If you are using an external authentication source, such as LDAP, this step does not apply.

Alternative 1: Convert the data using tools/bulk_copy.pl

This is the default way to migrate your system. Note cautions below about hidden files and performance. Assume that you have the following setup:

• Existing: /var/www/f119
• New: /var/www/f120

Use the bulk_copy.pl tool to migrate your data:

cd /var/www/f120/tools
perl bulk_copy.pl --xweb System --xweb _default --xweb _empty --latest '*.WebStatistics' /var/www/f119/bin /var/www/f120/bin

This will copy all webs, topics and attachments except for the contents of the System web. This is the recommended solution.

Note that bulk_copy.pl has limitations! It uses the Foswiki Store API to copy topic and attachments. If the store doesn't recognize a topic or attachment, it won't be copied.

• Files in pub that are "hidden" from the store are not copied.
  • Any filename beginning with an underscore. For ex _myfile. Frequently used by extensions as cache type files. Could also have been directly uploaded.
  • Any filename beginning with a dot. For ex. .htaccess files.
  • Any filename beginning with an asterisk. (This appears to be historical, with no common use).
• Files in pub that are not associated with a topic attachment META are not copied. (ex. files manually copied into the pub directories).
  • Auto attachments are not copied if the {RCS}{AutoAttachPubFiles} feature is not enabled.
  • Auto attached files in the RCS store will be converted to regular attachments in the PlainFile store. The PlainFile store does not support auto attachments..
• Files in subdirectories of topic attachments are not copied. (ex. image caches, javascript & css subdirectories, etc.) This is common in the System web.
• All topic revisions must have the same encoding. If {Site}{CharSet} was changed after history exists, the history will not convert correctly and the copy may fail.
• bulk_copy.pl can be very slow converting topics with extensive histories. Each revision of a topic is separately converted. Any errors in the history can cause the conversion to fail.

bulk_copy.pl executes the code and APIs of the older Foswiki release. Older versions of Foswiki may fail on the latest versions of Perl due to language changes. If you encounter this issue, you could try an upgrade of the 1.1.x system to Foswiki 1.1.10, or use the CharsetConverterContrib to complete the migration.

bulk_copy.pl is not compatible with Foswiki 1.0.x versions. If you are copying data from Foswiki 1.0.x, there are two options:

1. Use CharsetConverterContrib to convert the data "in-place" on Foswiki 2.x -or-
2. Convert to Foswiki 1.1.10 first, and then convert to Foswiki 2.x.

The first option is faster and less complex.

Before proceeding with conversion, verify that the above limitations are not applicable to your installation. If they are, you will currently need to:

• Stay on the RCS style store
• Use CharsetConverterContrib to change encoding if desired.

Alternative 2: Convert data using CharsetConverterContrib

These steps can be used to manually migrate data when not changing the Store type. This is currently the only way to deal with hidden files.

Main changes for older Foswiki installations

For upgrades from an older Foswiki installation:

• Copy any topics shipped in the default Main web into your new Main, and check for any local customization of these default topics.
• Verify that the following default users are present in the Main.WikiUsers topic:
  • ProjectContributor - the Foswiki documentation is attributed to this user
  • RegistrationAgent - special user used during the new user registration process
  • UnknownUser - used where the author of a previously stored piece of data can't be determined
  • WikiGuest - guest user; used as a fallback if the user can't be identified
• If any of the default users are missing, then add them in manually to Main.WikiUsers, using the corresponding entries in Foswiki:System.UsersTemplate as an example.
  • Be sure to preserve the alphabetical order of the WikiUsers topic, and do not insert any blank lines.
• If you have customized <old_system_web>.UserRegistration, then either copy over <oldwiki>/data/<old_system_web>/UserRegistration.txt and <oldwiki>/data/<old_system_web>/UserRegistration.txt,v to the <newwiki>/data/System/ directory, or modify System.UserRegistration in the new installation to contain your customizations.

Convert empty DENY ACLs to ALLOW * wildcards

By default, empty DENYTOPIC rules will be ignored by Foswiki 2.0. You must change them to the equivalent ALLOWTOPIC * rules. The tools/convertTopicSettings.pl utility will scan the Webs & Topics, and will perform several optional conversions on the topics.

Get help text for convertTopicSettings

perl tools/convertTopicSettings.pl -help

Scan all webs / topics, report any topics with empty DENY rules

perl tools/convertTopicSettings.pl

Replace all empty DENY rules with ALLOW * wildcards

perl tools/convertTopicSettings.pl -fixdeny -update

The following two options are also available, but are not yet recommended.

Same, but convert all ACLs into META settings from inline topic settings, for just the Sandbox web

perl tools/convertTopicSettings.pl -fixdeny -convert -update Sandbox

Convert ALL settings into META settings, not just ACLs, for the Sandbox and Customer webs

perl tools/convertTopicSettings.pl -fixdeny -convert -all -update Sandbox Customer

When convertTopicSettings saves the modified topics, they will be saved by user UnknownUser.

DataForms Applications

If your site uses DataForms that used non-Ascii field names, the form data will require manual migration, or you must enable {LegacyFormfieldNames} in the configuration.

• Releases prior to Foswiki 2.0 stripped characters other than A-Z, a-z, 0-9 and _. So a field named Fühler would be stored as Fhler.
• The same DataForms definition on Foswiki 2.0 would be stored as Fühler.

If you do not enable {LegacyFormfieldNames}, then you will need to find and update the META:FIELD definitions in the topics. This would need to be done external to Foswiki.

  %META:FIELD{name="Fhler" title="Fühler" value="123"}%

would need to be changed to

  %META:FIELD{name="Fühler" title="Fühler" value="123"}%

Apply preferences from old installation

If you have not already set your desired site-wide preferences, as described in the section " Set Foswiki Preferences" in the InstallationGuide, then set your preferences. The location of your site preferences is specified in the Miscellaneous settings pane of the configure page, in the {LocalSitePreferences} setting (visible when Expert mode is enabled) — the default location is Main.SitePreferences. Copy any customized preferences from the site preferences topic in your old installation to the site preferences topic in the new installation. (Note you may have already copied over your customized preferences when you transfered the contents of the <old_users_web> web.) (These should have been copied when your Main was migrated.)

If, in your old installation, you customized the default preferences in <old_system_web>.DefaultPreferences, then transfer your customizations from this topic to the SitePreferences topic instead (i.e. the topic specified in your {LocalSitePreferences} setting), so that your customizations will not get overwritten on the next upgrade.

If you have any old extensions that use settings from the "System.<Extension>" topic, these settings should also be copied to the SitePreferences. Topics in the System should never be edited!

Apply additional site customizations

Modify skin with customizations for your site

If you did not already customize the appearance of your new installation, as described in the section " Customize the appearance of your Foswiki site" in the InstallationGuide, then reapply the customizations from your old installation to the new one. Ensure you transfer over any skin templates — .tmpl files, or topics referred to using VIEW_TEMPLATE or EDIT_TEMPLATE preferences — you need. Also ensure you transfer any style sheets or Javascript files required.

Customize pages for managing personal information

In your new installation, default copies of the following topics were installed:

• System.ChangePassword
• System.ResetPassword
• System.ChangeEmailAddress

If you customized these topics in your old installation, transfer the changes to these topics in the new installation. Use the corresponding files in the <oldwiki>/<old_system_web>/ directory as a reference.

Check your configuration and installation

Configure provides some tools to validate your installation. They should be run as the web server userid:

cd <path-to-foswiki-installation>
sudo -u www-data tools/configure -check
sudo -u www-data perl tools/configure -check {DataDir} -method validate_permissions
sudo -u www-data perl tools/configure -check {PubDir} -method validate_permissions
 ...  also can be run on: 
    {LocalesDir} {ScriptDir}  {TemplateDir} {ToolsDir} {WorkingDir}

Validate your Wiki applications and other key functionality

Execute your test plan to validate the Wiki applications and other key functionality that need to be up and running after the upgrade.

Execute your test plans for authentication and authorization. Test that users that you have transferred from the old installation can login with any problems, and that access controls work appropriately: check that users are able to view and edit pages for which they have access, and are denied permission to view or edit pages for which they do not have access. Also check that pages restricted to the admin group are not accessible by non-admin users, and that administrators continue to have access.

Switch your production site from the old installation to the new installation

If you are converting from RCS to PlainFile store, you must not repeat any copy step from the old to the new version once you've run the conversion.

If you had been running your old installation in parallel with the new one during a test phase, then disable your old installation, and repeat the steps:

• Copy content from non-default webs in old installation to the new installation".
• Copy extension operational information from working, to the new installation

Change your web server configuration so that the new installation is accessible to all of your users, and so the old installation is no longer accessible.

Change your web server configuration so that the new installation is accessible using the same URL prefix as your old installation. For purposes of discussion, assume that your old installation is accessible from http://yourdomain/wiki/. You can use one of the following approaches to make the new installation accessible using the same URL prefix:

• You can rename your <newwiki>/ directory to wiki/ (renaming the directory of your old installation if necessary).
• If your operating system supports links to other directories and your web server is configured to follow links, then you can create a link called wiki/ that points to <newwiki>/ (renaming the directory of your old installation if necessary).
• You can configure your web server so that requests to /wiki/ are served from your <newwiki>/ directory.

Re-execute your test plan to verify that your newly-upgraded site is accessible to your users, and that all authentication and authorization mechanisms work as expected (including denying access to those who are not authorized).

Re-execute your test plan to verify that your Wiki applications and other key functionality work as intended.

Related Topics: AdminDocumentationCategory, Foswiki:Support.InstallationGuide, Foswiki:Support.InstallingOnSpecificPlatforms, Foswiki:Support.ApacheConfigGenerator, Foswiki:Support.SettingFileAccessRightsLinuxUnix, ReleaseNotes01x00, ReleaseNotes01x01