TechDocs/Mainpage/build_process

Please note that this document is out of date and was moved here to be updated!

This document contains a description of the FSFE web pages, both the practice of maintaining them, and that of generating them. It is hoped that this document will be continuously updated by the webmasters of FSF Europe as the web pages evolve. It was last updated $Id: README.texi,v 1.3 2006-05-08 14:31:35 reinhard Exp $.

If you find bugs in this file, or have other ideas on how to improve it; please commit your ideas to paper in the README.texi file in CVS. All other versions are automatically generated from the TexInfo source.

Webmastering

This chapter will describe how to be a webmaster of FSFE. It will describe how the pages are envisioned, what files can be edited, how they should be edited and various other bits that are of particular interest to a webmaster.

Focuses

The FSFE web pages are divided into focuses. Each focus represents a view of the web tree that contains all the information of global importance, aswell as the information relevant to the focus. A focus is usually a geographic area, such as Germany or France. A visitor can choose which focus to browse the pages with and will see slightly different things depending upon their choices.

The pages themselves always contain the same information, no matter what focus is selected. The only exception to this is pages which are automatically generated. The notable things that change depending upon focus is the menu, and which projects are displayed on the project pages. This makes it easy for people to find information relevant for their interest.

Source files

As a webmaster, about the only information one is usually interested in is the content of the pages. This content is in the files named <tt>*.xhtml</tt>. Each file contains an XML document, conforming to XHTML 1.0. These files are particularly easy to update, since they look very much like every other HTML file.

The information in these files will be transformed by XSL to produce the finished web page. Be sure to maintain a correct structure in these files. It should be similar to this:

<source lang="xml"> <?xml version="1.0" encoding="UTF-8" ?>

</source>

Editing

Once you’ve gotten hold of the correct source file to edit, things should be smooth sailing. Just follow standard XHTML 1.0, and things will render well in most browsers. Also, do make sure that the file is able to be validated as an XHTML document. You can use the validator at [http://validator.w3.org/ http://validator.w3.org/] if you’re not sure. A source file can simply be uploaded to the validator and it should validate as XHTML 1.0 Transitional.

There is a program in the CVS tree called ‘<tt>tools/validate.pl</tt>’. If you install the XML::LibXML Perl module, you can use that program to make sure that your pages at least parse as valid XML before you commit them to CVS.

The XML::LibXML module can be found in the Debian package libxml-libxml-perl.

You use the program like this (after having made a change to index.en.xhtml and index.fr.xhtml):

<source lang="bash"> $ tools/validate.pl index.en.xhtml index.fr.xhtml</source> If the program croaks, you need to fix the file before committing it to CVS. If it does not, chances are that the page will render nicely. It’s still not guaranteed though, but it’s a good enough guess.

Automatic updates

It’s often the case that one wants to have certain pages updated automatically, for example to allow the front page list the latest five newsitems, or to have a list of projects automatically generated. There is a support system in the build process that will do these things in a general enough way to be useful for most tasks.

This will be explained in more detailed in the section about building the web pages, but sufficiently to say, the process involves parsing XML data in an intermediate step, producing XHTML information, which is then fed to the main XSL transformation.

For every page that is automatically updated, there exists two files; page.sources and page.xsl. The first file gives a list of all the XML source files that should be used as input. The format of this file is as follows:

<pre class="verbatim"> directory/*/filename:focus</pre> Several lines of this type can exist in each sources file. The first part of the line is a filename glob. The above example would match the source files <tt>directory/2001/filename.en.xml</tt>, <tt>directory/duck/filename.fr.xml</tt>, and so on, depending upon the language.

The focus specifies for what focuses the source files should be included. This can be either <tt>global</tt>, which would cause the files to be included for all focuses, or a country specific two-letter tag to include it only for that specific focus.

The XSL file takes the sum of all source files and should produce a valid XHTML document. Please see index.xsl, news/news.xsl and other files in CVS for an example of how this is done.

For the webmaster, it’s usually enough to note that the XHTML files should be updated as usual, and that the XML files contain the input to the automatic build process.

Special files

There are several special files in CVS that a webmaster should care particularly for. Besides the file mentioned in the previous section about automatic updates, there exists a directory called tools/. This directory holds all kinds of tools used by the build process. It also contain information that relates to the menus and the static texts on the pages.

Translating

As a translator, your job is to make sure that as many pages as possible have translations into your local language. The process of translating is simple; usually, you take the original file, translate the contents, and then save it under another filename, indicating the language you’re translating into. For example, ‘<tt>eucd.en.xhtml</tt>’ becomes ‘<tt>eucd.de.xhtml</tt>’ for the German translation, and so on.

The question that most often arises is; what should be translated? This is not a simple question to answer, and it’s up to each translator to do the best possible job at deciding this. Here are some guidelines though:

# Always make sure that the ‘<tt>texts-en.xml</tt>’ file in tools/ are translated into your language. If you’re making a German translation, you should name the resulting file ‘<tt>texts-de.xml</tt>’, and similarly for other languages. If this file is out of date, or not translated, some texts might appear in English on your pages, despite other translations. # Keep up to date with the news items in ‘<tt>news/YEAR/</tt>’ and ‘<tt>COUNTRY/news/YEAR/</tt>’ (where YEAR is something like 2002 or 2003, and COUNTRY is a countrycode such as de, or fr). Make sure that every item is translated and put into the translated file. # All projects have a file called ‘<tt>project.en.xml</tt>’ in their root directory. Take care to keep these translated and updated. # Aside from the above files, you should work on translating the most important pages first. What pages constitutes as important, apart from these guidelines, are left open. Please take care to keep all pages you translate updated though. If you have very little time to check for updates, it might make sense to translate pages that are not updated very frequently instead of those that is.

Building

This chapter of the FSFE webmastering manual will describe the steps involved in building the web pages from scratch. Normally, this is not something that anyone should have to bother about. As a webmaster and translator, it’s sufficient to know how it works so that one can fix problems with it as they arise.

Building a complete set of pages can take up to 15-20 minutes, so it should not be done frequently.

There are several sections to this problem. The first one describes the software requirements that are needed to build the pages. The second describes the actual process of building pages, and the third describe the ‘<tt>build.pl</tt>’ script that does the actual building.

<table> <tbody> <tr class="odd"> <td align="left">3.1 Requirements</td> <td align="left"></td> <td align="left">What you must have to build the pages</td> </tr> <tr class="even"> <td align="left">3.2 Process</td> <td align="left"></td> <td align="left">How the build process works</td> </tr> <tr class="odd"> <td align="left">3.3 build.pl</td> <td align="left"></td> <td align="left">How the build script works</td> </tr> </tbody> </table>

Requirements

The build script is created using Perl and requires several Perl modules to function properly. Most of them are included in the standard Perl distribution, but a few are not.

The non-standard modules that needs to be installed are:

* ‘<tt>File::Find::Rule</tt>’. This module is a simple interface to make recursive searches for files. It’s used extensively by the build script. It does not exist in any known packages, so it needs to be installed from CPAN with the following command:

<source lang="bash"> # perl -MCPAN -e 'install File::Find::Rule;'</source> <ul> <li><p>‘<tt>XML::LibXSLT</tt>’ and ‘<tt>XML::LibXML</tt>’. These modules are both wrappers to the GNOME projects libxml and libxslt libraries. They exists in the Debian packages libxml-libxml-perl and libxml-libxslt-perl.</p> <p>Version 1.50 of XML::LibXST and version 1.52 of XML::LibXML is known to work. Later versions should work too. Those versions are, as of this writing, found in the Debian testing distribution. The versions in Debian stable are slightly too old and the build script would have to be modified somewhat to work with those.</p></li></ul>

Process

The first point in understanding how the web pages work on a technical level is to understand how the build process works. There are two items that require additional explanations; translations and focuses.

<ul> <li><p>When we create the pages from their respective sources, we make sure that we produce files for all languages that we have translations into. If the source file does not exist in a particular language, we use the language of the original source file instead, and put a special marker on the page saying that a translation to the selected language could not be found.</p> <p>This means that links can point to, for example, /help/help.de.html, even if the help page does not have a German translation. This is mostly useful because it means that if all the links in help.de.html (despite that the content of the page might be in English), point to German pages, the user will retain his or her choice of language thruought the web site.</p> <p>The build script has a post-processing instruction that makes sure that links are changed to reflect this. This does mean that language negotiation does not work, as such. Language negotiation only works when someone visits the first page. Thereafter, the user will retain the selected language even if the preference in the browser is changed.</p> <p>This also means that if a user selects another language on our web pages, that language will be retained thruought the site, despite the preference of the users browser.</p></li> <li><p>Focuses are ment to aid visitors in finding the right information for their interest. At the moment, we have three focuses; Global, French, and German. More will be added as time permits.</p> <p>Every focus will always contain the information that is of global interest, but that information might be supplemented by localised information.</p> <p>The things that are “focused” today is news items on the front page, and the projects that are shown to a visitor.</p></li></ul>

Having noted that, we will turn our attention to the technical implementation of these things. The first thing we have in our hands is a source file, such as ‘<tt>help.en.xhtml</tt>’. The format of this file should be as follows:

<source lang="xml"><?xml version="1.0" encoding="UTF-8" ?>

<html>

</html></source> It’s important to note that not all pages follow this convention. It’s a nice gesture to make sure that they do, but the build script contains some magic to be able to parse and handle files in other (older) formats aswell.

This source file needs additional information before it can be sent to the final XSL transformation. In particular, we need to know what translations exists for it, what the menu should look like for this page and some special static text strings for the page.

The build process should find all this information and create a new document from the above, merged with the additional information. When finished, the result should look like this:

<source lang="xml"><?xml version="1.0" encoding="UTF-8" ?>

<buildinfo>

</buildinfo></source> The translations are looked up in the filesystem. The menu is taken from file menu files in the ‘<tt>tools</tt>’ directory. The texts are also taken from the text files in the ‘<tt>tools</tt>’ directory. If possible, the build process must try to pick the text files for the language that it is currently building for. If they don’t exist, it should fall back on the English texts.

This is, unfortunately, not all information that the XSL transformation needs. It needs additional information about filenames and other things. We must add these attributes:

* buildinfo/@original. This attribute gives the language code of the original document (en for most pages). * buildinfo/@filename. The XSL process is made self-aware with this tag. It includes the filename that we’re currently building, but without language tag or the trailing ‘<tt>.html</tt>’. * buildinfo/@language. The language that we’re currently building the page for. * buildinfo/@outdated. This attribute is set to “yes” if the original file is newer than this translation. * document/@language. Set to the language of the document. This might be different than the language of the page, if we could not find a proper translation.

With these addition, the result could look like this:

<source lang="xml"><?xml version="1.0" encoding="UTF-8" ?>

<buildinfo original="en" filename="about/background"

</buildinfo></source> With all this information, the XSL transformation is ready to begin. The resulting buildinfo node is passed to the ‘<tt>fsfe-new.xsl</tt>’ XSL file, which transforms the document into the final XHTML file that is put into the web tree. Observe that this is done once for every focus.

And if you thought that that was it, you’re in for a surprise now! We havn’t even begun to mention automatically updated pages. Luckily, they are not very complex.

If there exists a file, ‘<tt>name.xsl</tt>’ in addition to the normal source files, ‘<tt>name.lang.xhtml</tt>’, this means that the page is updated automatically from the source files mentioned in ‘<tt>name.sources</tt>’.

To unravel this mystery, we will first look at the sources file, which can take the following form:

<pre class="verbatim">projects/*/project:global de/projects/*/project:de</pre> The sources file contains a list of glob patterns and their respective focus. The special focus, “global”, means that the source files matching that glob pattern, will be included for all focuses.

When the build process creates an automatically updated page, it begins by picking out the glob patterns for the current focus and adding “.lang.xml” to each pattern, once for each language. All files that match that glob pattern are then assembled into one set. Note that if we’re generating a page in German, and there does not exist a file, ‘<tt>projects/a/project.de.xml</tt>’, but there does exist a file ‘<tt>projects/a/project.en.xml</tt>’, the English version of the file will be included for completeness.

Suppose we have two files with the following content:

<source lang="xml"><?xml version="1.0" encoding="UTF-8" ?>

<newsset>

</newsset></source> and

<source lang="xml"><?xml version="1.0" encoding="UTF-8" ?>

<newsset>

</newsset></source> The assembled file will look like this:

<source lang="xml"><?xml version="1.0" encoding="UTF-8" ?>

<set>

</set></source> Note that we loose the name of the top node. This is irrelevant for the transformation and having a common name means that we can, theoretically, include sets of different names into the resulting output.

This resulting file is then merged with the ‘<tt>file.en.xhtml</tt>’ file before it is passed to ‘<tt>file.xsl</tt>’ for transformation. The way it does this is to simply insert the <set\> node as a child of the <html\> node.

This transformation MUST result in an XML file that can then be used as input to ‘<tt>fsfe-new.xsl</tt>’, which in turn produces the finished page.

build.pl

The ‘<tt>build.pl</tt>’ script implements the build process described in the previous section. It has not been extensively tested and has never been optimized for speed.

The script reads files from the current directory and produces the resulting web tree in the directory specified. It currently accepts the following options:

* -q. Quiet more. This will cause the script to be quiet about everything except plain errors. * -u. Update only. This can be used to speed up the processing a bit. Normally, the script will remove everything from the destination directory before creating new pages there. This will leave all files in their former destinations and only build them if they have been changed. * -d. Print some debug information. * -n. Dry-run only. Using this switch prevents the script from writing any information to disk. All pages are generated, but the results are discarded. Can be used to verify that a tree will build correctly. * -o <directory\>. This switch must be specified. It lets the script know to which directory it should write the finished pages. This top level directory will contain a secondary directory for each focus.

Administrating

Apache

Perl

People

Webmasters

Translators

Guides

This chapter contains simple step-by-step guides to performing certain functions on the FSFE web site, such as posting news, adding new projects and so on.

Posting news

When you’re posting a news item, you have a choice to make; should it be a global item, or a localised item? The decision you make will influence which directory to put the information in. For a global news item, you should place it in ‘<tt>news/</tt>’ and ‘<tt>country/news/</tt>’ for localised news.

Each directory mentioned should have one subdirectory for each year. Those directories should contain files of the form ‘<tt>news.en.xml</tt>’ which can contain any number of news items. An English version of the news files are mandatory.

Global news will be included for every focus. Localised news only for each respective focus. A news item in ‘<tt>news.en.xml</tt>’ can look like this:

<source lang="xml"> <news date="2002-02-01">

Once this has been commited to CVS, the item will be included in the news listings once the next update has been run.

Adding a project

As for news items, the location of a project depends on whether it is of global interest or a local project. Place a directory with all project information in the ‘<tt>projects/</tt>’ or ‘<tt>country/projects/</tt>’ hierarchy.

In addition to a directory, decide on a classification for the project (one of community, legal, other and technical). Put that information in a ‘<tt>project.en.xml</tt>’ file in the root directory of the project.

<source lang="xml"><?xml version="1.0" encoding="UTF-8" ?>

<projectset>

</projectset></source>

TechDocs/Mainpage/build_process (last edited 2016-12-04 20:17:44 by max.mehl)