• Events

• LocalGroups

• Activities

• Teams

• KnowHow

• RecentChanges

Build FSFE websites locally

Those who create, edit, and translate FSFE websites already know that the source files are XHTML files which are build with a XSLT processor, including a lot of custom stuff. One of the huge advantages from that is that we don't have to rely on dynamic website processors and databases, on the other hand there are a few drawbacks as well: websites need a few minutes to be generated by the central build system, and it's quite easy to mess up with the XML syntax. Now if an editor wants to create or edit a page, she needs to wait a few minutes until the build system has finished everytime she wants to test how the website looks like. So in this guide we will show how to build single websites on your own computer in a fraction of the FSFE's system build time, so you'll only need to commit your changes as soon as the file looks as you want it. All you need is a bit hard disk space and around one hour of time to set up everything.

1. Set up necessary packages

If you have not used Git before, please install it now via the package manager of your distribution (such as apt).

For the build process itself, you also need a few additional packages. (Warning: it's possible your system lacks some other required applications which were already installed on mine. If you encounter any command not found errors, please report them in the comments or by mail). Here is a list that you can install via apt on Debian and its derivates; for other distributions, please install the equivalent packages viat the package manager:

make xsltproc libxslt1.1 realpath libxml2-utils rsync python3 python3-bs4

Note: libxslt1.1 may have a different name in your distribution, e.g. libxslt or libxslt2.

If you see errors, while trying to run a command later, you might have to first install realpath which is included in the package coreutils in Debian and its derivates. So if you encounter errors, give that package a try.

2. Clone website repository

If you have write access to the main FSFE website repository, you can clone that. If not, please clone your personal fork of the main repo. If you have made changes before, chances are you already have a copy of this repo and you don't need to clone it again.

git clone https://git.fsfe.org/FSFE/fsfe-website

3. Clone helper repository

Now, clone a git repository which will give you most needed files and directories for the further operations. It has been created by Max Mehl and contains configuration files and the script that will make building of single files easier.

git clone https://git.fsfe.org/FSFE/fsfe-local-build

Now, your directory should look like this (if you cloned both repositories to ~/Web/FSFE/:

~/Web/FSFE/fsfe-local-build/
~/Web/FSFE/fsfe-website/

(For those not so familiar with the GNU/Linux terminal: ~ is the short version of your home directory, so for example /home/user. ~/Web is the same as /home/USER/Web, given that your username is USER)

Now let's make the helper scripts easy to access via the terminal for future usage. For this, we'll create a short link to the script from one of the binary path directories. Type in the terminal:

sudo ln -s ~/Web/FSFE/fsfe-local-build/fsfe-build.sh /usr/local/bin/fsfe-build
sudo ln -s ~/Web/FSFE/fsfe-local-build/fsfe-preview.sh /usr/local/bin/fsfe-preview

4. Build full website

Now we have to build the whole FSFE website locally. This will take a longer time but we'll only have to do it once. Later, you'll just build single files and not >14000 as we do now.

The full website build can be started with

cd ~/Web/FSFE/fsfe-website
fsfe-build

The final website will reside in ~/Web/FSFE/fsfe-local-build/fsfe.org. Mind differing directory names if you have another structure than in this example. This process will take a long time, depending on your CPU power. Also, it sometimes may look as if the process is stuck, but please be patient. Most of the time, however, you should see some output informing you of the progress.

Afterwards, the output of git status should be empty when you are in the directory fsfe-website. It is? Fine, let's go on! If not, please also remove those files (and tell us which files have been generated by the build script and should be ignored, or directly edit it in this wiki).

After the first full build, the fsfe-build command will automatically detect changes and only rebuild the files that need to be rebuilt. If you want to do a full build again, use fsfe-build --full. Note that for both commands, the source directory (~/Web/FSFE/fsfe-website in our example) has to be the current directory.

5. Use local webserver to display the build

After the full build is completed, you can setup a local webserver. This is necessary to actually display the locally built files in your browser. With the solution described here, you'll have nothing to configure on most systems since python3 and its modules http.server and socketserver are installed on most GNU/Linux distributions by default. With downloading the helper repository all necessary files and configuration values should be shipped in order to make it compatible with the build script presented later on.

cd ~/Web/FSFE/fsfe-local-build/fsfe.org
python3 -m http.server -b 127.0.0.1

If you only have installed Python 2 (i.e. the command python3 --version throws an error), you can use a slightly different syntax for running the http server.

python -m SimpleHTTPServer

In both cases the website should be visible on your browser using the URL http://localhost:8000/index.en.html (we don't create the index.html file because full-fledged webservers will automatically map index.en.html to index.html).

6. Advanced webserver setup (optional)

If you set up the minimal Python webserver above and are happy, please skip this step; you do not need it. In case you have more complex needs (such as running CGI scripts), but don't yet have a webserver set up, you can use lighttpd.

In order to set up lighttpd, please install the following packages (these are names for Debian again, so the package names may be slightly different on your distribution).

lighttpd gamin

To make Lighttpd running properly we need a configuration file. This has to point the webserver to show files in the fsfe.org directory. You already downloaded my recommended config file (lighttpd-fsfe.conf.sample) by cloning the git repository. But you'll have to modify the path accordingly and rename it. So rename the file to lighttpd-fsfe.conf, open it and change following line to match the actual and absolute path of the fsfe.org directory (~ does not work here):

server.document-root = "/home/USER/Web/FSFE/fsfe-local-build/fsfe.org"

Now you can test whether the webserver is configured correctly. To start a temporary webserver process, execute the next command in the terminal:

lighttpd -Df ~/Web/FSFE/fsfe-local-build/lighttpd-fsfe.conf

Until you press Ctrl+C, you should be able to open your local FSFE website in any browser on the same computer using the URL http://localhost:5080. For example, open the URL http://localhost:5080/contribute/contribute.en.html in your browser. You should see basically the same website as the original fsfe.org website. If not, double-check the paths, if the lighttpd process is still running, or if the full website build is already finished.

7. Single page build script

In this step, we'll configure and start using a Bash script (fsfe-preview.sh) Max has written to make a preview of a locally edited XHTML file as comfortable as possible. You already downloaded it by cloning the repository.

First, rename and edit the script's configuration file config.cfg.sample. Rename it to config.cfg and open it. The file contains all paths we already used here, so please adapt them to your structure if necessary. Normally, it should be sufficient to modify the values for LOC_trunk (trunk directory) and LOC_out (fsfe.org directory), the rest can be left with the default values.

If you decided to use lighttpd as your webserver, please change the value of HTTPD to lighttpd. HTTPD_conf can be left with the default value if you didn't rename something in the helper repository. HTTPD_port is only necessary with using SimpleHTTP and only has to be changed if you are not able to use port 5080.

Another feature of the fsfe-preview is to automatically check the XML syntax of the files. We installed libxml2-utils in step 1 in order to make this function working.

From this moment on, you should be able to call fsfe-preview from anywhere in your terminal. Let's make a test run. Modify the XHTML source file contribute/contribute.en.xhtml and edit some obvious text or alter the title. Now do:

fsfe-preview ~/Web/FSFE/fsfe-website/contribute/contribute.en.xhtml

As output, you should see something like:

[INFO] Using file /home/max/Web/FSFE/fsfe-website/contribute/contribute.en.xhtml as source...
[INFO] XHTML file detected. Going to build into /home/max/Web/FSFE/fsfe-local-build/fsfe.org/contribute/contribute.en.html ...
[INFO] Starting python webserver

[SUCCESS] Finished. File can be viewed at http://localhost:5080/contribute/contribute.en.html

Now open the mentioned URL http://localhost:5080/contribute/contribute.en.html and take a look whether your changes had an effect.

8. Recommended workflows

In this section we'll present a few of the cases you might face and how to solve them with the script. We presume you have your terminal opened in the trunk directory.

8.1. Preview a single file

To preview a single file before uploading it, just edit it locally. The file has to be located in the fsfe-website directory, so we suggest to only use one copy of the repo on your computer. It makes almost no sense to store your edited files in different folders. To preview it, just give the path to the edited file as argument for fsfe-preview, just as we did in the preceding step:

fsfe-preview activities/radiodirective/statement.en.xhtml

The script detects whether the file has to be built with the XSLT processor (.xhtml files), or if it just can be copied to the website without any modification (e.g. images).

To rebuild the file automatically when changed, run the following command:

echo activities/radiodirective/statement.en.xhtml | entr fsfe-preview  /_

This requires the entr package.

8.2. Copy many files at once

Beware that all files you added in your session have to be processed with the script. For example, if you create a report with many images included and want to preview it, you will have to copy all these images to the output directory as well, and not only the XHTML file. For this, there is the --copy argument. This circumvents the whole XSLT build process and just plainly copies the given files (or folders). In this example, the workflow could look like the following: The first line copies some images, the second builds the corresponding XHTML file which makes use of these images:

fsfe-preview --copy news/2016/graphics/report1.png news/2016/graphics/report2.jpg
fsfe-preview news/2016/news-20161231-01.en.xhtml

8.3. Syntax check

In general, it's good to check the XHTML syntax before editing and commiting files to the SVN. The script fsfe-preview already contains these checks but it's good to be able to use it anyway. If you didn't already do it before, install libxml2-utils on your computer. It contains xmllint, a syntax checker for XML files. You can use it like this:

xmllint --noout work.en.xhtml

If there's no output (--noout), the file has a correct syntax and you're ready to continue. But you may also see something like

work.en.xhtml:55: parser error : Opening and ending tag mismatch: p line 41 and li
      </li>
           ^

In this case, this means that the <p> tag starting in line 41 isn't closed properly.

9. Drawbacks

The presented process and script has a few drawbacks. For example you aren't able to preview certain very dynamic pages or parts of pages, or those depending on CGI scripts. In most cases you'll never encounter these, but if you're getting active with the FSFE's webmaster team it may happen that you'll have to fall back on the standard central build system.

Any other issues? Feel free to report them as they will help to improve FSFE's editors to work more efficiently

10. Changelog

Major changes to this guide

• 29 November 2016: Jonas has pointed out a few bugs and issues with a different GNU/Linux distribution. Should be resolved.
• 15 December 2016: Introducing simpler local webserver which doesn't need any configuration