Category:MediaWiki extension Mediawiki collection extension installation | |
---|---|
Extension name | Collection |
About this article / disclaimer | |
Logo | |
Screenshot | |
Location of the main author | Germany |
Coordinates of the main authors | 51.163818, 10.447831 |
Developers | PediaPress.com, Siebrand Mazeland, Marcin Cieślak |
Licences | GPL |
Description | This extension makes it possible to collect a number of pages. Collections can be edited, persisted and optionally retrieved as PDF, ODF or DocBook (XML) |
Mediawiki requirements | Version 1.17 and later |
Dependencies | |
Related extensions (documented here) | |
Related extensions | [[Is related to information::PDF Writer, Open Document Export]] |
Discussion | There are two ways of using this extension.
|
Language support | Many |
Status | stable |
First release date | |
Last release date (as of !) | |
Last version number | |
Programming language | PHP, Python |
Alternatives | |
Website | home page |
Publications | |
Support websites | web site, web site |
Example websites | |
Last edited |
[UPDATE: Please also read the discussion page ! Both Debian/Ubuntu upgrades and MW 1.27 upgrades do require changes that are not documented here] - Daniel K. Schneider (talk) 21:54, 13 September 2016 (CEST)
The Mediawiki collection extension allows a user to organize personal selections of pages in a collection. For an educational or academic wiki, this is a must-have extension. E.g. you could use to create print manuals and textbooks or also to valorize student productions at the end of year (hey "daddy" we made a book...)
Collections can be:
There are two major levels of installation/configuration
This page includes some centralized help links and installation tips made for our own use in May 2009, updated several times since then. Latest attempt was to get this working again for MW 1.31 (sept. 2018). After upgrading the service stopped working, but the issue is probably related to messy installation of python libraries. We sort of fixed this.
See also:
Our current PDF creator installation (feb 2019) has trouble rendering images that sit in tables and galleries and that are part of books. Curiously, generating individual files does seem to work like before.
As of Feb 2019, there seems to be some progress with respect to creating a new architecture.
There is an alternative, which as of Feb 7 does not work with wiki books in our wiki since we use different than wikipedia URL's
There are several interesting features: It can create epub and ODT, there is a way to handle templates (not tested), Latex equations do work, not surprising :). On could export the Latex and then hand tune.
Documentation for the collection extension and the servers doing the work in the background is very chaotic and depending on the solution you will adopt...
As far as Wikipedia is concerned, there are some news (sept. 2018). In the past few years there have been attempts to replace the old (and reliable) solution with Offline content generator (also here), but that project seems to have become a failure.
Documentation for mw-lib (the old but still working solution)
Documentation entry page at PediaPress:
Help (can also be used for informal bug reports)
Information about the collection extension and related server-side software
Installing the whole suite requires some installation skills, but should go fairly smoothly on any Unix system and should be easy on a Debian-based Linux.
If you own a small MediaWiki and do not plan to customize PDF rendering, all you need is this extension. If you use this service often and/or you plan to tweak, then you also have to install the rendering servers. See the sections below.
The collection extension installs like any other Mediawiki extensions. Really easy with Mediawiki=> 1.14 (Spring 2009).
Get it from Extension:Collection on Mediawiki.org.
You may try the latest version, however sometimes it doesn't work with your MW installation. E.g. it breaks for MW 1.16.4 on April 20 2011. Get it from GIT
cd extensions git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Collection.git git branch -r git checkout REL1_25
Read:
README.txt
You then can just leave all the defaults and the PDF will be generated by PediaPress. However, if you have a slow server or a high traffic server, you also should install a local render server (read the whole rest of this page). When we first installed this extensions in 2008, the Pediapress server did loose pages due to server overload. By now, this problem should be fixed (not tested ....)
If you have and old and slow server, I suggest changing the file Collection.i18n.php and change the string (adjust to the power of your server)
'coll-rendering_text' => "<p><strong>Please wait while the document is being generated.
Depending on the size of book you may have to wait 5, 10, 15 minutes or longer.
</strong></p>
......"
'coll-save_collection_text' => 'Choose a storage location for your book and enter a name:',
If you want users to be able to save and to share collections, add these permission to file Localsettings.php:
$wgGroupPermissions['user']['collectionsaveasuserpage'] = true; $wgGroupPermissions['user']['collectionsaveascommunitypage'] = true;
or maybe a more restrictive alternative:
$wgGroupPermissions['user']['collectionsaveasuserpage'] = true; $wgGroupPermissions['sysop']['collectionsaveascommunitypage'] = true;
You should define the following (language-dependant) templates and categories:
For content to be excluded from the PDF, use the CSS class "noprint":
<div class="noprint">blub noprint</div>
For content to be included in PDFs and excluded in the wiki page, use the following CSS
<div class="onlyinprint">blob only in print</div>
This css class is ignored by the PDF renderer and thus displayed normally in the PDF. Modify your MediaWiki:Common.css file to hide the content in the wiki page.
.onlyinprint {display: none}
Of course, you then could create simple templates, e.g. a user could write
There is a {{hide in print|unprintable}} {{only in print|printable}} word.
Template:Hide in print:
<includeonly><span class="noprint">{{{1}}}</span></includeonly>
Template:Only in print:
<includeonly><span class="onlyinprint">{{{1}}}</span></includeonly>
Read again
Add in Localsettings.php (if not already done) the rendering engines you will support. In case you also installed a local rendering server, typing mw-render --list-writers will list the ones you installed.
Example setting for Localsettings.php:
$wgCollectionFormats = array( 'rl' => 'PDF', 'odf' => 'ODT', );
In Localsettings.php, if not already done so, add the name and port of the mwlib server.
$wgCollectionMWServeURL = "http://xxx.yyy:8899";
Extra stuff:
For the license, make sure to give the correct RAW wiki URL. Or if this doesn't work, remove the line and the user will see a URL.
$wgLicenseURL = "http://edutechwiki.unige.ch/mediawiki/index.php?title=EduTech_Wiki:Copyrights&action=raw";
Limit number of articles that a book can contain, e.g.
$wgCollectionMaxArticles = 150;
Tested in this server, using Ubuntu 10.04.4 LTS, 12.04.2 LTS, 14.04 LTS, 16.01 LTS (server editions). Instructions below are only valid for Ubuntu 14 LTS and have been more or less updated for 16 LTS and 18 LTS. Daniel K. Schneider (talk)
Notice: In the documentation, mwlib has been replaced in 2015 by OCG on Wikipedias (at least that's what they say). However, OCG seems to be less stable so far and maybe requires even more installation efforts. However, as of sept. 2018, this "new" product that we actually never used, seems to be dead. So, until Pediapress comes up with a new solution, we will stick to mw-lib - Daniel K. Schneider (talk) 11:22, 13 September 2018 (CEST)
Since my first install unil the time of writing (dec 2012 / august 2013 / April 2015), the manual at PediaPress is fine and should include everything a system person needs to know in order to understand and configure the service. However, the installation instructions are not good enough. The pip installer will find Python packages but cannot add missing Ubuntu libraries.
mwlib refers to a whole series of software that will allow:
This section describes the easy way, however easy does not mean real easy. Depending on your Ubuntu version and the state of your server, you could suffer:
Have root access and become root (or use 'sudo ...')
Install/check python
Firstly, you must have python 2.7.* installed (usually already done) plus the pip packaging manager (not done on a fresh machine), plus a lot of graphics and text libraries.
apt-get install python apt-get install python-pip
pip is a tool for installing and managing Python packages, such as those found in the Python Package Index. It's a replacement for easy_install that was used in the past.
In case pip is installed, make sure that you got the latest version
pip install --upgrade pip
In case the Ubuntu pip version doesn't work, you could install a fresh one (this happened to me sometimes in the past)
git clone https://github.com/pypa/pip.git cd pip python setup.py install
Dependencies
(this is not complete, see the Ubuntu package list below)
Install Ubuntu packages
Below is a complete list of the Ubuntu packages I installed sometimes in the past. Some is already on the system, but that's neither a problem for apt-get nor for your server. If you get error messages when you install mwlib (see below), then it is likely that some packages are missing or that you got doubles installed (from non-standard setup procedures)
sudo apt-get install -y gcc \ dvipng\ g++\ git-core\ imagemagick\ libevent-dev\ libfreetype6-dev\ libjpeg-dev\ liblcms-dev\ libxml2-dev\ libxslt-dev\ libz-dev\ make\ ocaml-nox\ pdftk\ ploticus\ python-all-dev\ python-dev\ python-imaging\ python-lxml\ python-pip\ python-virtualenv\ python\ tcl\ texlive-latex-recommended\ tk webp\
Below some extra python stuff that seems to be needed.
pip install greenlet pip install gevent
Pillow, (the PIL Python imaging library) seems to be installed from PediaPress.
(written for Ubuntu 14, but should also be ok for Ubunt 18)
pip install --trusted-host pypi.pediapress.com -i http://pypi.pediapress.com/simple/ mwlib : This will take some time, since files have to be compiled pip install --trusted-host pypi.pediapress.com -i http://pypi.pediapress.com/simple/ mwlib.rl
For each missing piece, try to upgrade, e.g. for apipkg:
pip install --upgrade apipkg
In addition, you may have to remove packages, see below. You also may have to remove/reinstall mwlib and mwlib.rl if you did something wrong ...
After fixing missing dependencies, e.g. a missing image library, you may have to force upgrade. Successfully installed python packages that will break during runtime, will not upgrade without --force-reinstall.
pip install --upgrade --trusted-host pypi.pediapress.com --force-reinstall -i http://pypi.pediapress.com/simple/ pil pip install --upgrade --trusted-host pypi.pediapress.com --force-reinstall -i http://pypi.pediapress.com/simple/ mwlib pip install --upgrade --trusted-host pypi.pediapress.com --force-reinstall -i http://pypi.pediapress.com/simple/ mwlib.rl
Gevent problem
gevent.coros was missing, mw-lib used a dead module of gevent :( So I had to backtrack
pip install gevent==1.0.2
Some Ubuntu 16 LTS problems (you can most likely skip these)
Then I encountered another ugly problem: requirements.py fails. Edit the file and make a change
cd /usr/local/lib/python2.7/dist-packages/packaging/ cp requirements.py requirements.old.py
Edit the requirements.py file and change the following
old: MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker") new: MARKER_EXPR = originalTextFor(MARKER_EXPR(""))("marker")
Warning: This is not a good idea, but it's what I found.
Some python libraries did change place or could be outdated. I suggest uninstalling and reinstalling.
sudo -H pip uninstall lxml sudo pip uninstall greenlet
Then install them again. As you will see, some of these are already there...
sudo -H pip install lxml sudo -H pip install greenlet
(thanx to Kghbln !)
Step 1 - reinstall and upgrade "pil"
pip install --trusted-host pypi.pediapress.com --upgrade --force-reinstall -i http://pypi.pediapress.com/simple/ pil
Step 2 - upgrade "mwlib"
sudo su pip install --trusted-host pypi.pediapress.com -i http://pypi.pediapress.com/simple/ --upgrade mwlib
Step 3 - upgrade "mwlib.rl"
pip install --trusted-host pypi.pediapress.com -i http://pypi.pediapress.com/simple/ --upgrade mwlib.rl
Step 4 - update "Mwlib server daemon"
per suggestion of the example below
Step 5 - create user "mw-serve" since "www-data" will no longer work
adduser mw-serve
Step 6 - change permissions of the log directory
chown mw-serve:mw-serve -R /var/log/mediawiki/
Step 7 - start render servers
/etc/init.d/mw-serve start
Step 8 - enjoy
Only works with a recent version of pip, e.g. pip-2.6 or better !!
Syntax:
pip uninstall package_name
Sometimes, you may just have to remove packages manually (probably older not finished installs ?)
cd /usr/local/lib/python2.6/dist-packages rm -r package_name
.... but this is not recommended IMHO
Get something from Wikipedia (this is the default)
mw-zip -c :en -o test.zip Acdc Number mw-render -c test.zip -o test.pdf -w rl
Look at test.pdf and enjoy ....
xpdf test.pdf
Try it on yours (however, this does not work anymore when I tried last time (Ubuntu 18, sept. 2018)
mw-zip -c http://edutechwiki.unige.ch/mediawiki/ -o test2.zip Mediawiki_collection_extension_installation mw-render -c test2.zip -o test2.pdf -w rl xpdf test2.pdf
On the machine
Firstly, create a directory where files can be collected. Then, make them writable for the process that will run mwlib. For example:
mkdir /data/mwcache chown www-data:www-data mwcache/
The following will launch the combo. Each program has various options. At least for testing, I suggest to accept the defaults. However, define a cache directory for the nslave.
nserve & mw-qserve & # directory mwcache must be writable to the user that runs the servers nslave --cachedir /data/mwcache & postman --cachedir /data/mwcache &
Note about older versions
In LocalSettings.php of the wiki
You must add the following line, since by default your wiki will use the PediaPress rendering service:
$wgCollectionMWServeURL = "http://machine.you.domain:port";
For example (using the default port):
$wgCollectionMWServeURL = "http://edutechwiki.unige.ch:8899";
Log information output by the render services
All these servers will write back log information to the console. E.g. if you test creating a simple PDF file you might see something like this (slightly formatted and edited, version with the old file names for scripts.):
=== report Wed Dec 12 14:15:13 2012 === have 0 jobs count: 0 all channels idle new-collection 1 'http://edutechwiki.unige.ch/mediawiki' 'rl' 2012-12-12T14:15:14 mwlib.serve.info >> render 9da9af29fbb8b623 rl 129.194.xxx- - [2012-12-12 14:15:14] "POST / HTTP/1.0" 200 200 0.012274 129.194.xxx- - [2012-12-12 14:15:14] "POST / HTTP/1.0" 200 215 0.002743 0 1.42332196236 ['mw-zip', '-o', '/data/mwcache/9d/9da9af29fbb8b623/collection.zip', '-m', '/data/mwcache/9d/9da9af29fbb8b623/metabook.json', '--status', 'qserve://localhost:14311/9da9af29fbb8b623:makezip', '--config', 'http://edutechwiki.unige.ch/mediawiki', '--template-blacklist', 'MediaWiki:PDF Template Blacklist', '--template-exclusion-category', 'Exclude in print', '--print-template-prefix', 'Print', '--print-template-pattern', '$1/Print'] finish: 9da9af29fbb8b623:makezip: None 0 2.253221035 ['mw-render', '-w', 'rl', '-c', '/data/mwcache/9d/9da9af29fbb8b623/collection.zip', '-o', '/data/mwcache/9d/9da9af29fbb8b623/output.rl', '--status', 'qserve://localhost:14311/9da9af29fbb8b623:render-rl', '--template-blacklist', 'MediaWiki:PDF Template Blacklist', '--template-exclusion-category', 'Exclude in print', '--print-template-prefix', 'Print', '--print-template-pattern', '$1/Print', '--language', 'en'] finish: 9da9af29fbb8b623:render-rl: {'url': 'http://129.194.7.75:8898/cache/9d/9da9af29fbb8b623/output.rl', 'suggested_filename': 'Mediawiki collection extension installation', 'size': 125658} watchdog: dropped 0 jobs, marked 2 jobs with a deadline [....] === report Wed Dec 12 14:15:33 2012 === have 2 jobs count: 2 all channels idle [.....] === report Wed Dec 12 14:16:53 2012 === have 2 jobs count: 2 all channels idle 129.194.xxx - - [2012-12-12 14:16:59] "POST / HTTP/1.0" 200 565 0.039982 129.194.xxx - - [2012-12-12 14:17:01] "POST / HTTP/1.0" 200 565 0.002620 129.194.xxx - - [2012-12-12 14:17:01] "GET /cache/9d/9da9af29fbb8b623/output.rl HTTP/1.0" 200 125919 0.001031
Killing jobs launched from a terminal
jobs [1] Running nserve.py & (wd: /data/portails/mediawiki) [2] Running mw-qserve & (wd: /data/portails/mediawiki) [3]- Running nslave.py --cachedir /data/mwcache & [4]+ Running postman.py --cachedir /data/mwcache &
kill %1 kill %2 etc.
(not yet done, weekend is coming up - Daniel K. Schneider (talk) 19:21, 10 April 2015 (CEST)
You could imagine redirecting logfiles to a log file, running the processes under a better user than root and also writing a start/stop script in case you boot.
In fact, Pediapress suggests using runit for process supervision. daemontools is similar solution. Another alternative is to use supervisor. The advantage of such a solution is that processes are automatically restarted after a process dies (that can happen for various reasons). Installing supervisor is what I personally would attempt since it doesn't seem to interfere much with standard practice from what I could understand and it doesn't look too complicated. Anyhow, in the meantime I made this script (suggestions welcome, I really am not a sysadmin)
Prerequisites:
chown mw-serve:mw-serve -R /var/log/mediawiki/
File etc/init.d/mw-serve used on a typical installation
#!/bin/sh
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
case "$1" in
start)
su - mw-serve -c 'nserve >> /var/log/mediawiki/mw-serve_log.txt 2>&1 &'
su - mw-serve -c 'mw-qserve >> /var/log/mediawiki/mw-serve_log.txt 2>&1 &'
su - mw-serve -c 'nslave --cachedir /var/log/mediawiki/ >> /var/log/mediawiki/mw-serve_log.txt 2>&1 &'
su - mw-serve -c 'postman --cachedir /var/log/mediawiki/ >> /var/log/mediawiki/mw-serve_log.txt 2>&1 &'
;;
stop)
mv /var/log/mediawiki/mw-serve_log.txt /var/log/mediawiki/mw-serve_log.txt.old
killall nserve
killall mw-qserve
killall nslave
killall postman
;;
force-reload|restart)
$0 stop
$0 start
;;
*)
echo "Usage: /etc/init.d/mw-serve {start|stop}"
exit 1
;;
esac
exit 0
File /etc/init.d/mw-serve used locally for EduTechWiki
#! /bin/bash PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin case "$1" in start) su - www-data -c 'nserve >> /data/mwcache/log.txt 2>&1 &' su - www-data -c 'mw-qserve >> /data/mwcache/log.txt 2>&1 &' su - www-data -c 'nslave --cachedir /data/mwcache/ >> /data/mwcache/log.txt 2>&1 &' su - www-data -c 'postman --cachedir /data/mwcache/ >> /data/mwcache/log.txt 2>&1 &' ;; stop) mv /data/mwcache/log.txt /data/mwcache/log.old killall nserve killall mw-qserve killall nslave killall postman ;; force-reload|restart) $0 stop $0 start ;; *) echo "Usage: /etc/init.d/mw-serve {start|stop}" exit 1 ;; esac exit 0
Make it executable
chmod 755 /etc/init.d/mw-serve
Then link it in the appropriate RC directories, most importantly:
cd /etc/rc1.d ln -s ../init.d/mw-serve K20mw-serve
cd /etc/rc2.d ln -s ../init.d/mw-serve S20mw-serve
cd /etc/rc3.d ln -s ../init.d/mw-serve S20mw-serve
It is best just to "hand edit" a stored book. E.g. you may start by adding a category or isolated articles to a book with the collection interface, but once you got most of the articles:
{{saved_book}} == My Book == === Example === ;Foo :[[First article]] :[[Second article]] :[[Third article|This article renamed in the book]] ;Bar :[[Fourth article]] :[[Fifth article]] :[{{fullurl:Sixth article|oldid=20}}Sixth article version:20] [[Category:Books]]
If you have articles that include computer source code (XML, ActionScipt or whatever), you also should install the source extension (CodeSyntaxHighlight MediaWiki for formatting computer code.
If you use XML within "pre" tags, the parser may become confused.
Also, the printed Pediapress book will look much prettier. Editing all your wiki pages may represent work, but user reading the on-line pages also will be grateful, colored and indented code is really much more readable !
(1) Readjust some image sizes
I have the suspicion that this extension cannot handle pictures that are typical for cell phones, e.g. 1 840 × 3 552 pixels (4K). Try reducing the size of the uploaded image.
(I'll have to be more precise about this, but I'll first need to analyze both a wiki book I got and the generated PDF...)
(2) Color Printed PDF is mostly grey (affordable Pediapress books certainly are). E.g. when you create drawings or annotate images with text, you should make sure that shades of grey still allow to identify critical elements. A related very difficult issue is writing the text. Avoid writing about "blue" and "green" arrows and "green" or "yellow" dots ...
(3) Galleries may not work.
Workaround: Use tables instead. However, the latter also may encounter problems
But before that try using widths=.... and heights=.... parameters. It seems (as of MW 1.27 / April 2018) that simple galleries with these parameters do work, unless they are too large or include a perrow attribute. Try creating several galleries instead of using perrow ?
Always use the file: (or fichier:, in french) prefix. Without it, it will ignore the line with the picture inside.
As of mid May 2009 (so this may change)
Exemple (discussion page fr:Stitch Era - logiciel de broderie machine et de hotfix)
<graphviz border="frame" caption="Flux pour créer un design de broderie avec Stitch Era" alignment="left" location="none">
graph SE_workflow_francais_2 {
labelfontname = "Arial";
node [fontsize="24", fontname="Arial"];
edge [fontsize="22", fontname="Arial", labelfloat=false];
...
Example link: [[File:Discussion Stitch Era logiciel de broderie machine et de hotfix graph SE workflow francais 2 dot.png|900px|thumb|none|Modèle workflow pour créer une broderie avec Stitch Era]]
Use the following CSS classes in a wiki page. You also can use this within templates. E.g. an example is the in tutoriel template of the french sister wiki.
Only in print
<div class="onlyinprint">
...
</div>
Hide in print
<div class="noprint">
.....
</div>
You should modify the CSS of your wiki in MediaWiki:Common.css and the following line:
/* collection extension */
.onlyinprint {display: none}
Excluding page sections
Placing a div before a == title == does not seem to work.
Workaround: Use <h2> instead