KDE Buildsystem
From The Chakra Project - Wiki
THIS PAGE IS IN THE PROCESS OF BEING UPDATED AND NOT COMPLETE YET. ONLY THE BASIC STEPS OF SETTING UP AND TESTING THE LATEST VERSION OF OUR BUILD ENVIRONMENT ARE FINISHED FOR NOW. THAT MEANS YOU CAN BUILD PACKAGES, BUT THERE IS NO DOCUMENTATION ABOUT THE STRUCTURE OF THE ENVIRONMENT OR THE AUTOMATED BUILD MECHANISMS YET. THESE INFORMATIONS WILL BE ADDED (VERY) SOON.
Contents |
Features
- Chrooted build environment.
- Clean and separated. No changes on the build machine are needed, just one new directory will be created.
- Transparent creation of separate packages with debug symbols
- Does not run only on Arch but every Linux distro that has pacman.static installed
- Automatic dependency handling for depends and makedepends
- Does not break functionality. All standard makepkg stuff works, like repackaging for example
- Flexibility, like creating a complete KDE package set with just one command (from tarballs or SVN tag/trunk, even if there is no KDE installed yet)
- Support for stable branch updates, if needed. (this works only with KDE currently)
- Automatic repo handling (creating/updating databases, cleaning/moving packages, atomic uploads)
- Install/reinstall/uninstall complete chroots with just one command
- Simple. Not over-engineered but clear, straight and understandable scripts, even for newbies.
- A nice setup to get started in almost no time
How split packages are built
This feature is now available in the official Arch makepkg implementation, heavily inspired by our solution. Because there is no need to have two different implementations/branches, our makepkg is now based on the Arch implementation.
How separated debug packages are built
Well, thats another story and out of the scope of this document, but you can check the upstream feature request at the Arch Bugtracking System for more information:
http://bugs.archlinux.org/task/10975
Before you start
Our buildscripts may look a bit "Alien" when first looking at them, so dont despair ;) They are really easy to understand and you will just love the flexibility provided by them. Just be patient when figuring them out, the more you see of them, the more logical it gets :)
Setting up the Buildsystem
If you can remember one of the first versions, you may still have the complicated setup steps in your mind which were one of the key features of the old build system (just kidding). Well, with the current versions this all has changed and we provide a fancy setup script for it... Now, if you dont mind, lets start with the setup...
1 - Create a new directory in your $HOME or somewhere else where your user can write. Your development directory is also ok, just one additional directory will be created.
2 - Download the following script:
Outdated? (but working):
wget http://chakra-project.org/svn/packages/_buildsystem/scripts/buildenv-setup.sh
Latest (needs more testing):
wget http://chakra-project.org/code/index.php/view/packages:/_buildsystem/scripts/buildenv-setup.sh
3 - Place the script into the aforementioned directory
4 - Make the script executable and run it:
chmod +x buildenv-setup.sh ./buildenv-setup.sh
5 - This is the output from just running it without any parameters:
$ ./buildenv-setup.sh :: starting up, this could take some secs... :: running on arch linux :: using pacman :: loading package information :: running on user: jan :: you need to specify a repository: chakra core extragear playground testing tools unstable
As you can see, a repository is needed, so we just add that one on the next step. And while we are at it, we also add the target architecture:
6 - Run the script to install a build environment for [kdemod-core]:
$ ./buildenv-setup.sh core i686
7 - Now the script will start up and ask for your password:
$ ./buildenv-setup.sh core i686 :: starting up, this could take some secs... :: running on arch linux :: using pacman :: loading package information :: running on user: jan
:: Initializing sudo, you will be asked for your password
Password:
8 - Enter your password to move forward to the next step that looks like this:
>> KDEmod Build Enviroment - Setup-o-Matic v0.4.4.0-1
:: This script will create a dir called KDEmod in the current directory :: and install a chrooted build environment for kdemod-core. Please do :: not move this installer script after creating the first chroot. You can :: use it later to install/reinstall/uninstall chroots/repos.
:: Installation dir: /home/jan/test/KDEmod :: (^^the base dir containing everything related to KDEmod^^)
:: Repository dir: /home/jan/test/KDEmod/core-i686 :: (^^the installation directory of this repository^^)
:: Do you want to continue? (y/n)
9 - If everything seems to be correct, enter "y" to proceed to the next step (otherwise press "n" to exit). Now the next question will be asked:
:: Do you want to continue? (y/n) y
>> Creating Chroot: core-i686
:: Do you want to install KDEmod into the chroot? :: (in this case the script will run "pacman -S kdemod automoc4") :: Answer with no here if you are building a KDEmod [core|testing|unstable] repo from scratch.
:: (y/n)
In case you want to compile your own packages from scratch and start in a clean chroot, answer with "n" here. Otherwise, if you answer with "y", the script will at first install a basic Arch Linux and then a complete KDEmod on top of it.
In this howto we will answer this question with "n" and install KDEmod after the main installation has been finished:
10 - After answering the question with "n", the script starts working and installs a Arch Linux chroot. Here is the output of this procedure, as you can see, just follow pacmans instructions and install everything:
>> Creating Chroot: core-i686
:: Do you want to install KDEmod into the chroot? :: (in this case the script will run "pacman -S kdemod automoc4") :: Answer with no here if you are building a KDEmod [core|testing|unstable] repo from scratch.
:: (y/n) n
:: creating special dirs DONE :: mounting special dirs DONE :: creating pacman.conf DONE :: installing archlinux base system :: follow pacman instructions from here
:: Synchronizing package databases... kdemod-core 60.1K 551.0K/s 00:00:00 [############################################] 100% core 36.0K 278.8K/s 00:00:00 [############################################] 100% extra 448.1K 906.4K/s 00:00:00 [############################################] 100% community 359.2K 559.2K/s 00:00:01 [############################################] 100% resolving dependencies... looking for inter-conflicts...
Targets (95): A LOT OF PACKAGES WILL BE LISTED HERE
Total Download Size: 0.00 MB Total Installed Size: 279.33 MB
Proceed with installation? [Y/n] Y checking package integrity... (95/95) checking for file conflicts [############################################] 100% ( 1/95) installing linux-api-headers [############################################] 100% ( 2/95) installing tzdata [############################################] 100% ... ... ... ...
:: Synchronizing package databases... kdemod-core is up to date core is up to date extra is up to date community is up to date base-devel package not found, searching for group... :: group base-devel (including ignored packages): autoconf automake bin86 bison ed fakeroot flex gcc libtool m4 make patch pkgconfig :: Install whole content? [Y/n] Y xorg package not found, searching for group... :: group xorg (including ignored packages): xf86-video-vesa xorg-docs xorg-fonts-100dpi xorg-fonts-75dpi xorg-res-utils xorg-server xorg-server-utils xorg-twm xorg-utils xorg-xinit xterm :: Install whole content? [Y/n] Y resolving dependencies... looking for inter-conflicts...
Targets (132): A LOT OF PACKAGES WILL BE LISTED HERE
Total Download Size: 45.81 MB Total Installed Size: 419.02 MB
Proceed with installation? [Y/n] Y checking package integrity... (132/132) checking for file conflicts [############################################] 100% ( 1/132) installing run-parts [############################################] 100% ( 2/132) installing ca-certificates [############################################] 100% ... ... ... ...
11 - After the basic installation has been finished, the script configures the just installed chroot:
:: skipping kdemod package installation :: configuring system DONE :: adding user: jan DONE :: setting up /etc/sudoers DONE :: unmounting special dirs DONE
>> Creating buildsystem :: creating needed directories DONE :: fetching buildsystem from SVN DONE :: creating sources dir DONE :: compiling additional package(s): repo-clean :: ********* DONE :: installing additional package(s): repo-clean DONE :: fetching PKGBUILDs from SVN DONE
>> Preconfiguring buildsystem :: creating directories DONE :: installing makepkg DONE :: installing chroot configs DONE :: installing chroot scripts DONE
>> Configuring buildsystem for: core-i686 :: creating config files DONE :: creating user symlinks DONE :: 2 cpu('s) detected, setting MAKEFLAGS to -j3 :: setting up makepkg config DONE :: setting up repo config DONE :: setting up buildsystem config DONE :: finishing... DONE
>> All done! :: Now open _buildsystem/core-i686_makepkg.conf and edit the :: DLAGENTS, CFLAGS, CXXFLAGS and PACKAGER settings to your :: liking and you are ready to build packages :)
:: (Very) Quick Start: :: ------------------- :: 1 -> cd KDEmod :: 2 -> ./enter-chroot.sh core-i686 :: 3 -> cd package :: 4 -> ../makepkg
Please notice that sometimes the script will ask for your password again during the installation. This often happens on slower machines. If that happens, dont panic but just enter your password again.
12 - You have now installed your first chroot, in this case for [kdemod-core]. Please dont move the installation script after the first run, as you can use it later to install/reinstall/uninstall chroots/repos.
13 - Now we are going to enter the just installed chroot:
$ cd KDEmod/ $ ./enter-chroot.sh core-i686
:: entering chroot: core-i686
Password:
14 - After entering your password, you will see something like this:
_buildsystem kdebindings show-pkglists.sh _repo kdeedu support-google-gadgets-qt _sources kdegames support-libindicate _supportpkg kdegraphics support-libindicate-qt build.sh kdelibs support-meanwhile check-kde-version-in-svn-trunk.sh kdemultimedia support-mediastreamer clean-builddir.sh kdenetwork support-qedje clean-kde-branch-patches.sh kdepim support-qjson clean-workdir.sh kdepim-runtime support-qzion create-kde-tarballs-from-svn-tag.sh kdepimlibs support-xkit create-l10n-tarballs-from-svn-tag.sh kdeplasma-addons sync-complete.sh decrease-pkgrels.sh kdesdk sync-down.sh get-tarballs.sh kdetoys sync-up.sh increase-pkgrels.sh kdeutils test-session.sh kde-common kdewebdev themes kde-l10n makepkg tools-aqpm2-git kde-oxygen-icons only-up.sh tools-arxin kdeaccessibility qtmod tools-chase-git kdeadmin qtmod-phonon tools-kgrubeditor kdeartwork repoclean-builddir.sh tools-networkmanagement kdebase repoclean-repodir.sh tools-shaman2-svn kdebase-runtime reset-pkgrels.sh update-kde-branch-patches.sh kdebase-workspace show-config.sh
jan@[chroot]:~/kdemod/core$
You are now inside the builddir of the [kdemod-core] chroot. Before building anything, we just need a tiny bit of configuration:
15 - Edit _buildsystem/core-i686_makepkg.conf and fill out the CFLAGS, CXXFLAGS and PACKAGER settings:
jan@[chroot]:~/kdemod/core$ nano _buildsystem/core-i686_makepkg.conf
16 - Check if repo-clean has been correctly compiled and installed:
jan@[chroot]:~/kdemod/core$ repo-clean Source directory not defined! Current directory will used as source. Total: 0 Ignored: 0 Corrupted: 0 Can not find packages with several versions. Nothing to do.
If that is not the case, you need to build repo-clean manually:
jan@[chroot]:~/kdemod/core$ pushd _buildsystem/tools/repo-clean/ jan@[chroot]:~/kdemod/core/_buildsystem/tools/repo-clean$ rm -rf pkg src *.pkg.tar.gz *.pkg.tar.xz jan@[chroot]:~/kdemod/core/_buildsystem/tools/repo-clean$ makepkg jan@[chroot]:~/kdemod/core/_buildsystem/tools/repo-clean$ pacman -U *.pkg.tar.xz jan@[chroot]:~/kdemod/core/_buildsystem/tools/repo-clean$ popd
Then test again, repo-clean should now work correctly.
17 - Because we did not install KDEmod during the installation of the chroot, we can do that now. As said, this step is not necessary if you want to compile your own packages from scratch. If you just want to update one or another package, you will need to install KDEmod. To do that, just run:
pacman -S kdemod-complete
And you'll need automoc4, run:
pacman -S automoc4
After all is installed run:
./sync-down.sh
18 - Thats it! You are now ready to build packages!
19 - Now, lets build a package to test it. We start with a standard one and use kde-common as an example:
cd kdemod-common ../makepkg
Of course building kdemod-common requires some more dependencies like Qtmod to be installed, so this test works only if you have installed KDEmod into the chroot like described before. If you havent installed KDEmod into the chroot, you need to build all dependencies at first, a good start is the qtmod package:
cd qtmod ../makepkg
I hope you noticed how makepkg has been called in the examples. This is a must, always call it like this...
20 - Congrats! You have just mastered the basic entrance to our build scripts.
Dont forget to run both updater scripts from time to time, to update the buildsys and the chrooted arch systems:
cd kdemod ./update-buildsystem.sh ./update-chroot.sh
THE FOLLOWING INFORMATION IS OUTDATED AND WILL BE IMPROVED/CORRECTED SOON!
OUTDATED: Configs
All of our scripts and PKGBUILDs are using just two config files in _buildsystem: buildsystem.conf and kdemod.conf.
- buildsystem.conf contains all settings for the build-scripts, like package lists, SVN addresses, the name of the database et cetera. Its actually pretty easy to understand, just check it out!
- kdemod.conf contains all settings that will be sourced directly from the PKGBUILDs, like the KDE version or the FTP-URL for sources... Also easy to understand.
OUTDATED: Scripts
Inside the work directory you can find a lot of scripts that are used to automate builds, repo handling/updating and uploads. Here is a small explanation for all of them. For a usage example, check the next section on this page.
- build-all.sh - builds and installs all packages defined in BUILD_STAGE_ALL in buildsystem.conf
- build-kde-and-l10n.sh - builds and installs all KDE packages defined in BUILD_STAGE_KDE in buildsystem.conf and all KDE localizations
- build-kde.sh - builds and install all KDE packages defined BUILD_STAGE_KDE in buildsystem.conf
- build-qt.sh - builds and install all Qt packages defined in BUILD_STAGE_QT in buildsystem.conf
- build-tools.sh - builds and installs all tool packages defined in BUILD_STAGE_TOOLS in buildsystem.conf
- clean-branch-patches.sh - cleans out all branch patches if branch updates are enabled.
- clean-repo-pkgs.sh - cleans out all temporary package locations (/packages/build and /packages/upload)
- clean-workdir.sh - cleans up the build environment and deletes all pkg/src/dbg dirs
- create-kde-tarballs-from-svn-tag.sh - creates ready-to-use KDE tarballs from the svn tag configured in buildsystem.conf. (Section SETTINGS FOR SVN TAG -> TARBALL CREATION)
- create-kde-tarballs-from-svn-trunk.sh - creates ready-to-use KDE tarballs from svn trunk configured in buildsystem.conf. (Section SETTINGS FOR SVN TRUNK -> TARBALL CREATION)
- create-l10n-tarballs-from-svn-tag.shcreates ready-to-use KDE localization tarballs from the svn tag configured in buildsystem.conf. (Section SETTINGS FOR SVN TAG -> TARBALL CREATION)
- decrease-all-pkgrels.sh - decreases the pkgrel by 1 on all PKGBUILDs/SPLITBUILDs
- decrease-kde-pkgrels.sh - decreases the pkgrel by 1 on all KDE PKGBUILDs/SPLITBUILDs
- decrease-qt-pkgrels.sh - decreases the pkgrel by 1 on all Qt PKGBUILDs/SPLITBUILDs
- decrease-tools-pkgrels.sh - decreases the pkgrel by 1 on all Tool PKGBUILDs/SPLITBUILDs
- increase-all-pkgrels.sh - increases the pkgrel by 1 on all PKGBUILDs/SPLITBUILDs
- increase-kde-pkgrels.sh - increases the pkgrel by 1 on all KDE PKGBUILDs/SPLITBUILDs
- increase-qt-pkgrels.sh - increases the pkgrel by 1 on all Qt PKGBUILDs/SPLITBUILDs
- increase-tools-pkgrels.sh - increases the pkgrel by 1 on all Tool PKGBUILDs/SPLITBUILDs
- prepare-local-repo.sh - creates/updates the local repo in /packages/repo with newly built packages in /packages/build and puts all created/updated packages and a created/updated pacman db into /packages/upload
- reset-all-pkgrels.sh - resets all pkgrels to 1
- reset-kde-pkgrels.sh - resets all KDE pkgrels to 1
- reset-qt-pkgrels.sh - resets all Qt pkgrels to 1
- reset-tools-pkgrels.sh - resets all Tool pkgrels to 1
- show-config.sh - shows the buildsystem configuration
- update-kde-branch-patches.sh - in case the branch patches are enabled in kdemod.conf, you can create/update them with this script
- update-pkgbuilds-from-svn.sh - updates the build environment from our SVN
- upload-pkgs.sh - uploads packages, see the next section for an explanation
OUTDATED: Scripts Usage
A typical workflow with the scripts looks like this...
To update the chroot, before building:
svn up pacman -Sc pacman -Syu ./sync-down.sh
Then check with desktop: [1]
Imagine that you want to automate a KDE build from an SVN tag, create a working repo and upload it somewhere... At first, check buildsystem.conf and kdemod.conf if all versions are set up correctly. If you just want to rebuild one of our current packages from our repos, no changes are required.
Well, lets follow these steps:
- build and install the repo-clean PKGBUILD from _buildtools/repo-clean
- run ./create-kde-tarballs-from-svn-tag.sh - this will create ready-to-use tarballs from the svn tag configured in buildsystem.conf. (Section SETTINGS FOR SVN TAG -> TARBALL CREATION). Beware, it will take a lot of time to fetch and build the tarballs.
- in case you also need localization packages, run ./create-l10n-tarballs-from-svn-tag.sh - this will take even more time than building the other tarballs.
- run ./clean-workdir.sh - this will clean up the build environment and deletes all pkg/src/dbg dirs
- build some packages, or run one of the build-* scripts like ./build-kde.sh, which will build all KDE modules and place the packages into /packages/build
- run ./prepare-repo.sh - this will copy the new packages from /packages/build/ to /packages/repo, then run repo-clean over it and finally creates a pacman database. When this has been finished, the new packages and the db will be put into /packages/upload and are ready to be uploaded. Be sure to setup BUILD_REPO in buildsystem.conf, it is simply the name of the db that will be created.
- run ./upload-pkgs.sh - this will upload the new/updated packages and the pacman db to a FTP server, which has been configured in buildsystem.conf. Here is an example entry:
# # UPLOAD SETTINGS # BUILD_UPLOAD_SERVER="example-server.org" BUILD_UPLOAD_DIR="/the/dir/on/the/server" BUILD_UPLOAD_USER="the_ftp_username" BUILD_UPLOAD_PASSWORD="the_ftp_password"
For subsequent builds, just increase the pkgrel of the packages and build them. They will be placed into /packages/build. Then run ./prepare-repo.sh again to update the repo and place the updated packages into /packages/upload, where they can be uploaded by ./upload-pkgs.sh again.
Basically, /packages/repo is a local copy (or better: the master copy) of the repo on the server. And just to be sure you understand it, a typical workflow looks like this
- build packages (either with the scripts or manually)
- run ./prepare-repo.sh to create/update the local repo
- run ./upload-pkgs.sh to upload the created/updated packages and the pacman db
OUTDATED: Building KDE trunk with the scripts
To build KDE packages from SVN trunk, just follow these steps:
- run ./check-kde-version-in-trunk.sh and add the version to the _kdever variable in _buildsystem/kdemod.conf
- run ./create-kde-tarballs-from-svn-trunk.sh to checkout the source and create tarballs out of it
- now you can proceed as described in the scripts usage part of this howto
Trunk is always in-flux, so be prepared for massive changes occurring randomly ;) Also check _buildsystem/buildsystem.conf for all the trunk-related settings (Section SETTINGS FOR SVN TRUNK -> TARBALL CREATION)