Wednesday, November 30, 2011

Mutex vs. Semaphore, what is the difference?

The Toilet Example  (c) Copyright 2005, Niclas Winquist ;)

Mutex:

Is a key to a toilet. One person can have the key - occupy the toilet - at the time. When finished, the person gives (frees) the key to the next person in the queue.

Officially: "Mutexes are typically used to serialise access to a section of  re-entrant code that cannot be executed concurrently by more than one thread. A mutex object only allows one thread into a controlled section, forcing other threads which attempt to gain access to that section to wait until the first thread has exited from that section."
Ref: Symbian Developer Library

(A mutex is really a semaphore with value 1.)

Semaphore:
Is the number of free identical toilet keys. Example, say we have four toilets with identical locks and keys. The semaphore count - the count of keys - is set to 4 at beginning (all four toilets are free), then the count value is decremented as people are coming in. If all toilets are full, ie. there are no free keys left, the semaphore count is 0. Now, when eq. one person leaves the toilet, semaphore is increased to 1 (one free key), and given to the next person in the queue.

Officially: "A semaphore restricts the number of simultaneous users of a shared resource up to a maximum number. Threads can request access to the resource (decrementing the semaphore), and can signal that they have finished using the resource (incrementing the semaphore)."
Ref: Symbian Developer Library

From: http://koti.mbnet.fi/niclasw/MutexSemaphore.html

Saturday, November 5, 2011

Can you still believe it? :-)


Linux Kernel Summit 2011

Linux Kernel Summit 2011 second day official photo. I'm there on top, under the fourth from right to left.

Friday, August 5, 2011

Finally I have my books!

My books were delivered today. The DHL guy woke me up today morning, less than a week after placing my order. Amazon were fast this time. I'm happy. :-)

Saturday, July 30, 2011

Bad Amazon

On May 20 2011 I've placed an order at Amazon for three books:

C Programming Language (2nd Edition)
The Design of the UNIX Operating System [Prentice-Hall Software Series]
Operating Systems Design and Implementation (3rd Edition)

The delivery estimate was set to June 20, 2011. Today July 30, 2011, the books did not arrived yet.

I've contacted Amazon after June 20. Amazon asked me to wait until July 22. On July 25 they gave me total refund.

Well, this is my first non success story about Amazon but I'm warned about no tracking information when using Expedited International Shipping. It is almost unbelievable. How can Amazon send $ 300 USD order without having tracking information?

Yesterday I've ordered the same books plus other two. But this time I'm using the expensive Priority International Courier. The delivery estimate is: August 3, 2011 - August 9, 2011. Let's see if it will arrive this time. The new order:

C: A Reference Manual (5th Edition)
C Programming Language (2nd Edition)
Operating Systems Design and Implementation (3rd Edition)
The Design of the UNIX Operating System [Prentice-Hall Software Series]
Practical C Programming, 3rd Edition

Friday, July 8, 2011

Write and Submit your first Linux kernel Patch

In a 40 min talk, Greg Kroah-Hartman creates and submit a patch. Is this possible to do in a weekend? I'll try.

http://www.youtube.com/watch?v=LLBrBBImJt4

Is Eclipse-cdt my friend?

I'm testing Eclipse-cdt to see if helps me develop for the Kernel. Following only the first steps of:

http://wiki.eclipse.org/HowTo_use_the_CDT_to_navigate_Linux_kernel_source

Made Eclipse understand all Linux Source code. I'll need some time as the interface is richer than vi/gedit...

Monday, June 27, 2011

Adding header to kernel-headers rpm package

I wanted to distribute the header files placed on /usr/include/linux by a custom kernel-headers package.

Isn't it just put the file at /home/peter/rpmbuild/BUILD/kernel-2.6.38.fc15.new/include/linux ?

No! This is only 50% of the work. The other 50% consists of adding a line to the file:
/home/peter/rpmbuild/BUILD/kernel-2.6.38.fc15.new/include/linux/Kbuild

The line looks like:
header-y += pst_syscalls.h

And voila! Now it is done!

The expected result is:

[peter@KDPeter ~]$ rpm -qf /usr/include/linux/pst_syscalls.h
kernel-headers-2.6.38.6-27.peter4.fc15.x86_64

How to tell rpmbuild the number of simultaneous jobs? (make -j)

On Fedora 15, setting the environment variable RPM_BUILD_NCPUS allows you to configure the number up to 16 simultaneous jobs. For example to set the number of simultaneous jobs to 12, so rpmbuild will call make -j12:

$ export RPM_BUILD_NCPUS=12

But if you need more than 16 simultaneous jobs, you will need to edit ~/.rpmmacros file. To allow numbers greater than 16, I've modified this file to:


[peter@notepeter ~]$ cat .rpmmacros

%_topdir %(echo $HOME)/rpmbuild

%_smp_mflags %( \
[ -z "$RPM_BUILD_NCPUS" ] \\\
&& RPM_BUILD_NCPUS="`/usr/bin/getconf _NPROCESSORS_ONLN`"; \\\
if [ "$RPM_BUILD_NCPUS" -gt 16 ]; then \\\
echo "-j$RPM_BUILD_NCPUS"; \\\
elif [ "$RPM_BUILD_NCPUS" -gt 3 ]; then \\\
echo "-j$RPM_BUILD_NCPUS"; \\\
else \\\
echo "-j3"; \\\
fi )

%__arch_install_post \
[ "%{buildarch}" = "noarch" ] || QA_CHECK_RPATHS=1 ; \
case "${QA_CHECK_RPATHS:-}" in [1yY]*) /usr/lib/rpm/check-rpaths ;; esac \
/usr/lib/rpm/check-buildroot

There were the number 16 where is the $RPM_BUILD_NCPUS.

Reference:
Fedora Source RPM

distcc is my friend!

Last month I spent some time on distcc trying to reduce Kernel build time. The original post is here. I had no success that time. I've tried it again. Now it was a true success. :-)

distcc reduced the build time of Fedora Kernel ~ 53% when using 3 PCs instead of one. From 25m5.938s to 11m52.751s.

Details:

With distcc:
[peter@notepeter SPECS]$ cd /home/peter/distcc/
[peter@notepeter distcc]$ ls -la
total 8
drwxrwxr-x 2 peter peter 4096 Jun 27 20:33 .
drwx------. 49 peter peter 4096 Jun 27 20:48 ..
lrwxrwxrwx 1 peter peter 15 Jun 27 20:33 c++ -> /usr/bin/distcc
lrwxrwxrwx 1 peter peter 15 Jun 27 20:33 cc -> /usr/bin/distcc
lrwxrwxrwx 1 peter peter 15 Jun 27 20:33 g++ -> /usr/bin/distcc
lrwxrwxrwx 1 peter peter 15 Jun 27 20:33 gcc -> /usr/bin/distcc

[peter@notepeter distcc]$ cd -
[peter@notepeter SPECS]$ rpmbuild --clean kernel.spec
[peter@notepeter SPECS]$ export PATH=/home/peter/distcc/:$PATH
[peter@notepeter SPECS]$ export DISTCC_HOSTS='localhost 10.0.0.111 10.0.0.121'
[peter@notepeter SPECS]$ export RPM_BUILD_NCPUS=20
[peter@notepeter SPECS]$ time rpmbuild -bb --with baseonly --without debuginfo --target=`uname -m` kernel.spec

real 11m52.751s
user 20m5.209s
sys 6m47.263s

Without distcc:
[peter@notepeter SPECS]$ rpmbuild --clean kernel.spec
[peter@notepeter SPECS]$ time rpmbuild -bb --with baseonly --without debuginfo --target=`uname -m` kernel.spec

real 25m5.938s
user 69m33.427s
sys 10m57.658s

Hardware:

localhost: Core i7 2620M, 8GB RAM, 256 GB SSD, Gigabit LAN
10.0.0.111: Core2 Quad CPU Q8400, 4GB RAM, 500GB SATA, Gigabit LAN
10.0.0.121: Core2 Quad CPU Q8400, 4GB RAM, 500GB SATA, Gigabit LAN

Reference:

Faster compilation with distcc

Thursday, June 23, 2011

It works!

I've just successfully run my code into Kernel 2.6.38.6-27.

# This shows that my kernel_thread is running
[peter@KDPeter ~]$ ps -ef |grep pst
root 1373 2 0 21:09 ? 00:00:00 [pst_daemon2]


I wasn't being able to invoke my system calls from user land because I was using the obsolete _syscall1 macro instead of using syscall().

The original and old documentation that describes system calls and kernel_thread is here.

There will be new documentation soon... :-)

Sunday, June 5, 2011

Peter's "content" contribution 1

It is not Kernel code yet, but I liked it.

I added some lines to Fedora Wiki on Building Custom Kernel:
http://fedoraproject.org/wiki/Docs/CustomKernel#Building_a_Kernel_from_the_source_RPM

As I do not know about how many time this content will be available on the site, I took a screenshot of it. :-)

Monday, May 23, 2011

Your Amazon.com order has shipped

The following items have been shipped to you by Amazon.com:
--------------------------------------------------------------------

Amazon.com items (Sold by Amazon.com, LLC) :
1 C Programming Language (2n...
1 Operating Systems Design a...
1 The Design of the UNIX Ope...

via Expedited Int'l Shipping (estimated delivery date: June 20, 2011).

Steps to test custom Kernel RPM build on Fedora 15

These are minimum steps required to test if your Fedora 15 has everything it needs to build custom Kernel RPM. Lines starting with "#" are actions done with root. Lines starting with "$" are done with non-root user.

Downloading
# yum install @"Development Tools"
# yum install rpmdevtools yum-utils
$ rpmdev-setuptree
$ yumdownloader --source kernel
# yum-builddep kernel-2.6.38.6-26.rc1.fc15
# yum install qt3-devel qt3 libXi-devel

Note for Fedora 15: If you are experiencing problems related to missing dependencies, mainly on x64 systems, try to add --enablerepo=updates-testing on the end of yum commands, so it will look like:
# yum install @"Development Tools" --enablerepo=updates-testing


Preparing
$ rpm -Uvh kernel-2.6.38.6-26.rc1.fc15.src.rpm
$ cd ~/rpmbuild/SPECS
$ rpmbuild -bp --target=$(uname -m) kernel.spec
$ cd ../BUILD
$ cp -r kernel-2.6.38.fc15/ kernel-2.6.38.fc15.orig
$ cp -a kernel-2.6.38.fc15.orig kernel-2.6.38.fc15.new

$ cd ../SPECS
$ vi kernel.spec
From: http://fedoraproject.org/wiki/Docs/CustomKernel
Open the kernel.spec file for editing to give the kernel a unique name. This is important to ensure the custom kernel is not confused with any released kernel. Add a unique string to the kernel name by changing the 'buildid' line. Optionally, change ".local" to your initials, a bug number, the date or any other unique string.

Change this line:

#% define buildid .local

To this (note the extra space is removed in addition to the pound sign):
%define buildid .custom_text


Building
$ rpmbuild -bb --with baseonly --without debuginfo --target=`uname -m` kernel.spec

Copying and installing custom Kernel RPM into test VM
$ scp ../RPMS/x86_64/kernel-2.6.38.6-26.rc1.peter.fc15.x86_64.rpm root@kdpeter:/tmp; ssh root@kdpeter 'rpm -ivh --force /tmp/kernel-2.6.38.6-26.rc1.peter.fc15.x86_64.rpm'

Saturday, May 21, 2011

distcc to paralel compile the Linux Kernel

Have you heard about distcc? distcc home page

distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network. distcc should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile.


As I'll compile Linux Kernel hundreds of times in the upcoming months, and as I have two available servers, I decided to give distcc a try. It is easy to install and use. There are two modes: Regular and Pump-mode. Regular mode is easy to use but I was not able to boost the performance over 15%. The pump-mode did not work on my 64 bits system. I've tried pump-mode on 32 bits system, it worked easily but the performance gain wasn't as high as I was expecting. For now I decided to not use distcc.

But I learned about a magic performance argument for "make". It is "-jX", where X is the number of simultaneous actions that "make" will start. On x86 systems, you can set X to something between the number of cores and its double. For my notebook with Intel Core 2 Duo, setting "-j3" gave me the best results.

ADDED ON JUN 27: I got great results with distcc. See it here.

Peter's plan

I got inspired by: The origins of Linux[2]

I was bored and looking for genuine challenges! I think I found it! I'll become Linux Kernel Developer. This goal is turning my boredom into fun!

My 4 fun steps plan:
1 - Update my 2006 article about kernel modification. The updated version will follow the community rules, and will expand concepts explanation.
2 - Read three books that Linus talks about: C Programming Language, The Design of the Unix Operating System, and Operating Systems Design and Implementation.
3 - After being away from the keyboard to read the books... Start to hunt and kill Kernel Bugs. Today I would choose to do this at Device Mapper or at Memory Swapping.
4 - Becoming productive! This means being able to make patches that are accepted by the Kernel Development community.

make xconfig on Fedora 15

If you are planning to make custom kernel on Fedora 15, I suggest that you follow:

http://fedoraproject.org/wiki/Docs/CustomKernel

But the command:

su -c 'yum-builddep kernel-version.src.rpm'

Will not resolve all dependecies you need. Before running:

make xconfig

You will need to:

su -c 'yum install qt3-devel qt3 libXi-devel'

This fixes the errors:

[peter@notepeter kernel-2.6.38.fc15]$ make xconfig
CHECK qt
* Unable to find the QT4 tool qmake. Trying to use QT3
*
* Unable to find any QT installation. Please make sure that
* the QT4 or QT3 development package is correctly installed and
* either qmake can be found or install pkg-config or set
* the QTDIR environment variable to the correct location.
*

HOSTCC scripts/kconfig/conf.o
sed < scripts/kconfig/lkc_proto.h > scripts/kconfig/lkc_defs.h 's/P(\([^,]*\),.*/#define \1 (\*\1_p)/'
HOSTCC scripts/kconfig/kconfig_load.o
HOSTCC scripts/kconfig/kxgettext.o
SHIPPED scripts/kconfig/lex.zconf.c
HOSTCC scripts/kconfig/zconf.tab.o
make[1]: *** No rule to make target `scripts/kconfig/.tmp_qtcheck', needed by `scripts/kconfig/qconf.o'. Stop.
make: *** [xconfig] Error 2

AND

[peter@notepeter kernel-2.6.38.fc15]$ make xconfig
CHECK qt
* Unable to find the QT4 tool qmake. Trying to use QT3
/usr/lib64/qt-3.3/bin/moc -i scripts/kconfig/qconf.h -o scripts/kconfig/qconf.moc
HOSTCXX scripts/kconfig/qconf.o
HOSTLD scripts/kconfig/qconf
/usr/bin/ld: cannot find -lXi
collect2: ld returned 1 exit status
make[1]: *** [scripts/kconfig/qconf] Error 1
make: *** [xconfig] Error 2