3. Questions related to any architectures

3.1 Kernel Side

1. Does Linux support multi-threading? If I start two or moreprocesses, will they be distributed among the available CPUs?

   Yes. Processes and kernel-threads are distributed among processors.User-space threads are not.

2. What kind of architectures are supported in SMP?

   From Alan Cox:

   SMP is supported in 2.0 on the hypersparc (SS20, etc.) systems and Intel486, Pentium or higher machines which are Intel MP1.1/1.4compliant. Richard Jelinek adds: right now, systems have beentested up to 4 CPUs and the MP standard (and so Linux) theoreticallyallows up to 16 CPUs.

   SMP support for UltraSparc, SparcServer, Alpha and PowerPC machines isin available in 2.2.x.

   From Ralf Bächle:

   MIPS, m68k and ARM does notsupport SMP; the latter two probly won't ever.

   That is, I'm going to hack on MIPS-SMP as soon as I get a SMP box ...

3. Does SMP distribute the threads among the processors or is the library the one in charge of it?

   (Matti Aarnio) The way Linux implements threads is to treat them at scheduling the same way as any process - thread just happens to share several resources of the originating process; memory space, file descriptors. See clone(2) for part of explanation.

4. How do I make a Linux SMP kernel?

   Most Linux distributions don't provide a ready-made SMP-aware kernel,which means that you'll have to make one yourself. If you haven't madeyour own kernel yet, this is a great reason to learn how. Explaininghow to make a new kernel is beyond the scope of this document; refer tothe Linux Kernel Howto for more information. (C. Polisher)

   Configure the kernel and answer Y to CONFIG_SMP.

   If you are using LILO, it is handy to have both SMP and non-SMPkernel images on hand. Edit /etc/lilo.conf to create an entryfor another kernel image called "linux-smp" or something.

   The next time you compile the kernel, when running a SMP kernel,edit linux/Makefile and change "MAKE=make" to "MAKE=make -jN"(where N = number of CPU + 1, or if you have tons of memory/swapyou can just use "-j" without a number). Feel free to experimentwith this one.

   Of course you should time how long each build takes :-)Example:

   ---

   make configtime -v sh -c 'make dep ; make clean install modules modules_install'

   ---

   If you are using some Compaq MP compliant machines you will need to setthe operating system in the BIOS settings to "Unix

   In kernel series 2.0 up to but not including 2.1.132, uncomment theSMP=1 line in the main Makefile (/usr/src/linux/Makefile).

   In the 2.2 version, configure the kernel and answer "yes" to thequestion "Symmetric multi-processing support" (Michael ElizabethChastain).

   AND

   enable real time clock support by configuring the "RTC support" item (in"Character Devices" menu) (from Robert G. Brown). Note thatinserting RTC support actually doesn't afaik prevent the known problemwith SMP clock drift, but enabling this feature prevents lockup when theclock is read at boot time. A note from Richard Jelinek saysalso that activating the Enhanced RTC is necessary to get the second CPUworking (identified) on some original Intel Mainboards.

   AND

   (x86 kernel) do NOT enable APM (advanced power management)! APM and SMPare not compatible, and yoursystem will almost certainly (or at least probably ;)) crashwhile booting if APM is enabled (Jakob Oestergaard). AlanCox confirms this : 2.1.x turns APM off for SMP boxes. BasicallyAPM is undefined in the presence of SMP systems, and anything couldoccur.

   AND

   (x86 kernel) enable "MTRR (Memory Type Range Register) support". SomeBIOS are buggy as they do not activate cache memory for the secondprocessor. The MTRR support contains code that solves such processormisconfiguration.

   You must rebuild all your kernel and kernel modules when changing to andfrom SMP mode. Remember to make modules and makemodules_install (from Alan Cox).

   If you get module load errors, you probably did not rebuild and/orre-install your modules. Also with some 2.2.x kernels people havereported problems when changing the compile from SMP back to UP(uni-processor). To fix this, save your .config file, do makemrproper, restore your .config file, then remake yourkernel (make dep, etc.) (Wade Hampton). Do not forgetto run lilo after copying your new kernel.

   Recap:

   ---

   make config # or menuconfig or xconfigmake depmake cleanmake bzImage # or whatever you want# copy the kernel image manually then RUN LILO # or make lilomake modulesmake modules_install

   ---

5. How do I make a Linux non-SMP kernel?

   In the 2.0 series, comment the SMP=1 line in the mainMakefile (/usr/src/linux/Makefile).

   In the 2.2 series, configure the kernel and answer "no" to the question"Symmetric multi-processing support" (Michael ElizabethChastain).

   You must rebuild all your kernel and kernel modules when changing to andfrom SMP mode. Remember to make modules and makemodules_install and remember to run lilo. See notes above aboutpossible configuration problems.

6. How can I tell if it worked?

    cat /proc/cpuinfo 

   Typical output (dual PentiumII):

   ---

   processor       : 0cpu             : 686model           : 3vendor_id       : GenuineIntel[...]bogomips        : 267.06 processor       : 1cpu             : 686model           : 3vendor_id       : GenuineIntel[...]bogomips        : 267.06

   ---

7. What is the status of converting the kernel toward finergrained locking and multithreading?

   Linux kernel version 2.2 has signal handling, interrupts and some I/O stuff fine grainlocked. The rest is gradually migrating. All the scheduling is SMPsafe.

   Kernel version 2.3 (next 2.4) has really fine grained locking. In the 2.3 kernelsthe usage of the big kernel lock has basically disappeared, all majorLinux kernel subsystems are fully threaded: networking, VFS, VM, IO,block/page caches, scheduling, interrupts, signals, etc. (IngoMolnar)

8. What has changed between 2.2.x and 2.4.x kernels?

   (Mark Hahn) In many parts of the kernel, there's little relation between 2.2 and 2.4. One of the biggest changes is SMP - not just the evolutionary fine-graining of locks, but the radically revamped VM, memory management, interrupt handling that's basically unrelated to 2.2, fairly revolutionary net changes (thread and zero-copy), etc.

   In short, 2.2 doesn't use the hardware like 2.4 does.

9. Does Linux SMP support processor affinity?

   Standard kernel

   No and Yes. There is no way to force a process onto specific CPU's butthe linux scheduler has a processor bias for each process, which tendsto keep processes tied to a specific CPU.

   Patch

   Yes. Look at PSET - Processor Sets for the Linux kernel:

   The goal of this project is to make a source compatible and functionallyequivalent version of pset (as defined by SGI - partially removed fromtheir IRIX 6.4 kernel) for Linux. This enables users to determine whichprocessor or set of processors a process may run on. Possible usesinclude forcing threads to separate processors, timings, security (a`root' only CPU?) and probably more.

   It is focused around the syscall sysmp(). This function takes a number ofparameters that determine which function is requested. Functionsinclude:

   • binding a process/thread to a specific CPU
   • restricting a CPU's ability to execute some processes
   • restricting a CPU from running at all
   • forcing a cpu to run _only_ one process (and its children)
   • getting information about a CPU's state
   • creating/destroying sets of processors, to which processes may be bound

10. Where should one report SMP bugs to?

    Please report bugs to linux-smp@vger.kernel.org.

11. What about SMP performance?

    If you want to gauge the performance of your SMP system, you can run some tests made byCameron MacKinnon and available at http://www.phy.duke.edu/brahma/benchmarks.smp.

    Also have a look at this article by Bryant, Hartner, Qi andVenkitachalam that compares 2.2 and 2.3/2.4 UP and SMP kernels : SMP Scalability Comparisons of Linux¨ Kernels 2.2.14 and 2.3.99(Ray Bryant)

3.2 User Side

1. Do I really need SMP?

   If you have to ask, you probably don't. :)Generally, multi-processor systems can provide better performancethan uni-processor systems, but to realize any gains you needto consider many other factors besides the number of CPU's.For instance, on a given system, if the processor is generallyidle much of the time due to a slow disk drive, then this systemis "input/output bound", and probably won't benefit from additionalprocessing power. If, on the other hand, a system has manysimultaneously executing processes, and CPU utilization is veryhigh, then you are likely to realize increased system performance.SCSI disk drives can be very effective when used with multipleprocessors, due to the way they can process multiple commandswithout tying up the CPU. (C. Polisher)

2. Do I get the same performance from 2-300 MHz processorsas from one 600 MHz processor?

   This depends on the application, but most likely not. SMP addssome overhead that a faster uniprocessor box would not incur(Wade Hampton).:)

3. How does one display mutiple cpu performance?

   Thanks to Samuel S. Chessman, here are some useful utilities:

   Character based:

   http://www.cs.inf.ethz.ch/~rauch/procps.html

   Basically, it's procps v1.12.2 (top, ps, et. al.) and some patches tosupport SMP.

   For 2.2.x, Gregory R. Warnes as made a patch available athttp://queenbee.fhcrc.org/~warnes/procps

   Graphic:

   xosview-1.5.1 supports SMP. And kernels above 2.1.85 (included) the cpuXentry in /proc/stat file.

   The official homepage for xosview is:http://lore.ece.utexas.edu/~bgrayson/xosview.html

   You'll find a version patched for 2.2.x kernels by Kumsup Lee :http://www.ima.umn.edu/~klee/linux/xosview-1.6.1-5a1.tgz

   By the way, you can't monitor processor scheduling precisely with xosview,as xosview itself causes a scheduling perturbation. (H. PeterAnvin)

   And Rik van Riel tell us why:

   The answer is pretty simple. Basically there are 3processes involved:

   1. the cpu hog (low scheduling priority because it eats CPU)
   2. xosview
   3. X

   The CPU hog is running on one CPU. Then xosview wakes up(on the other CPU) and starts sending commands to X, whichwakes up as well.

   Since both X and xosview have a much higher priority thanthe CPU hog, xosview will run on one CPU and X on the other.

   Then xosview stops running and we have an idle CPU --> Linuxmoves the CPU hog over to the newly idle CPU (X is stillrunning on the CPU our hog was running on just before).

4. How can I enable more than 1 process for my kernel compile?

   use:

   ---

           # make [modules|zImage|bzImages] MAKE="make -jX"        where X=max number of processes.        WARNING: This won't work for "make dep".

   ---

   With a 2.2 like kernel, see also the file/usr/src/linux/Documentation/smp.txt for specific instruction.

   BTW, since running multiple compilers allows a machine with sufficientmemory to use use the otherwise wasted CPU time during I/O caused delays,make MAKE="make -j 2" -j 2 actually helps even on uniprocessorboxes (from Ralf Bächle).

5. Why is the time given by the time command inaccurate?(from Joel Marchand)

   In the 2.0 series, the result given by the time command isfalse. The sum user+system is right *but* the spreading between user andsystem time is false.

   More precisely: "The explanation is, that all time spent in processorsother than the boot cpu is accounted as system time. If you time aprogram, add the user time and the system time, then you timing will bealmost right, except for also including the system time that iscorrectly accounted for" (Jakob Østergaard).

   This bug is corrected in 2.2 kernels.

3.3 SMP Programming

Section by Jakob Østergaard.

This section is intended to outline what works, and what doesn't whenit comes to programming multi-threaded software for SMP Linux.

Parallelization methods

1. POSIX Threads
2. PVM / MPI Message Passing Libraries
3. fork() -- Multiple processes

Since both fork() and PVM/MPI processes usually do not share memory,but either communicate by means of IPC or a messaging API, they willnot be described further in this section. They are not very specificto SMP, since they are used just as much - or more - on uniprocessorcomputers, and clusters thereof.

Only POSIX Threads provide us with multiple threads sharing ressourceslike - especially - memory. This is the thing that makes a SMPmachine special, allowing many processors to share their memory. Touse both (or more ;) processors of an SMP, use a kernel-threadlibrary. A good library is the LinuxThreads, a pthread library made by Xavier Leroy which is nowintegrated with glibc2 (aka libc6). Newer Linux distributionsinclude this library by default, hence you do not have to obtain a separate package to use kernel threads.

There are implementations of threads (and POSIX threads) that areapplication-level, and do not take advantage of the kernel-threading.These thread packages keep the threading in a single process, hence do not take advantage of SMP. However, they are good for many applications and tend to actually run faster than kernel-threadson single processor systems.

Multi-threading has never been really popular in the UN*X worldthough. For some reason, applications requiring multiple processes orthreads, have mostly been written using fork(). Therefore, when usingthe thread approach, one runs into problems of incompatible (notthread-ready) libraries, compilers, and debuggers. GNU/Linux is noexception to this. Hopefully the next few sections will sched a littlelight over what is currently possible, and what is not.

The C Library

Older C libraries are not thread-safe. It is very important that youuse GNU LibC (glibc), also known as libc6. Earlierversions are, of course possible to use, but it will cause you much moretrouble than upgrading your system will, well probably :)

If you want to use GDB to debug your programs, see below.

Languages, Compilers and debuggers

There is a wealth of programming languages available for GNU/Linux,and many of them can be made to use threads one way or the other(some languages like Ada and Java even have threads as primitives inthe language).

This section will, however, currently only describe C and C++. If youhave experience in SMP Programming with other languages, pleaseenlighten us.

GNU C and C++, as well as the EGCS C and C++ compilers work with thethread support from the standard C library (glibc). There arehowever a few issues:

1. When compiling C or C++, use the -D_REENTRANT definein the compiler command line. This is necessary to make certainerror-handling functions work like the errno variable.
2. When using C++, If two threads throw exceptions concurrently, theprogram will segfault. The compiler does not generate thread-safeexception code.The workaround is to put apthread_mutex_lock(&global_exception_lock) in the constructor(s) ofevery class you throw(), and to put the correspondingpthread_mutex_unlock(...) in the destructor. It's ugly, but it works.This solution was given by Markus Ferch.

The GNU Debugger GDB as of version 4.18, should handle threadscorrectly. Most Linux distribution offer a patched, thread-aware gdb.

It is not necessary to patch glibc in any way just to make itwork with threads. If you do not need to debug the software (thiscould be true for all machines that are not development workstations),there is no need to patch glibc.

Note that core-dumps are of no use when using multiplethreads. Somehow, the core dump is attached to one of the currentlyrunning threads, and not to the program as a whole. Therefore,whenever you are debugging anything, run it from the debugger.

Hint: If you have a thread running haywire, like eating 100%CPU time, and you cannot seem to figure out why, here is a nice way tofind out what's going on: Run the program straight from the shell, noGDB. Make the thread go haywire. Use top to get the PID ofthe process. Run GDB like gdb program pid. This will makeGDB attach itself to the process with the PID you specified, and stopthe thead. Now you have a GDB session with the offending thread, andcan use bt and the like to see what is happening.

Other libraries

ElectricFence: This library is not thread safe. It shouldbe possible, however, to make it work in SMP environments by insertingmutex locks in the ElectricFence code.

Other points about SMP Programming

1. Where can I found more information about parallelprogramming?

   Look at the Linux Parallel Processing HOWTO

   Lots of useful information can be found at Parallel Processing using Linux

   Look also at the Linux Threads FAQ

2. Are there any threaded programs or libraries?

   Yes. For programs, you should look at:Multithreaded programs on linux (I love hyperlinks, didyou know that ? ;))

   As far as library are concerned, there are:

   OpenGL Mesa library

   Thanks to David Buccarelli,Andreas Schiffler and Emil Briggs, it exists in amultithreaded version (right now [1998-05-11], there is a workingversion that provides speedups of 5-30% on some OpenGL benchmarks). Themultithreaded stuff is now included in the regular Mesa distribution asan experimental option. For more information, look at theMesa library

   BLAS

   Pentium Pro Optimized BLAS and FFTs for Intel Linux

   Multithreaded BLAS routines are not available right now, but a dual proclibrary is planned for 1998-05-27, see Blas News for details.

   The GIMP

   Emil Briggs, the same guy who is involved inmultithreaded Mesa, is also working on multithreaded The GIMPplugins. Look at http://nemo.physics.ncsu.edu/~briggs/gimp/index.html for moreinfo.

3.4 MultiProcessor Specification Support (MPS)

(Randy Dunlap) Linux supports MPS (MP spec.) version 1.1 and 1.4.

Linux doesn't have full support for all of MPS version 1.4.

Experience has shown that Linux usually works best when the BIOS is configure for MP Spec. version 1.1 if that is an option in your system's BIOS. I don't see why the MP Spec. version should matter to Linux, but it would be an interesting exercise to find out the differences as presented by BIOS tables, to determine why Linux fails with MP Spec. version 1.4 in some cases, and to fix Linux so that this wouldn't matter.

This document summarizes the major changes in MP spec. version 1.4 and their support status in Linux.

Symmetric I/O Mode

The hardware must support a mode of operation in which the system can switch easily to Symmetric I/O mode from PIC or Virtual Wire mode. When the operating system is ready to swtich to MP operation, it writes a 01H to the IMCR register, if that register is implemented, and enables I/O APIC Redirection Table entries. The hardware must not require any other action on the part of software to make the transition to Symmetric I/O mode.

Linux recognizes and supports this MP configuration mode.

Floating Point Exception Interrupt

For PC/AT compatibility, the bootstrap processor must support DOS-compatible FPU execution and exception handling while running in either of the PC/AT-compatible modes. This means that floating point error signals from the BSP must be routed to the interrupt request 13 signal, IRQ13, when the system is in PIC or virtual wire mode. While floating point error signals from an application processor need not be routed to IRQ13, platform designers may choose to connect the two. For example, connecting the floating point error signal from application processors to IRQ13 can be useful in the case of a platform that supports dynamic choice of BSP during boot.

In symmetric mode, a compliant system supports only on-chip floating point units, with error signaling via interrupt vector 16. Operating systems must use interrupt vector 16 to manage floating point exceptions when the system is in symmetric mode.

Linux does not use the floating point interrupt at all except in genuine i386 processor systems which are not SMP-capable. [In these systems, if they wire the FPU exception line in the PC/AT-compatible way, a run-time check for #MF exception availability is performed. If the #MF exception is available, then Linux handles this interrupt if it happens. (Maciej W. Rozycki)

Multiple I/O APIC Configurations

Multiple I/O APICs are supported in Linux.

MP Configuration Table

This table was made optional in MPS version 1.4. If the table isn't present, one of the default configurations should be used. An extended section was also added to it for new table entry types.

Linux supports the optional MP Configuration Table and uses a default configuration if the MP Config. Table is not present.

Linux tolerates extended section table entries by skipping over them if they are found. Data in the extended table entries is not used.

MP Configuration Table Header Fields

New or changed fields for MP Spec. version 1.4:

• OEM Table Pointer: supported in Linux
• Extended Table Length: supported (tolerated, skipped) in Linux
• Extended Table Checksum: supported (tolerated, skipped) in Linux

Extended MP Configuration Table Entries

Entry types for System Address Space Mapping, Bus Hierarchy Descriptor, and Compatibility Bus Address Space Modifier are defined.

Linux skips over (does not use) these extended MP Configuration table entries. Apparently this isn't critical to any shipping systems.