The shell, of course.
The original metaphor that got us the word “kernel” for this in the first place is too often forgotten. The metaphor is that an operating system is a seed or a nut. The “kernel” of the seed is the core of the operating system, providing operating system services to applications programs, which is surrounded by the “shell” of the seed that is what users see from the outside.
Some people want to tie “kernel” (and, indeed, “shell”) down to be more specific than that. But in truth there’s a lot of variation across operating systems. Not the least these variations is what constitutes a “shell” (which can range from Solaris’
sh through Netware’s Console Command Interpreter to OS/2’s Workplace Shell and Windows NT’s Explorer), but there’s also a lot of variance from one operating system to another in what is, and isn’t, a part of a “kernel” (which may or may not include disk I/O, for example). It’s best to remember that these terms are metaphors.
There are other terminologies. In IBM mainframe terminology, the “kernel” in the seed/nut metaphor is called the control program. Other names include the supervisor, the monitor, the core, and the executive. Core is another fairly obvious biological metaphor. It is interesting that so too is another name. Harvey M. Deitel, in his Operating Systems, calls the control program the nucleus, and says
The nucleus normally represents a small portion of the code of what is commonly thought to be the entire operating system […]
The word “nucleus” comes from the Latin nucleus, and means the kernel of a nut (c.f. the Latin nux, which means “nut”). So even calling the control program “the nucleus” is in fact making this same operating-system-as-a-nut analogy. It’s an analogy that is used for more than just operating systems. If you are familiar with chemistry, for example, you’ll know that an atomic nucleus is surrounded by electrons in shells.
What the pseudonymous person
new123456 writes below exemplifies the mistakes resulting from forgetting the origins of these metaphors to the extent that one mistakes the part for the whole and mistakes the kernel for the whole operating system. Being a “userland” program does not make the shell any less the outside of the operating system, nor make it any less that which encloses the kernel. Moreover, far from being a DOS-only idea that doesn’t apply to Unix beyond version 7, this idea is both widespread and current in the Unix world.
(It is ironic that
new123456 mistakes it for a DOSism. The kernel/shell metaphor was fairly foreign to DOS in its early years, which favoured command processor and similar, from CP/M. Like CP/M before it, MS-DOS was divided into the basic disk operating system (BDOS), the basic input-output system (BIOS — not the system firmware, note), the command processor, and the housekeeping utilities. The code was actually parcelled up into files this way. This is another example, in fact, of how the kernel/shell metaphor doesn’t really cover or fully match up with the actual divisions in many operating systems.)
new123456 is forgetting what Deitel wrote about the nucleus being a part of the overall operating system. An operating system is both kernel and shell. That is, after all, the metaphor. Deitel isn’t alone, either. There are reams of Unix books explaining the Unix-as-a-nut metaphor, from the 1980s onwards. I recommend Urban’s and Tiemann’s Teach Yourself FreeBSD in 24 Hours (SAMS, ISBN 9780672324246), for the simple reason that on page 54 it has a diagram illustrating the operating-system-as-a-seed metaphor, and — surprisingly — is one of few books to do so.
The name “nucleus” is probably the older name; certainly predating Unix, which favours the kernel/shell terminology that today — probably in no small part thanks to Unix — is the more prevalent. Here’s what’s in Alan C. Shaw’s 1974 book The logical design of operating systems (Prentice-Hall, ISBN 9780135401125) for example from page 96:
That part of the system which permanently resides in main storage has historically been called the nucleus. The nucleus will usually consist of a minimal set of primitives and processes for the management of processes, resources, and input-output.
Shaw has a diagram, too, see Figure 4-4(b). ☺
Other parts of the operating system include the drivers that interact with the kernel, there are several layers of helper programs that deal with the user interface, the networking and other core parts of the OS that sit between kernel and the actual application.
To give an example, I’ve shown the structure of OS X, as you can see on top of the Kernel (light blue) lies several layers of “core” functionality, system tools, services and APIs (Application Programming Interface) before you even get to the GUI which is what you typically work with in the actual applications themselves.
Most operating systems have a similar structure, but there is wide scope for differences with regard to just how much is actually part of the kernel. See the Wikipedia article on Microkernels for a comparison.
EDIT – To answer your comment (my comment got a bit long)
You could boot just a linux kernel, but you would not be able to really achieve much, it’s all those helper applications that provide all the interface that you see and use.
Once the kernel is booted (and initialised the hardware) it will attempt to hand off to other parts of the GNU/Linux Operating System which provide the functionality for starting services (such as UDEV which populates
/dev with raw device interfaces for programs to use things like hard drives and serial ports and so on) and programs to set up the network (dhcpcd) and APIs such as X.org for applications to be able to show windows and icons though a Window Manager such as GNOME or KDE.
But yes, you could simply boot the kernel itself, it would just sit there with a “I’ve booted, now what?” prompt.