If you want something that will run on a commodity PC, QNX will do this, and supports a GUI called Neutrino. Some other embedded system platforms also support graphical user interfaces, such as Wind River’s Tilcon toolset for VXWorks.
IBM’s OS/2 has been sold to a third party and is still marketed as eComStation. It is largely sold as a legacy platform supporting existng OS/2 software, with relatively little new development activity. However, it is perfectly capable of functioning as a general purpose desktop O/S and I’ve seen OS/2 in UK HSBC branches within the past few years. The alarm clock ‘wait’ cursor icon is quite distinctive.
Some other operating systems such as Haiku (a BeOS clone) or ReactOS (A Windows clone) have been produced by open-source development communities. In theory, ReactOS has a substantial degree of binary compatibility with Windows. Most third party software support for Haiku is based on ports of open-source applications.
If you relax the ‘must run on a PC’ constraint, some other reasonably ‘modern’ OS platforms come out of the woodwork.
IBM’s I series is architecturally a fairly modern O/S, and was possibly the last major O/S done by people who had no exposure to Unix. It was originally designed as a replacement for IBM’s mainframe O/S platforms and then re-branded as a minicomputer platform. It is a capable platform in many ways but does not have a native GUI, although IBM have done a pretty credible job of supporting J2EE based web applications on it.
You can actually still buy machines that will run software written for the Amiga or Acorn Archimedes. I have seen it estimated that the latter architecture actually still has a user base of about 10,000 in the UK, and the Amiga still has a large worldwide fan base. However, I suspect that there are is not a lot of new build software being developed for either platform. More recently there is also a RiscOS port for the Raspberry Pi.
Vax, Alpha and Itanium based machines will run VMS, although the Vax and Alpha are out of production and HP does not sell purpose-built itanium based workstation systems anymore. However, used hardware can be readily purchased on Ebay and HP will still provide VMS installers for it. They even have a VMS hobbyist program that is still active and will let you buy an install CD for a nominal price of about $USD30. VMS is architecturally quite different to Unix and was not designed to be compatible, although it uses X as a GUI.
Several mobile platforms can support a variety of application software. Although the dedicated ones like iPhone, Symbian or Windows Mobile are unlikely to be practical as a general purpose computing platform for various reasons. Android or other linux based platforms could in theory be used for a broader range of tasks. Theoretically, Android could be self-hosting – i.e. one could (in theory) actually port and run an Android development environment on Android and use an Android-based O/S on a general purpose workstation. Again, this might not work all that well in practice.
Plan 9, although it is a bit researchy.
#3 makes the answer No. There are things like AmigaOS, ReactOS, etc but none are production/non-hobby.
VMS. (Has many things in common with WinNT, but is not in the “family”.)
You’d have to research this further, but look into something called 4960 OS. It’s DOS-like, not Unix-like; it’s not NT based; it’s in use in millions of IBM 496X-compatible POS terminals everywhere (Wal-Mart uses them), and from what I could tell it seems like it’s x86 hardware.
DOS. Yes, I know it’s old, but there are a VERY surprising number of these still out there. As little in common with (modern) Windows as it has with Linux. Runs on commodity x86 hardware.
Being VERY generous with the definition of “x86”, precursors to the x86 like the 8080, 8085, 8008 etc may still be in use in things like (cheap and programmable) calculators, dumb terminals, etc.
Maybe Windows CE? It’s a bit dubious: it runs on x86, not x64 (IIRC), but also on ARM and MIPS a.o.. The kernel is not related to the NT one, so it satisfies 2, but the API’s are definitely Windows inspired (often identical). 1 and 3 it does satisfy. It has been used in tablet PC’s (but not very often).
While there’s a lot of unix in MacOSX, it’s not X11 based, nor was unix compatability probably the primary goal when they chose to base large portions of it on *nix.
Other that that, most modern production/non hobby-research OSs are intended for embedded device or enterprise/mainframes. QNX might be something inbetween though.
Well MenuetOS (http://www.menuetos.net/) probably doesn’t meet requirement #3 but I have enjoyed tinkering with it.
MenuetOS is an operating system 100% written in assembler. I haven’t loaded it in a VM in years, but from what I recall it was reasonably stable, depending on what tools you were trying to use.
Embedded applications fit the description. While a lot, probably most, embedded devices probably use some Linux derivative, there are ones out there that are not. There are low-end SoC that are Intel x86, and to keep them cheap, memory runs at an absolute minimum.
I read somewhere (can’t site source, so take this as heresay) that my old Garmin Etrex runs an 80386 cpu. There are bound to be others.
Finding a specific example would be very difficult. Embedded devices don’t usually advertise their CPU or software. Most mobile embedded devices won’t qualify as they are ARM, and most of them run a linux kernel too. I am firmly convinced they are out there. Probably older devices; DVD players, maybe even some washing machines or dishwashers.