I want to understand what is going on
Rufus developer here.
What way too many people fail to understand, because Linux ISOs are applying this method, but this is essentially a MAJOR HACK CALLED ‘ISOHYBRID’, is that, in most cases, you cannot simply take an ISO image and copy it byte for byte to a USB drive, and expect that to boot.
That is because the ISO format and the underlying file systems it uses (
UDF) are designed for optical boot, which is a completely different beast from regular HDD or USB boot. For one thing optical media, and therefore (regular) ISO images, don’t have a partition table, which is (usually) essential for HDD or USB boot, and they also (usually) don’t have a Master Boot record, a.k.a. MBR, which is essential for BIOS boot.
This means that, if you do a 1:1 copy of a regular ISO, such as Windows one, onto a disk, and try to boot this is what’s going to happen:
- A BIOS system or UEFI system in Legacy/CSM mode will not see any MBR, and especially it will not see the
0xAAsequence in the very last 2 bytes of the MBR that indicates that a disk is BIOS-bootable. Therefore it won’t be able to boot that disk in BIOS mode.
- A UEFI system will (usually) not mount
ISO9660partitions from a disk or flash drive media, because, even as it has drivers for these files systems, the resulting disk you created will be missing an
GPTpartition table. When booting a regular disk, UEFI is designed to first look for a partition, and then look for a bootloader (e.g.
/efi/boot/bootx64.efi) on that partition. So if there is no
GPTpartition table on the media, which will be the case for a regular ISO, then it doesn’t matter if the ISO contains a bootloader file, because the UEFI firmware will not be able to mount the partition it resides on.
So, what utilities like Rufus do when creating a bootable disk media from a Windows ISO, which is a completely standard optical media image, is:
- Create a partition table, either
GPTaccording to what the user selected, and create at least one partition, that will typically use
NTFSas the file system (notice that it uses completely different file systems from what an ISO uses).
MBRis used, a bit of code in the
MBRthat locates the secondary boot loader, on the relevant
MBRpartition, which is in designed to start the execution of the Windows kernel, in disk mode, from that partition. Oh, and it also ensures that the
0xAAboot marker is added at the end of the
MBRso that BIOS sees the disk as bootable. Then it also copies the content of the ISO onto a
GPTis used, Rufus verifies that there actually exists a UEFI bootloader file, such as
/efi/boot/bootx64.efi(well, actually it does that before you it allows you to select GPT, coz there’s not much point in trying to create a GPT bootable drive if there is no UEFI bootloader) and then copies it, along with the rest of the ISO files, typically onto a
FAT32partition, since boot from a
FAT32partition is a mandatory requirement of UEFI (but that does not mean UEFI can’t boot from
exFATif you have the relevant UEFI drivers, which can come handy if you have a Windows ISO with a file that is larger than 4 GB, as FAT32 cannot accommodate such files).
Now, the above only works when the secondary bootloaders (i.e. the ones that comes from Windows and which Rufus doesn’t modify) are designed to support both optical and regular boot, which typically mean they need to handle both
NTFS file systems, as well as the other differences that present themselves when booting from disk vs from optical. But Microsoft did design its bootloader precisely for that, which is the smart thing to do, because, if your target system is UEFI, it means you (usually, as long as the 4 GB max filesize issue of
FAT32 doesn’t rear its ugly head) don’t need a utility to convert an ISO to a bootable USB, but you can just format that USB to FAT32 and copy the ISO files onto it (file copy, not byte copy), and you have a bootable media.
And now that we have gone through all of the above, I can get into a rant and explain why I believe that the Linux distro maintainers, who usually are smarter than that, are actually doing some disservice to their users, even as they are trying to help them:
Almost all recent Linux distros use a MAJOR HACK called “IsoHybrid”, where someone managed to figure out a way to make an
ISO9660 optical image masquerade as a regular disk image, with a partition table, an
MBR and everything… In other words, most Linux ISOs you find these days are abusing the ISO9660 file system to make it look like something it was never designed to look like: a dual disk and optical image.
Obviously, the goal is to create an ISO that can also be used with the
dd command, even as an ISO should never be able to work that way. And I agree that, in theory, this sounds awesome, because being able to use a single image for completely different uses should be great for users, but in practice, this leads to issues that are often overlooked:
- A lot of Linux distro maintainers don’t want to bother using a secondary file system that Windows can mount (e.g. they will use
extas the “secondary” file system on top of
ISO9660), which means that a lot of Windows people, who are creating a bootable drive to use Linux for the first time, are super confused as to why they can no longer access the content of their flash drive. It’s even worse if the “IsoHybrid” also includes an EFI System Partition (ESP) because then these users get the impression that their drive has completely shrunk in size. If you go on reddit or elsewhere, you will many posts from users who are utterly confused as to what happened to their USB media, which doesn’t make for a great Linux first impression…
- As lot of Linux distro maintainers focus so much on making ISOHybrid work that they completely disregard the option of creating a UEFI bootable media by simply copying the content onto a
FAT32formatted partition, which, really, should always be the preferred method of creating UEFI bootable drives (because it’s usually a lot less risky to format a partition and then copy files than it is to use the
ddcommand). Because of this, we’ve seen several issues that make for a subpar user-experience with Manjaro, Ubuntu… This is actually my main point of contention with “ISOHybrid”: It should not serve as an excuse to ditch established means of creating bootable media!
- GPT and “ISOHybrid” can be problematic on account that the secondary GPT table will be seen as corrupted when using
dd… which actually leads to a BSOD on Windows 7 (but that’s really a Windows bug rather than an ISOHybrid issue). Still, not the best experience for Windows folks creating bootable drives…
- And finally, because “ISOhybrids” are presented as if they were the most natural media in the world (which they certainly aren’t), people like yourself are led to believe that every ISO image can be applied using
dd, when it’s the exception rather than the rule. This is very unfortunate, because it creates TONS of user confusion, with some Linux users telling people who want to create Windows bootable media that they should just be able to use
ddwhen that most certainly will never work! Also, if you pick any Linux ISO from 10 years ago, I’m pretty confident that you’ll find that almost none of them can actually be used to create a bootable media using
ddbecause this “IsoHybrid` thing is actually a recent development.
As far as I know, Microsoft have no plans to switch to the “hack” that is ISOHybrid for their Windows ISOs, which means that you’re unlikely to ever be able to use
dd to create a bootable USB media from it, and therefore, if you want to create Windows bootable media from an ISO you either:
- (UEFI) Need to format a drive with a file system that Windows can boot from (
FAT32and more recently
exFAT) and extract the ISO files onto it. Now, if using
exFAT, you may have to do a little extra work as well…
- (BIOS/Legacy) Need to format a drive with a file system that Windows can boot from (
exFATwill not work because Microsoft never published BIOS bootloaders for it), and then create the relevant bootloader chain, from MBR boot code to volume boot records.
It’s actually not that complicated to achieve, but it does take a bit more work than a 1:1 copy from an ISO file.
Hope that answers your question.
dd isn’t the right tool to create a windows bootable USB. The easy way is to use
sudo apt-get install devscripts equivs gdebi-core cd WoeUSB ./setup-development-environment.bash mk-build-deps sudo gdebi woeusb-build-deps_3.3.1_all.deb dpkg-buildpackage -uc -b sudo gdebi ../woeusb_3.3.1_amd64.deb
Now, the package version is
3.3.1 , in case of package update ,the command
./setup-development-environment.bash will print the current version, you should replace it in the above commands.
You can use the GUI , run
woeusbgui from the terminal. Or you can use the CLI:
Unmount the USB device (important). Then run:
sudo woeusb -v --device /path/to/windows.iso /dev/sdc
Used the insights here as motivation to find a way to build a Windows Server 2019 bootable USB drive from Mac OS. The catch is that you need a GPT formatted disk as FAT32 and there are limitations around 4GB max filesize that you need to work around using
wimlib-imagex extension. I originally tried using
dd utility – but quickly realized that the disk format could not be used for WinOS boot.
Create Windows Server 2019 Bootable Disk on MacOS
brew install wimlib diskutil list diskutil eraseDisk MS-DOS "WIN2K19" GPT /dev/disk2 hdiutil mount en_windows_server_2019_updated_sep_2020_x64_dvd_2d6f25f2.iso rsync -vha --exclude=sources/install.wim "/Volumes/SSS_X64FREV_EN-US_DV9\ 1" /Volumes/WIN2K19 wimlib-imagex split "/Volumes/SSS_X64FREV_EN-US_DV9 1/sources/install.wim" /Volumes/WIN2K19/sources/install.swm 4000 diskutil eject /dev/disk2 diskutil eject /dev/disk3