Formatting USB disk is slow under Linux

Hello,

I’m formatting a USB disk for backup using the backup application: rear

The formatting is really slow and I’m wondering if there is a quickformat option within Ubuntu that I might be able to use instead of rear to prepare the disk.

The disk is being formatted with the ext3 filesystem.

In Windows there is a quickformat option for NTFS partitions and I’m wondering if there is something similar for mkfs (not that I can see it though)

Thanks

VW

Are you looking for a GUI or CLI option?

@nixcraft

Preferably CLI, because I may be able to patch Rear, and then use the existing process.

But if there is a GUI fix and that’s the only option, then I’ll prepare the disks myself via the GUI.

The current process is painful and has to be changed if at all possible.

Thanks
VW

@VeeDub I just looked into manual page. There is no such thing as a ‘quick’ format for the ext4/ext3 file system. Such option is only available for ntfs and fat32 filesystem under Linux. The mkfs.ext4 is much faster tho. May I know your USB disk size? Is it bigger than 4TB?

I’ve just noticed, that according to GParted the disk has been formatted with ext4, despite the fact that rear reported that the partition was formatted with ext3.

Not sure if this matters, but reporting it, in case it is relevant.

The disk is 4TB, per marketing, which means approx 3.64TB in reality.

There are some optimization option in the /etc/mke2fs.conf. For example:

cat /etc/mke2fs.conf
[defaults]
	base_features = sparse_super,large_file,filetype,resize_inode,dir_index,ext_attr
	default_mntopts = acl,user_xattr
	enable_periodic_fsck = 0
	blocksize = 4096
	inode_size = 256
	inode_ratio = 16384

[fs_types]
	ext3 = {
		features = has_journal
	}
	ext4 = {
		features = has_journal,extent,huge_file,flex_bg,metadata_csum,64bit,dir_nlink,extra_isize
		inode_size = 256
	}
	small = {
		inode_size = 128
		inode_ratio = 4096
	}
	floppy = {
		inode_size = 128
		inode_ratio = 8192
	}
	big = {
		inode_ratio = 32768
	}
	huge = {
		inode_ratio = 65536
	}
	news = {
		inode_ratio = 4096
	}
	largefile = {
		inode_ratio = 1048576
		blocksize = -1
	}
	largefile4 = {
		inode_ratio = 4194304
		blocksize = -1
	}
	hurd = {
	     blocksize = 4096
	     inode_size = 128
	}

[options]
	fname_encoding = utf8

Your disk is 4TB, so if I were you I would try something as follows:

sudo mkfs.ext4 -T largefile4 /dev/diskNameHere 

Another option is to pass the -E lazy_itable_init=0. For example:

sudo mkfs.ext4 -E lazy_itable_init=0 -T largefile4 /dev/diskNameHere 

Here is what you get:

lazy_itable_init[= <0 to disable, 1 to enable>]
                          If enabled and the uninit_bg feature is enabled, the inode table will not be fully initialized by mke2fs.  This speeds up filesystem initialization noticeably, but it requires the kernel to finish initializing the filesystem in the background  when
                          the filesystem is first mounted.  If the option value is omitted, it defaults to 1 to enable lazy inode table zeroing.

So when you mount the disk first time it will initialize creating those inodes. That is all I can think. Try it out.

I’ve since realised that I didn’t give you the full picture.

The disk needs to be bootable as well, so there are two partitions.

I think I might try setting up the next disk via GParted and see how long that takes.

@nixcraft

I’ve had a look both at disks and gparted, and I’m having trouble working out how I can setup another bootable USB disk with the same properties as the one prepared by rear.

Here are the two partitions on the existing disk that were created by the rear setup


What I want to try and do is work out a way that I can setup a 2nd disk with the same properties where the data volume is setup quickly. I don’t know how exactly rear is formatting the data partition, but it is taking hours!

I want to see if I can create and format the data partition in some other way that doesn’t check all the sectors, write 0’s or whatever it is doing.

Thanks

You can copy the partition table layout of a whole disk using standard tools named sfdisk. For example, if device is /dev/sda and want to copy that to /dev/sdb, then. Make a copy:

# *******************************************
# !!! WARNING  !!!
# BE CAREFUL WITH DEVICE NAMEs
# *******************************************
sudo sfdisk -d /dev/sda > sda.part.copy.txt

There here is how to restore keeping the same disk and partition IDs to the /dev/sdb:

# *******************************************
# !!! WARNING  !!!
# BE CAREFUL WITH DEVICE NAMEs
# *******************************************
sudo sfdisk /dev/sdb < sda.part.copy.txt
## OR ##
sudo sfdisk -f /dev/sdb < sda.part.copy.txt

Of course, partition table replication is possible with one command as the root user when you wish to keep the same partitions and IDs:

sfdisk -d /dev/sda | sfdisk -f /dev/sdb

Here is how to restore generating a new disk and partition IDs. First use the grep and sed command to remove older IDS:

grep -v "^label-id"  sda.part.copy.txt | sed -e 's/, *uuid=[0-9A-F-]*//' > without-IDs-sda.part.copy.txt
diff sda.part.copy.txt  without-IDs-sda.part.copy.txt
sudo -f /dev/sdb <  without-IDs-sda.part.copy.txt

More options and info here How to backup and restore a partition table on Linux - nixCraft

@nixcraft

Thanks, this is what I want to do.

I want to restore generating a new disk and partition IDs

Unfortunately, I don’t follow your comments where you say

First use the grep and sed command

I just don’t understand how the grep command fits in with the earlier examples.

I think I understand the earlier examples, I just need clarity on how I include the grep / sed commands as part of the procedure.

Basically, I want to replicate the partition / file system structure but with new partition IDs

Thanks
VW

The grep and sed will remove uuid. This is useful when using two disks on the same machine. Typically uuid is used to mount and do other stuff. Hence, having different uuid helps. I hope this helps.

@nixcraft

I have executed the following commands:

sudo sfdisk -d /dev/sda > sda.part.copy.txt

grep -v “^label-id” sda.part.copy.txt | sed -e ‘s/, uuid=[0-9A-F-]//’ > without-IDs-sda.part.copy.txt
diff sda.part.copy.txt without-IDs-sda.part.copy.txt

sudo sfdisk -f /dev/sdb < without-IDs-sda.part.copy.txt

This appears to have created, but not formatted the partitions as per disks. If you compare the screenshots from disks below with the previous screenshots


When I try:

sudo e2label /dev/sdb2 REAR-002

The response is:
e2label: Bad magic number in super-block while trying to open /dev/sdb2

sudo mkfs.ext3 /dev/sdb2

took around 2.5 hours

That’s the issue, is there any alternative to provision a disk quicker?

Does it need to be ext3? Or in other words: Is there a reason why you don’t use ext4?

Also this is wrong:

mkfs.ext4 -E lazy_itable_init=0 /dev/sdX

You want it to be enabled so you need to set it to “1”:

mkfs.ext4 -E lazy_itable_init=1 /dev/sdX

To make a disk bootable you can simpy use GParted and set the “Boot”-Flag on your Disk.

@Flo

I would have to test that, for whatever reason when rear prepares the disks it uses ext3

So, if I prepare a disk myself, and make it ext4, then I’ll need to confirm that all other rear operations work.

But my initial (admittedly un-informed) feeling is that ext4 ought to be fine.

The key question from my perspective is that assuming an ext4 partition is compatible with rear; will an ext4 format complete significantly quicker?

It is crazy how long it takes to format a disk.

Well, that’s the question. We would have to find out.

rear format /dev/sdX does not seem to do more than just creating a partition and formatting it so it shouldn’t be a problem to just format the disk yourself.

After formatting you just need to set the following in your /etc/rear/local.conf:

OUTPUT=USB
BACKUP_URL=usb:///dev/sdX

I haven’t tried it out but this is how it should work.

@Flo

mkfs.ext4 -E lazy_itable_init=1 /dev/sdb

worked really well, it formatted the partition in around 1 minute. So that was perfect.

Rear has been able to backup to the data partition, but there were some errors during the backup related to the EFI partition (see below).

For some reason the boot partition was resized from 1GB to 419MB and this has created some issues. So I need to be able to setup the EFI partition as 1GB.

You could simply clone the partition table to the destination drive like mentioned above. Just be careful with the device names.