Squashfs Is it possible to create image and compress in one procedure?

At the moment I

  1. Create an image
  2. Then Squasfs the image to a new .sqshfs file
  3. Delete the original image after (2) completed.

Problem is that I backup freespace-nulled disk images and therefore the img is as big as the partition on the archive drive. The squashfs compression of the drive is way smaller.

I am wondering if I can create an image and compress at the same time (same as targz can do) in order not to first have to create the large image that takes up huge space during the process.

Assuming I want to back up /dev/sda5

Then using the pseudo file switch in mksquashfs I tried the following

imagetobackup="/dev/sda5"
pathtoincludingimagefile=sda5.img
newimagefilenameinsidesquashfs=sda5-new.img

mksquashfs $pathtoincludingimagefile -p '$newimagefilenameinsidesquashfs  dd if=$imagetobackup bs=10M'

.

But I cant get it to work from what I sort of understand from the squashfs manual. I am making a syntax error as I dont know where to include the target squashed filename etc.

Anyone knows how ?

Untested idea:

imagetobackup="/dev/sda5"
pathtoincludingimagefile=sda5.img
newimagefilenameinsidesquashfs=sda5-new.img

dd if=$imagetobackup bs=10M | mksquashfs - $pathtoincludingimagefile $newimagefilenameinsidesquashfs

If that doesn’t work, you could use a named pipe like this (also untested):

mkfifo np0
dd if=$imagetobackup ibs=10M status=progress conv=noerror > np0 &
mksquashfs -comp xz np0 $pathtoincludingimagefile 

I really like your second suggestion.
I understand that dd pipes the contents of /dev/sda5 into a named pipe and spawns while doing it.
Then I do
mksquashfs Pipecontents NameOfCompressedSquashArchive

The problem I get is with “Pipecontents” namely;
"-comp xz np0"
It gives me a syntax error.
If I can get past the syntax error this will work.

So far I cannot figure out why squashfs doesnt want to read from the named pipe.
Although I understand what you suggested very well, the hurdle to get squashfs to read from a named pipe is a bit over my head at the moment and the manual doesnt help me much so I am a bit out of luck getting it to work.
I really would like to continue with this suggestion, as I think it is brilliant, so if there are any other suggestions how to make squashfs read the named pipe successfully I would really like to know.

Excellent suggestion thanks. It is surely a brilliant construct I didnt think of.

I think the problem with this method is that the fifo is non persistent, so there is no way that mksquashfs can read the fifo from the first data sent through it after dd is spawned.
The fifo has to be redirected somehow directly into mksquashfs

How about rather placing mksquashfs in a while do loop that reads the fifo whenever there is data going through it.

Basically what I suggest is the other way round. Spawn off a while do loop containing makesquashfs reading a named pipe, and then start dd to pipe the device into the named pipe.

I will try and write a bash script for the above, but the named pipe shure is a winner idea. I cannot see why it wont work unless dd interrupts.

Good points.

According to squashfs-tools/README-4.5 at 23996602b0f7db407914aa9479fc41cb12b4c732 · plougher/squashfs-tools · GitHub, version 4.5 supports stdin by reading an uncompressed tarball.

^ Ubuntu doesn’t have it until 22.04 LTS, though, but Debian Bookworm (Testing) and Sid (Unstable) have it.

Darn, I have mksquashfs version 4.3-git installed and I have a kernel issue upgrading mksquashfs to 5.x. I am dependent on this kernel and since mksquashfs is a kernel patch it is not going to install due to libc6 dependency and the kernel requirements.

I will have to try it on another machine just to make sure it works in principle but i really needed that on this particular server.

Anyway I will try the concept on another server and report back.
It will be a while.

Thanks for the information.

Version 4.5 does not need a kernel patch anymore. Just read that in the source directory. So the above roadblock is resolved it seems.
I installed it from sources but although the package it is marked as 4.5, mksquashfs -version reports version 4.4 !

 ./mksquashfs -version
mksquashfs version 4.4-19fcc936 (2021-09-14)
copyright (C) 2021 Phillip Lougher <phillip@squashfs.org.uk>

I have contacted the developers. Seemingly wrong source associated with 4.5.

Update: Developer came back to me saying that it is a bug in the packaging. Could ignore it, but can get a corrected source soon. Very nice guy.

The correct sources are found here with the bug fixed for anyone who may need it at this date.
https://github.com/plougher/squashfs-tools/archive/refs/heads/master.zip

It would probably be the link to an old version in future; so dont rely on it for more than a few months from this date. At this date it is the newest.

You are not using the correct syntax. You are making two mistakes.

  1. You should be including a source on the Mksquashfs comand line, which normally will be a dummy directory
  2. Your pseudo file definition is missing the command type, mode, uid and gid.

So the above lines should be something like

imagetobackup="/dev/sda5"
pathtoincludingimagefile=sda5.img
newimagefilenameinsidesquashfs=sda5-new.img

mkdir /tmp/dummy

mksquashfs /tmp/dummy $pathtoincludingimagefile -p '$newimagefilenameinsidesquashfs  f 0444 0 0 dd if=$imagetobackup bs=10M'

rmdir /tmp/dummy

This will create a file named sda5-new.img owned/group-owned by root, with access permissons r–r–r–

The following is a worked example

root@phoenix:/tmp# mkdir dummy

root@phoenix:/tmp# mksquashfs dummy img.sqsh -p "loop0 f 0777 0 0 dd if=/dev/loop0" -info

Parallel mksquashfs: Using 4 processors
Creating 4.0 filesystem on img.sqsh, block size 131072.
160+0 records in
160+0 records out
81920 bytes (82 kB, 80 KiB) copied, 0.0136727 s, 6.0 MB/s
file /loop0, uncompressed size 81920 bytes 
directory / inode 0x20

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments,
	compressed xattrs, compressed ids
	duplicates are removed
Filesystem size 20.05 Kbytes (0.02 Mbytes)
	24.99% of uncompressed filesystem size (80.24 Kbytes)
Inode table size 44 bytes (0.04 Kbytes)
	66.67% of uncompressed inode table size (66 bytes)
Directory table size 25 bytes (0.02 Kbytes)
	92.59% of uncompressed directory table size (27 bytes)
Number of duplicate files found 0
Number of inodes 2
Number of files 1
Number of fragments 1
Number of symbolic links 0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 1
Number of ids (unique uids + gids) 1
Number of uids 1
	root (0)
Number of gids 1
	root (0)

Linux sysadmin blog - Linux/Unix Howtos and Tutorials - Linux bash shell scripting wiki