3Answers

tee -a does not recreate file

18.8k Views

3 Answers

I am running the following command on a Raspberry Pi 3 Debian latest release:

cat /dev/ttyUSB0 | tee -a /media/pi/KINGSTON/klima.out | grep -F $ | tee -a /media/pi/KINGSTON/log

The command works fine and does what it should; however, when I delete (manually or by CRON) the klima.out file, it is not re-created. The command keeps running, the log file continues to be appended, but the klima.out file doesn't come back. (also no buffering). I want to delete it once a week for not letting it grow over all boundaries. Any suggestions?

Allister Bedard Aug 21,2020

Bear in mind that deleting a file (like with rm) is essentially an edit operation on a directory. It deletes the entry pointing to the file from the directory. The file itself will only be deleted when nothing is using it; but tee is using it.

Nadyusha Chagfarov Aug 21,2020

This looks like a useless use of cat. Instead, tee < /dev/ttyUSB0 -a ... | ... would work the same, but tee would read directly from the char dev instead of from a pipe from cat.

Maurice Daurat Aug 21,2020

@Kusalananda, only in the rc/es or fish shells. Bourne-like shells and csh-like shells should be ok.

Brit Zijdemans Aug 21,2020

Kusalananda's answer is right; note that if your source was a growing log file instead of a device, you might want to "tail -f" so you don't re-cat what's already been processed.

Francisca Bardes

Francisca Bardes Added an answer on Aug 21,2020

I'm assuming that the pipeline of yours is running for a long time and that you are trying to remove the log file while it's running.

When you delete the file, the tee process still has it open for writing, which means that the disk space is not handed back to the system. That won't happen until all open file descriptors that references the file are closed.

It's perfectly OK to write to a file that's been deleted, as long as the file descriptor was allocated before the deletion.

You will have to restart the pipeline for the file to be recreated, and to allow the space taken by the old (now nameless) log file to be reclaimed.

To avoid having to restart the pipeline, you may choose to truncate the file, i.e. trim its size down to zero without removing it. This would allow tee to continue appending to the file without having to re-open it.

Truncating a file may be done like jlliagre showed in his answer, or by using truncate (a non-standard utility which is part of GNU coreutils):

truncate -s 0 /media/pi/KINGSTON/klima.out

See the manual for truncate for further info about that utility.

Sophia Carne Aug 21,2020

@PeterCordes I tested this and it does not seem to create a sparse file. A 100M file opened in append mode, then truncated as above. The next write to it will occur at the start of the now empty file, not after some sparse gap, as far as I can see.

Sophia Carne Aug 21,2020

@PeterCordes Ah, yes, if you use >, but not if you use >> or, as the OP was doing, tee -a. Writing in append mode always adds to the logical end of the file whereas not using append mode will write at the physical location after the last write.

Nadyusha Chagfarov Aug 21,2020

as long as the file descriptor was allocated before the deletion. Actually, you can get new file descriptors for already-deleted-but-still-open files through /proc/pid/fd. It's also perfectly fine to write through those, too. So really, as long as you have an open file descriptor, the number of directory entries for that inode is pretty much irrelevant.

Nadyusha Chagfarov Aug 21,2020

After truncating, the next write to the still-open fd will result in a sparse file, so yes this does recover the space but you have that many bytes of zeros before the new text, because it doesn't reset the file position of already-open fds. (And some filesystems don't support sparse files, e.g. FAT or HFS+, so the next write will actually write zeros). Anyway, I don't think there's any way around that (other than a ptrace / debugger hack that resets the file position of tee's fd as well as truncating).

Nadyusha Chagfarov Aug 21,2020

Anyway, you could also just punch a hole out to the current len with fallocate -p, but then you need the right length and all it gains you is saving one inode update (never needs to set the length to zero).

Pavla Hradilová

Pavla Hradilová Added an answer on Aug 21,2020

If you want to recover the file blocks, you need to blank the file, not unlink it:

This portable way should work with most shells :

: > /media/pi/KINGSTON/klima.out

Unlinking the file (i.e. rm) is removing the directory entry but doesn't affect the file contents (inode) as long as the file is kept open by readers or writers.

František Krása Aug 21,2020

Thanks, Yes, I just want to shorten the file. Sorry to bother, but I don't completely understand your suggested code. What does the : stand for?

Pavla Hradilová Aug 21,2020

: is the null command, i.e. it does nothing specific. The only feature used here is provided by the shell: the redirection is resetting the file to an empty content.

Sophia Carne Aug 21,2020

@jlliagre You're perfectly correct on both points.

Pavla Hradilová Aug 21,2020

@Björn Indeed, although not strictly equivalent, echo "" would write one byte to the file. Note that echo alone does the same as echo "".

Pavla Hradilová Aug 21,2020

@Björn Well, echo -n behavior is implementation defined. Depending on the shell, this command might replace the file content with the string -n instead of fully truncating it. A portable way to write "nothing" is printf "".

Yulichka Galiullin

Yulichka Galiullin Added an answer on Aug 21,2020

You don't understand how the system is handling files.

You delete the file entry, but the file still exists as long as the program keeps an handle on it. So tee is never notified the entry was deleted and it still writes to the file !

A unique file can have many entries thanks to the hard links (created by the ln command).

You could write your own version of tee which close and open the file on every line it writes to the file, but it would be very under-performing as it would generate so many system calls.

Here is a shell-function which will split its input to several files :

splitSizeInKio=100

splitInput(){
    local PS4='+splitInput+ '
    set -x
    local i=0
    local fname="$1"
    local ii

    while true
    do if [ $i -lt 10 ]
       then ii=0$i
       else ii=$i
       fi
       local outfile="$fname".$ii
       dd of="$outfile" bs=1024 count=$splitSizeInKio
       i=$((i+1))
    done
}

(You could use "head" instead of "dd" if you spilt on a number of lines instead of a size.)

With bash, you can use "process substitution" like this :

prog1 | tee >( splitInput somefilename ) | prog2

Vieira Elisete Aug 21,2020

Your splitInput might work ok using dd, since it "knows" exactly how many bytes to read from it's stdin, but using head is highly likely to result in corrupted data, due to buffering.

Nadyusha Chagfarov Aug 21,2020

Yeah, head will read more data from stdin than it writes, since it doesn't read 1 char at a time. And it doesn't try to "put back" any over-read beyond the number of lines it was looking for (if that's even possible on pipes).

tee -a does not recreate file