Posted By: Brent Nesbitt
I need to repeatedly remove the first line from a huge text file using a bash script.
Right now I am using
sed -i -e "1d" $FILE – but it takes around a minute to do the deletion.
Is there a more efficient way to accomplish this?
tail -n +2 "$FILE"
-n x: Just print the last
tail -n 5 would give you the last 5 lines of the input. The
+ sign kind of inverts the argument and make
tail print anything but the first
tail -n +1 would print the whole file,
tail -n +2 everything but the first line, etc.
The BSD version can be much slower than
sed, though. I wonder how they managed that;
tail should just read a file line by line while
sed does pretty complex operations involving interpreting a script, applying regular expressions and the like.
Note: You may be tempted to use
# THIS WILL GIVE YOU AN EMPTY FILE! tail -n +2 "$FILE" > "$FILE"
but this will give you an empty file. The reason is that the redirection (
>) happens before
tail is invoked by the shell:
- Shell truncates file
- Shell creates a new process for
- Shell redirects stdout of the
tailreads from the now empty
If you want to remove the first line inside the file, you should use:
tail -n +2 "$FILE" > "$FILE.tmp" && mv "$FILE.tmp" "$FILE"
&& will make sure that the file doesn’t get overwritten when there is a problem.