Change extension of Multiple files at once in UNIX

Posted by admin on February 6th, 2008 filed in Linux

How do I rename files using wildcards ala the DOS “REN” command?

Question:

I have a bunch of files with the extension “txt” that I would like to rename with the extension “sgl”. In DOS you would use the following command:

REN *.txt *.sgl

But the Unix command I’ve been told to use for renaming files, the “mv” command, seems to get weird on wildcards and doesn’t work. That is…

mv *.txt *.sgl

…does not work the way the REN command does under DOS.

Answer:

Unix commands for the most part work very differently than DOS or VMS commands. The underlying operating systems themselves are totally different (Unix being much more capable and powerful than DOS by a long shot, and much more flexible than VMS). Many DOS and VMS users new to Unix fall into the same trap with wildcards.

The answer given here is a summary of about four chapters of a “how to use Unix” book. There are a lot of powerful features of Unix shells and commands that I will mention. When trying to learn a new operating system, you really need to expect to spend some time reading some books and the documentation for commands to pick things up. This is just as true for when you learned DOS, so I hope you aren’t upset about this statement. A good “how to” book to read is:

Student’s Guide to Unix, Harley Hahn (McGraw-Hill, 1990, ISBN 0-07-025511-3)

Briefly, the DOS shell is pretty much brain dead and limited to a puny 124 character command line (or something like that), so all DOS commands must do their own wildcard processing. The DOS shell (COMMAND.COM) doesn’t even know what a wildcard character is. This means that when the REN command sees the two strings “*.txt” and “*.sgl”, it is coded to know this means “find all files that end with the extension .TXT and rename them so the extension is now .SGL”. This is specific to the REN command (other commands may behave differently, or not accept wildcards at all, like MORE I believe). If you don’t like the “brain dead” label, try doing this wildcard in some DOS commands and see what you get: *a.txt (most commands don’t even see the “a” and just think you mean *.txt–two very different sets of files).

In Unix, the shell command line length is much longer (usually more than 4096 characters) and shells have the concept of “wildcards” that really mean “find any characters that match, expand them into the command line, then try parsing the command line again.” Once file names are expanded, the shell passes the entire command line to the program you wish to run. The man pages for commands tell what they accept.

If you check the man page for “mv” (which you should whenever you want to learn what arguments and options a given command accepts, and how it will deal with them) you will see:

Name
mv – move or rename files
Syntax
mv [-i] [-f] [-] file1 file2
mv [-i] [-f] [-] file… directory

Description The mv command moves (changes the name of) file1 to file2.

If file2 already exists, it is removed before file1 is moved. If file2 has a mode which forbids writing, mv prints the mode and reads the standard input to obtain a line. If the line begins with y, the move takes place. If it does not, mv exits. For further information, see chmod(2).

In the second form, one or more files (plain files or directories) are moved to the directory with their original file-names.

Given the (admittedly brief) description of how the shell interprets wildcards, what you have told the shell is to expand file names in the current working directory that match the names “*.txt” and “*.sgl”. You either will or won’t have any files named “*.sgl”, and the last one may or may not be a directory. If there are no such files, you have told “mv” to move all the files with names that end with “.txt” into a non-existent directory named “*.sgl”, which will fail because there isn’t a directory. If there was a file that ended with “.sgl”, then you are telling “mv” to move a list of files to the directory whose name is the last file named “.sgl”, which will also fail because it isn’t a directory. That’s the long story.

So there must be something more to it, right?

Yes. Since “mv” wants you to give two names–the name of the file to rename, and the new name you want it to have–you will need to execute the “mv” command once for each file. “What a pain in the …” you say. Well, Unix shells have much more powerful iterative commands than do DOS or VMS, so all you need to know is that if you use the C shell (which is most likely your login shell, unless you’ve changed it) you can use the “foreach” command. “How does ‘foreach’ work?” you say. The short answer is:

% foreach file (*.txt) > echo $file > end 1.txt 2.txt 3.txt

(Note: this example shows what you type at the command line, but I don’t show the carriage returns at the end of each line; just the prompts “%” and “>”. Type the “foreach …” and “echo …” stuff at the command line, and if you have files named “1.txt 2.txt 3.txt” you will see the same thing I show above.)

Is there another command that one would use to do a multiple-file renaming under Unix, or am I doing something wrong?

There is one more step. You need to remove the “.txt” extension and insert the “.sgl” extension in the “mv” command. This is done with another powerful feature of Unix shells, called “in-line command substitution”. The command to strip off file extensions is “basename” (see “man basename”). To do in-line command substitution, use the “back-tick” character; the single quote that leans to the left: `

Use “echo” to see how this works:

% echo *.txt
1.txt 2.txt 3.txt <== see the files named ".txt"
% echo `basename 1.txt .txt`
1 <== the ".txt" is stripped off
% ls *.sgl
No match <== there are no ".sgl" files
% mv 1.txt `basename 1.txt .txt`.sgl
% ls *.sgl
1.sgl <== now there is

The command sequence is then:

% foreach file (*.txt)
> mv $file `basename $file .txt`.sgl
> end

Don’t put any spaces between the last back-tick and the “.sgl”, since a space would mean you wanted to rename everything into the directory “.sgl”, which probably doesn’t exist.

This may look kind of complicated, but it is really pretty simple. Instead of just removing the “.txt”, I could have taken all files with names like “foo1234-bar.txt” and changed them to be “BAR.1234-foo.sgl” by matching on the letters, numbers, and dashes. Let’s see DOS do that!

Batch File Rename By File Extension in Unix

These are one-liners which batch rename files meeting a certain criteria under unix

These are very powerful command line tools. They have been used on FreeBSD, Linux, and MacOSX with success. But as with any batch file changing, you are advised to use them with caution. Backups are your friend!

# change .htm files to .html
for file in *.htm ; do mv $file `echo $file | sed ‘s/\(.*\.\)htm/\1html/’` ; done

# change .html files to .htm
for file in *.html ; do mv $file `echo $file | sed ‘s/\(.*\.\)html/\1htm/’` ; done

#change .html files to .shtml
for file in *.html ; do mv $file `echo $file | sed ‘s/\(.*\.\)html/\1shtml/’` ; done

#change .html files to php
for file in *.html ; do mv $file `echo $file | sed ‘s/\(.*\.\)html/\1php/’` ; done

I have Mac OS X, and I need to rename a directory with files like:

[itoph:Coda] toph% ls -l
total 59576
-rw-r–r– 1 toph staff 4140123 Apr 18 09:43 Bonzo’s Montruex
-rw-r–r– 1 toph staff 4914598 Apr 18 09:43 Darlene
-rw-r–r– 1 toph staff 4131343 Apr 18 09:43 I Can’t Quit You Babe
-rw-r–r– 1 toph staff 2260992 Apr 18 09:43 Ozone Baby
-rw-r–r– 1 toph staff 2917166 Apr 18 09:43 Poor Tom
-rw-r–r– 1 toph staff 4331548 Apr 18 09:43 Walter’s Walk
-rw-r–r– 1 toph staff 2521786 Apr 18 09:43 We’re Gonna Grove
-rw-r–r– 1 toph staff 5264433 Apr 18 09:43 Wearing and Tearing

Based on your page I’ve got this:

[itoph:Coda] toph% more test.sh
for i in * ; do
echo mv \’$i\’ \’$i.mp3\’
done

That doesn’t work because of the ‘s in the file names. If I leave out the \’s in the shell script the spaces in the file names cause a problem.

He then follows up with:

I figured it out. This is the shell script that works:

[itoph:Coda] toph% more t.sh
for i in * ; do
echo mv \”$i\” \”$i.mp3\” | sh
done

It looks a bit weird. Why wouldn’t I just do this:

[itoph:Coda] toph% more t.sh
for i in * ; do
mv \”$i\” \”$i.mp3\”
done

Well, the reason is because it doesn’t work. I get this error:

usage: mv [-fi] source target
mv [-fi] source … directory

I don’t understand how echoing and piping it to the shell is any different, but, it works.

Any ideas?

I found your page “http://artlung.com/lab/other/unix-batch-file-rename/” when I searched on “batch rename”; found it useful.

Just as I was getting the hang of it, and looking to make it a tidy AppleScript, I discovered that Apple already provided many of those same tools. At least after the developer tools have been installed; I’m not sure if that’s required.

Take a look at /Applications/AppleScript/Example Scripts/Finder Scripts. I found a script that will add extensions and prefixes, as well as one that will do wildcard renaming. Very useful!
http://lab.artlung.com/other/unix-batch-file-rename/ using BASH you can append .mp3 and get rid of those ghastly embedded spaces with:

for e in *; do mv “$e” “`echo $e | sed -e ‘s/\ /_/g’`.mp3″; done
before:
$ l
Bonzo’s Montruex
We’re Gonna Grove
after:
$ l
Bonzo’s_Montruex.mp3
We’re_Gonna_Grove.mp3

the classic basename(1) tool works very nicely without all of the sed ugliness:

# rename every *.htm file *.html
for f in *htm ; do mv $f `basename $f htm`html; done”