Unix ln command and its usage for simplifing navigation

ln is used to make a link between files. A link allows a filename in one directory to point to a file in another directory. Once a link has been made the linkname can be used, in most circumstances, as if it were a normal file.

Create a link to the specified TARGET with optional LINK_NAME. If LINK_NAME is omitted, a link with the same basename as the TARGET is created in the current directory. When using the second form with more than one TARGET, the last argument must be a directory; create links in DIRECTORY to each TARGET. Create hard links by default, symbolic links with --symbolic. When creating hard links, each TARGET must exist.

Mandatory arguments to long options are mandatory for short options too.

--backup[=CONTROL]

make a backup of each existing destination file

-b

like --backup but does not accept an argument

-d, -F, --directory

hard link directories (super-user only)

-f, --force

remove existing destination files

-n, --no-dereference

treat destination that is a symlink to a directory as if it were a normal file

-i, --interactive

prompt whether to remove destinations

-s, --symbolic

make symbolic links instead of hard links

-S, --suffix=SUFFIX

override the usual backup suffix

--target-directory=DIRECTORY

specify the DIRECTORY in which to create the links

-v, --verbose

print name of each file before linking

--help

display this help and exit

--version

output version information and exit

The backup suffix is

In Unix, what is a symbolic link, and how do I create one?

A symbolic link, also termed a soft link, is a special kind of file that points to another file, much like a shortcut in Windows or a Macintosh alias. Unlike a hard link, a symbolic link does not contain the data in the target file. It simply points to another entry somewhere in the file system. This difference gives symbolic links certain qualities that hard links do not have, such as the ability to link to directories, or to files on remote computers networked through NFS. Also, when you delete a target file, symbolic links to that file become unusable, whereas hard links preserve the contents of the file.

To create a symbolic link in Unix, at the Unix prompt, enter the following command:

  ln -s source_file myfile

Replace source_file with the name of the existing file for which you want to create the symbolic link (this file can be any existing file or directory across the file systems). Replace myfile with the name of the symbolic link. The ln command then creates the symbolic link. After you've made the symbolic link, you can perform an operation on or execute myfile, just as you could with the source_file. You can use normal file management commands (e.g., cp, rm) on the symbolic link.

Note: If you delete the source file or move it to a different location, your symbolic file will not function properly. You should either delete or move it. If you try to use it for other purposes (e.g., if you try to edit or execute it), the system will send a "file nonexistent" message.

To find out more about symbolic links, you can view the man pages for the ln command. To do this, at the Unix prompt, enter the following command:

  man ln

Also see:

As a link takes up much less disc space than the file which it is pointing to this is a useful way to save disc space and remain within your quota. It is also useful as any changes to the file which is pointed to will be used when the link is used.

This example shows copying three files from a directory into the current working directory.

    [2]%cp ~team/IntroProgs/MoreUltimateAnswer/more*

    [3]%ls -l more*

    -rw-rw-r--   1 mrblobby  mrblobby    632 Sep 21 18:12 moreultimateanswer.adb

    -rw-rw-r--   1 mrblobby  mrblobby   1218 Sep 21 18:19 moreultimatepack.adb

    -rw-rw-r--   1 mrblobby  mrblobby    784 Sep 21 18:16 moreultimatepack.ads

The three files take a total of 2634 bytes. The equivalent ln commands would be:

    [2]%ln -s ~team/IntroProgs/MoreUltimateAnswer/moreultimateanswer.adb .

    [3]%ln -s ~team/IntroProgs/MoreUltimateAnswer/moreultimatepack.adb .

    [4]%ln -s ~team/IntroProgs/MoreUltimateAnswer/moreultimatepack.adb .

    [5]%ls -l

    lrwxrwxrwx   1  mrblobby  mrblobby     35 Sep 22 08:50 moreultimateanswer.adb ->

                     /users/team/IntroProgs/MorUltimateAnswer/moreultimateanswer.adb

    lrwxrwxrwx   1  mrblobby  mrblobby     37 Sep 22 08:49 moreultimatepack.adb ->                       /users/team/IntroProgs/MorUltimateAnswer/moreultimatepack.adb

    lrwxrwxrwx   1   mrblobby  mrblobby    37 Sep 22 08:50 moreultimatepack.ads ->

                     /users/team/IntroProgs/MorUltimateAnswer/moreultimatepack.ads

The ln command now indicates in the first character on the line that this entry is a link. The filename is followed by an arrow ( -> ) and the full path name of the file which it is pointing to. The total size of the three links is 109 bytes. Although mrblooby is shown as owning these files, he only owns the links, and these permissions do not relate to the file itself.

The -s option allows a sybolic link to be made, rather than a phsical link. This allows a link to be made between two file systems. As a symbolic link can be made whenever a non-symbolic link can be made, but a non-symbolic link link cannot always be made when a symbolic link can be made, a symbolic link should always be specified.

The useage of the link command is.

    [2]%ln -s ActualFilename LinkFileName

Where -s indicates a symbolic link. ActualFilename is the name of the file which is to be linked to and LinkFileName is the name by which the file should be known.

ln will complain if:

you do not have read permission to the file which is to be linked.
file which is to be linked does not exist.
you do not have permission to create the link entry.

Old News ;-)

Thomas Rude - DD and Computer Forensics - He's Worth a Deuce!

Moving your data to a backup device

9.2.1.2. Using the dd command to dump data

The dd command can be used to put data on a disk, or get it off again, depending on the given input and output devices. An example:

gaby:~>dd if=images-without-dir.tar.gz of=/dev/fd0H1440

98+1 records in

98+1 records out

gaby~>dd if=/dev/fd0H1440 of=/var/tmp/images.tar.gz

2880+0 records in

2880+0 records out

gaby:~>ls /var/tmp/images*

/var/tmp/images.tar.gz

Note that the dumping is done on an unmounted device. Floppies created using this method will not be mountable in the file system, but it is of course the way to go for creating boot or rescue disks. For more information on the possibilities of dd, read the man pages.

This tool is part of the GNU fileutils package.

	Dumping disks
	The dd command can also be used to make a raw dump of an entire hard disk.

SUMMARY How to build a Solaris bootable CDROM

One would think this would be a simple task. Half the commerical packages proclaim that they can do it. But alas, for me, none of them worked.

The task was to create a system installer that would boot a sun, load up the appropriate system images and tools required for a server in our environment. Yes, this is a perfect job for jumpstart, but the systems would be distributed across the state and not connected via a fast enough pipe to use jumpstart effectivly. The other option was to build external disks with the proper images on them and boot from there. The cost of producing a few hundrad of these was prohibitive, thus the CDROM approach was taken.

Of the few responses I received from Sun Managers readers, all of them basicly said 'Its easy to clone the install disc!' or 'it just cant be done!'

Well. It is possible.

First, a breakdown of the process. When the command > boot cdrom < is givin to openboot the system actually looks at slice 2 to 5 for its bootblks depending on the machine type. A sun4c is slice 2 to an sun4u at slice 5. These bootblks redirect the system to slice 1 to load its kernel. The root partition is also stored in slice 1 and slice 0 is usr as well as packages. All slices save 0 are ufs, and 0 is hsfs.

This brings us to a small problem point. First we must boot from a UFS partition, as the bootblks all require that. Second, a CDROM has no label by default, thus its kinda hard to make partitions. And third, even if we do flush a UFS filesystem image off to the cdrom, the geometries will be all wrong, unless your staging disk just happens to have the same
geometries as the CDROM.

My solution is probally not the best. What would be required is an application that simply converts the UFS geometries from the staging disk uses to those used by the cdrom. I didnt really have time to create this so I did it as follows. Im sure I will get lots of flak for this
solution, but it does work.

1) Use dd to grab the first cylinder off the solaris boot cdrom. This contains a valid disk label and VTOC for the cdrom. Once this is created, our limitation is that we must work within the defines of this VTOC. You should be able to use prtvtoc on the cdrom to get a look at this VTOC, but this dosnt work if volmgr is running.

dd if=/dev/dsk/c0t6d0s0 of=cdrom.vtoc bs=512 count=1

2) Now use dd to grab the UFS slices from 1 to 5

for slice in 1 2 3 4 5
do
dd if=/dev/dsk/c0t6d0${slice} of=cdrom.s${slice}
done

3) Create a staging area and copy the parts of the usr filesystem (slice 0) off the cdrom into it. I started by copying cdrom:/export to it and then trimmed out the parts I didnt need like X and openwindows.

4) Add in all the things you need for your disc. For me, this ment a shell script that automated the build process, and images of all the data I wanted to move out. Make sure you donot go beyond te size of the slice 0 on the cdrom you started with. prtvtoc will should you this, if you cant get prtvtoc to work on the cdrom (sometimes it does, sometimes it dosnt) then use

> dd if=/dev/dsk/c0t6d0s0 of=/dev/null bs=512 <

Recordthe exact size of the partition in blocks as you will need it later.

5) Patch the slice 1 image (cdrom.s1) to start your custom application rather then the suninstall. This can be done by finding the break point you wish to use in the file cdrom:/sbin/sysconfig, selecting a unique set of chars in this file ( I used the string #***** S30sysid.net ) and then searching via a hexeditor or emacs in bin mode for that string.

Then find a comment line, change the first # and chars after to point to your script, then add a # line after. Example, I patched my image so the line

#***** S30sysid.net

became

exec build #ysid.net

I then put a script in my staging area into the /usr/bin dir. (the staging area will be made into slice 0 which is hsfs)

6) Using mkisofs or the tools that came with your cdrom burning package (HyCD worked well, as well as Gear) turn your staging area into a hsfs filesystem image, making sure that symbloic links are unmodified. HyCD required changing a default option that would have modified all the links.

7) Using dd, throw away the first block of this image.

dd if=image of=image.data bs=512 skip=1

8) Subtract the block count of slice 0 from the solaris cdrom from the image size above, add one to the answer. Say dd reported for the above step that your image was 500000 blocks, and your solaris cdrom has a size of 787840 for slice 0

787840 - ( 500000 + 1 ) = 287839

9) Feed this number into dd reading from /dev/zero to build a pad file.

dd if=/dev/zero of=image.pad bs=512 count=287839

10) Cat all of the image files together with the VTOC and the UFS slices.

cat cdrom.vtoc > image
cat image.data >> image
cat image.pad >> image
cat cdrom.s1 >> image
..
cat cdrom.s5 >> image

11) Burn this image to the cdrom drive using cdrecord, HyCD, Gear, etc

12) put it in a machine and test boot it and make sure it does what you need.

Thats about it. We have to go through the gyrations because UFS is geometry sensitive. We cannot take a image of a hard disk built UFS as all the cylinder groups will be off. The RIGHT way of doing this would be to build a tool that did the conversion for you and built an image up for burning. But that requires a bit more work, this gets the job done with a
minimal amount of strain. I am working on the above mentioned tool, but so far my progress has been to create lots of coasters. Sun has a tool called MakeDisc that does this job, or something similar, but I do not have a copy of it, so had to develop a method, while under a big gun, to do it in a very compressed amount of time. The only real limitation this
has is you can only store about 400 megs of information, of which around 70 or so are needed by usr in the hsfs partition. Plus you do not have to go through the pain of figuring out which parts of the system need to be moved to the memfs filesystem (cdrom is readonly, so dev, devices, etc need to be linked to /tmp)

If you wish to flame the procedure and tell me that I did it really stupid, please correct me! I would willing stop development of my tools to do same and do it the right way :) But my original question on how to do this went unanswered.

Have Fun
James

Reference

The Source of All Tape Knowledge

dd The name dd stands for "copy and convert". Don't see it? Well, "cc" was already taken for the C compiler, so the author chose the next letter in the alphabet. The syntax has sort of an evil, JCL-like quality to it. According to The Jargon File, the interface was a prank.

Using dd

Most people use dd incorrectly. This is because dd is a piece of junk that should be replaced.

For example, one common misusage of dd is to try and get 64k blocks written to the tape with this command:

tar -cf - args... | dd of=/dev/rmt8 bs=64k

This won't work because (as you will see below), the bs argument gives you only one buffer. The dd process will attempt to read 64k chunks from the pipe into this buffer, but will only receive a maximum of PIPE_BUF bytes (usually 4 or 8k). It will then write this buffer out to the tape as a single record (it will not pad this block to 64k, fortunately).

GNU dd

This data taken from GNU fileutils 3.12

When dd starts up, it parses all the arguments on the command line in order. Note that the bs= argument will override any previous ibs= or obs= arguments. If neither the obs nor the ibs argument is presented, and bs is given, and no character-translation conversions are performed, then only one buffer will be used (more on this later). In all other cases, two buffers (input and output) are used. If you don't specify any *bs args, ibs and obs default to 512.

Next, based on the translations that you have specified, dd builds a translation table. This table is a 256 entry array, specifying a character-by-character mapping that is the composite of all specified translations. The actual order of application of translations is not the same as what is on the command line. It is:

1. ebcidic_to_ascii

2. lower_to_upper

3. upper_to_lower

4. ascii_to_ebcidic

5. ascii_to_ibm

Note that not all conversions can be specified at once. You have your choice of only one conv in {ascii,ebcdic,ibm}, {lcase,ucase}, {block,unblock}, {unblock,sync}.

Finally, dd enters the copy stage. It allocates enough room for the input buffer, and if using a two-buffer scenario, allocates an output buffer as well. It performs any skips on the input, then performs any seeks on the output.

The main loop of the copy stage occurs now. It attempts to read input_blocksize characters into the input buffer. Errors here may be trapped, depending on command line options. If a full input block is not read (for example, when reading from a communication line, the end of a file, a pipe or special file, especially tapes), the partial block count is incremented. If the sync option is in effect, partial input blocks are NUL padded and treated as full input blocks.

At this point, if we are single-buffering, we write the block out. TODO: finish up here (I got bored)

AIX dd

AIX dd is broken. From their manpage:

3. Use the  backup,  tar, or  cpio command instead of the dd com-

  mand whenever possible to copy files to tape.  These commands are

  designed for use with  tape devices.  For more information on us-

  ing tape devices see the  rmt special file.

6. To ensure that only whole blocks are written to the output

  device (such as an 8mm tape in fixed-block mode), specify the ibs

  flag, the obs flag, and the  conv=sync flag.  The ibs flag must

  be a multiple of the obs flag.

This was hard-won knowledge for me. When dding a tar file directly out to tape, I ended up using:

dd if=foo.tar of=/dev/rmt1.5 ibs=1 obs=10240 conv=sync

Alternatively, you can use catblock, which is more efficient. Note that their comment about ibs being a multiple of obs is simply wrong, as my example demonstrates.

Solaris dd

Solaris dd is also not necessarily broken, but somewhat unintuitive. From the manpage:

When dd reads  from  a  pipe,  using  the  ibs=X  and  obs=Y

operands,  the  output  will  always be blocked in chunks of

size Y.  When bs=Z is used, the output blocks will be  what-

ever was available to be read from the pipe at the time.

In other words, don't expect "bs=Z" to be the same as "obs=Z ibs=Z". That's because, like GNU tar, it probably uses one buffer if you put "bs=Z", whereas "obs=Z ibs=Z" forces it to use two buffers. GNU dd may be subject to the same deficiency here. (TODO: check) I suppose it depends on how it treats a short read.

Solaris - dd (1)

How to use floppy disks with Solaris

BigAdmin Submitted Article- Backup and Recovery Utilities

BigAdmin - Solaris (x86 Platform Edition) DCA Boot Diskette

TAR DD or CPIO command

Solaris Systems Administrator’s Guide, Second EditionAdministering Devices Try Copying Data from the Disk

Burning a bootable CDRom for Solaris

Recommended Articles

Thomas Rude - DD and Computer Forensics

In the most basic sense, the DD command is used for copying in the UNIX environment. For simplicity, we will consider 'copy' to mean 'to duplicate exactly.' The DD command is used in the Forensics Arena to perform a physical backup of the evidence. DD can be thought of as tool - in the sense that using it is a means of building an evidence file. There are other tools which can be used when making a physical backup, such as EnCase and SafeBack. However, the intent of this paper is to give some insight on what DD is and how to use it.

What is special about the DD copy command is that it has special flags available to it that make it suitable for copying block-oriented devices, such as tapes. DD is capable of addressing these block devices sequentially. We will discuss this later. But, for now, it is good to note that this is why DD can be a powerful tool when acquiring and copying tapes for cases.

I do not want to describe each and every flag option available to DD ('man DD' can show you them). I do, however, want to detail some key flags that are very useful when copying evidence. Before we can get into these, it is imperative to understand the basic syntax of the DD command:

dd if=/*source* of=/*destination*
where:
if = infile, or evidence you are copying (a hard disk, tape, etc.)
source = source of evidence
of = outfile, or copy of evidence
destination = where you want to put the copy

For example, if our acquired evidence is /dev/hda, the following would produce an exact copy with the name of 'case10img1':
dd if=/dev/hda of=/dev/case10img1

Now that we see the basic use of DD we can look at the options which make it very suitable for copying in the UNIX environment.
As mentioned earlier, DD is very useful when copying and/or restoring block-oriented devices, such as tapes. (NOTE: DD is an excellent tool to use when copying hard disks as well. I am stressing the usage with regards to tapes because it has proved quite useful in reducing the amount of time required to copy tapes of large sizes.) There are a few options available when copying tapes (or any device). Of the options available, I have found some more useful than others. These are shown below:
ibs = input block size
obs = output block size
count = number of blocks to copy
skip = number of blocks to skip at start of input
seek = number of blocks to skip at start of output
conv = conversion

Let's say we have a 2GB hard disk seized as evidence. We will use DD to make a complete physical backup of the hard disk:
dd if=/dev/hda of=/dev/case5img1

Now let's say we have an unknown tape to examine. If we are unsure of the block size used on the tape, we could use the ibs/obs flags to find the correct size. Finding the correct size speeds up the copying process - sometimes dramatically!
dd if=/dev/st0 ibs=128 of=/dev/case10img1 obs=1 count=1
The above usage will attempt to take 1 block with size of 128 from 'st0' and create 'case10img1' output with a block size of 1. The 'count' flag is used so that only 1 block is read. We do this because we want to limit DD to just the 1 block. If we did not set a count size DD would continue on and a whole lot of time would be wasted! What this example attempts to show is that by setting the input block size to 128 we can effectively find what the real block size is (unless, of course, it is 128!). With 512 as the standard block size, assuming 128 is virtually a failproof way to find the real block size. The output of the above command would most likely be an 'error' message (which was our intent) with the real block size revealed (say 1024, for example).

Another example of DD usage is the following. Let's say we have an image which we need to chop up into smaller pieces. Perhaps our backup media is limited to 4 1GB discs and the evidence is 4GB in size. We could use DD with the flags below to create 4 images of the evidence, each 1GB in size.
dd if=/dev/st0 count=1000000 of=/dev/case10img1
dd if=/dev/st0 count=1000000 skip=1000000 of=/dev/case10img2
dd if=/dev/st0 count=1000000 skip=2000000 of=/dev/case10img3
dd if=/dev/st0 count=1000000 skip=3000000 of=/dev/case10img4
Now, we have taken the 4GB evidence tape and chopped it into 4 separate 1GB images. Each image is 1GB in size. Let's look at this example more closely. Notice that the first command takes 1GB (count=1000000) and copies it, naming the copy 'case10img1.' The second command skips the first 1GB (skip=1000000) and then copies the next 1GB (count=1000000), naming this image 'case10img2.' We can now see exactly what the 'count' and 'skip' flags do.

As you can see, DD is a very resourceful tool to use when performing physical backups of evidence. It is especially useful when working with large hard disks and/or tapes. The examples above were created to show you different ways you can get DD to work for you. As you become more familiar with it, you will find that you can do more than what I've shown above. You may even find out that DD is also quite useful when restoring evidence! I recommend that you create some evidence disks and tapes and play with DD. Read the man page on it and try the different flags. The learning curve is not steep, and the cost (free) can't be beat!

forensics 2003-08 Using dd.exe to make forensic images of NTFS

Hi everyone,

I have tried time and time again to make images of my NTFS drives via the
dd command in windows. I use the FIRE cd forensic shell on the windows box and:

dd.exe if=\\.\f: |nc.exe <forensic machine IP> <port>

On my linux box I run:

nc -l -p <port> |dd of=/home/user/ntfs.dd

That all works fine and it makes and transfers the file but then I try to add the file in autopsy and it tells me its not an NTFS image and consequently doesn't add it.

I tried conv=noerrors and I tried just dumping the file on the linux box without dd on the of= side. I tried different NTFS partitions of different sizes as well. My linux box has the NTFS support kernel mod and everything else about autopsy works fine. Just these NTFS images. I have no probs using dd with linux partitions at all. I'd like to find a solution to this because commerical ware like Encase is outrageously expensive and dd is free making it perfect for my situation.

Thanks,
Sakaba

Wonders of 'dd' and 'netcat' Cloning OS harddrives

Anytime we think about installing OS on more than one system 'cloning' comes to our attention. Because we are too lazy :-). Well that is one of the important characteristics of Systems Administrator so that he/she is forced to automate. In this document we will try to exploit the power of low level data transfer command popularly known as 'dd' and netcat. These programs are available for all major UNIX, Linux and Windows platforms. These commands are fairly popular among Forensics Analysis professionals.

Problem Description:

You got more than 1 machine with almost same hardware. i.e. same hard drive, SCSI card, graphics etc. You setup one single machine from top to bottom and now it is time to replicate OS setup on other machines. Commercial Software such as Ghost does a good job in cloning Windows based machines and now many of these software support Linux ext2 file system also. dd although very crude but gives you enough flexibility to manipulate cloning as you wish. We have demonstrated cloning of hard drives in machines loaded with Linux, Win2K, Solaris, HPUX machines using dd . This document is not a single click solution so you may have to juggle through here first. Once you get a hold of this process then it is very powerful to create your own disk cloning schemes save lots of time and hassle.

Basic concept:

'dd' command can copy any data bit by bit from one location to another location. So a simple command
dd if=<src> of=<dst>
where, <src> and <dst> can be a file, file system partition or whole hard drive so anything which can be read/write in binary form, dd can handle it. dd however is not a network program. In order to support dd with networking feature we use another nice command 'netcat'. netcat can be used to connect any TCP/UDP servers and a very good tool for diagnostics also. A typical netcat can run both into client server mode. such as:

server% nc -l -p 30000 ==> (Listen for port 30000 on <server> )

client% nc <server> 30000 ==> (Connect to <server> at port 30000, ready to communicate)

This document will explain cloning under Linux, but concept is very similar for all other operating systems also for which 'dd' and 'netcat' binaries are available.

Operating System Cloning (Using STANDALONE machine):

Let us assume we have two drives (sda) and (sdb) attached to the system ( Example: Linux box, but can be any other OS). (sda) is drive with Master OS (let's call it Master OS drive) and (sdb) is a drive (slave drive) where we have to clone data from (sda).

· IMPORTANT (IF you are cloning RAID/root devices): If slave drive has already RAID and/or root partition setup before. (especially / (root) partition), make sure you run fdisk (Use some 3rd Linux box if required) on slave hard drives , remove any partitions on slave hard drives and make it a plain new disk. If you don't do this and connect both drives with Master Linux box, it is very likely that Master linux box may come up with slave drive as (sda) i.e. primary device rather than expected (sdb) because how they appear in SCSI scan list. While following steps below you may destroy contents of actual Master Linux drive. So you have been warned already :-).

· CAUTION (IF you are using SCSI drives) : In case slave drive is connected to external SCSI controllers and if external SCSI controllers appear first in SCSI scan list during boot and BIOS scan, then external drives will get first device name such as /dev/sda , /dev/sdb and your master Linux box may not boot from desired hard drive. In order to avoid this, DISABLE BIOS scan for external SCSI controller. This will cause not to scan SCSI drives connected with external SCSI controller, so Linux will get internal harddrive as /dev/sda and boot from desired disk. Drives on external controller will be available when Linux Kernel try to scan all SCSI bus while booting. For a typical Adaptec SCSI controller:

o During Boot time, Go to SCSI controller BIOS screen (Ctrl-A)

o Select Controller Configuration -> Configure/View Host Adapter Settings -> Advance Configuration options -> Host Adapter Bios = (Disabled: Not Scan) .

o Save options and reboot box, During next boot, you will see SCSI BIOS Not Installed for this card.

DO NOT DISABLE BIOS SCAN FOR INTERNAL SCSI BUS. THIS IS NECESSARY TO FIND PRIMARY BOOT DISK.

Basically in any case IDE ot SCSI make sure you are booting from correct harddrive and that should appear as /dev/sda (Master drive) before you start any cloning process.

Let's assume we have to clone a harddrive (sda). Which has a partition table setup below. It has 1 NTFS partition loaded with WinNT/Win2K and rest Linux partition. (swap, Linux and Raid partition). Assuming second (slave) harddrive (sdb) is also attached to the same system.

   Device Boot Start End Blocks   Id System
/dev/sda1 1   9 72261   83 HPFS/NTFS
/dev/sda2   10 75 530145   82 Linux swap
/dev/sda3   76 467   3148740   fd Linux raid autodetect
/dev/sda4 468   2200 13920322+ 83 Linux

A simple way to clone this drive (/dev/sda) to another drive attached to this system (/dev/sdb) is to use dd command.

dd if=/dev/sda of=/dev/sdb

This command will copy each bit from sda (Master drive) to sdb (Slave drive) including MBR (Master Boot record). Thus after cloning new drive (sdb) is ready for deployment. This will also copy any information like File System IDs etc.

Since these days drive size is getting bigger and may run upto 100+ GB, this whole dd process may take long time and obviously there is no point in cloning Linux swap area or empty partitions which doesn't contain any useful data yet. Hence in this situation it is best to clone only relevant partitions. For this you need to partition second drive beforehand.

Note: Both drives are partitioned exactly same. If you have different brand harddrives, make sure each partition on second drive must be equal to or greater than first drive partitions. Also make sure File system ID should match for second drive also.

Device Boot Start End Blocks   Id System
/dev/sdb1 1   9 72261   83 HPFS/NTFS
/dev/sdb2   10 75 530145   82 Linux swap
/dev/sdb3   76 467   3148740   fd Linux raid autodetect
/dev/sdb4 468   2200 13920322+ 83 Linux

Now cloning process partition by partition will look like:

· First step is to copy MBR (Master Boot Record) to second drive. MBR is read right after BIOS in PCs bootstrap process. In case of Linux this will store LiLo ( Linux Loader) setup to find out Linux or Windows kernel. MBR is located with in first 446 bytes in harddrive (or partition) selected to store MBR during Linux install time or Windows install time. In our Linux installation we have selected MBR to be stored on first harddrive. Steps below will make second drive as a bootable drive.

Note: here we are using whole drive sda and sdb as input and output arguments of dd. (This process of making Solaris, HPUX drives bootable may be different but they allow you to setup boot record also just like PC's MBR)

dd if=/dev/sda of=/dev/sdb bs=446 count=1

· There is no reason to clone swap partition. Swap is raw area.

· Now you can clone other relevant partitions. Let's say /dev/sda3 will contain Linux OS and mounted as / (root) having ext2 or some other file system and /dev/sda4 is mounted as /home but doesn't contain any data. (May or may not contain any file system)

dd if=/dev/sda1 of=/dev/sdb1 ==> Clone NTFS partition
dd if=/dev/sda3 of=/dev/sdb3 ==> Clone RAID-1 partition having ext2 FS or some other.

· So in this case we can save time just by cloning desired partitions.

Operating System Cloning over network:

One major bottleneck in above process is we have to physically open boxes, connect harddrive to Master box and the run clone process. This is easier in case of desktops where you have a liberty to connect external drives (IDE, SCSI bus). But Laptop can hardly house one IDE drive in general and there are no easy way to open and connect second drive for cloning. Thus above process will be highly useful if cloning process can be used over network. There are several possible combination presented here. Idea here is we have Master Linux box up and running over network and we boot slave box having harddrive which is to be cloned but we use some alternate media such as boot CD and boot slave linux using root file system on CD itself *NOT* on harddrive so that we are free to write on slave hardrive.

Master Box-----------network-----------Slave box
[] [NOT * using slave drive]

One of Following 3 methods can be used to boot slave box using alternative media.

Method [1] Making your own root filesystem on ext2 CDROM. (Not Scalable )

One can make a small Linux distribution (less than 650MB) which can fit into CDROM. Burn this CDROM with ext2 filesystem (not ISO9660) and then use Linux boot floppy to boot from and use CDROM ext2 file system as / (root) file system (read only) (instead of root file system on Harddrive). This process although is doable but has issues like you need to have all possible drivers for network, SCSI etc. Making your custom ext2 read only file system on CD and booting from it would be quite a trial and error issue. If you are interested in making such Cds or bootable CDs see reference section for links. I once did that to clone HP Omnibook 6000 laptops loaded with Linux+Win2K OS together and it worked pretty okay but this is not a scalable solution though.

Method [2] Using popular Linux distribution and floppy combination.

On a similar line Linux distribution such as RedHat/SuSe boot CDROM at OS install time will allow you to boot into some kind of rescue system. In case of RedHat boot from RedHat OS CD and at initial OS install prompt type 'linux rescue ' at the boot time and this will let you use CDROM as root file system and provide you a shell prompt. Linux distribution uses this facility to repair problematic Linux install but we will use this for getting just shell prompt. Great thing about this is most Linux distribution comes up with lots of popular SCSI, network drivers so you don't have to worry about cooking your custom bootable CD.

Many common utilities including 'dd' command usually available in rescue mode. However you need netcat (static binary not dynamically linked) command. You can download netcat distribution and recompile it as a static binary (use -static flag). When I compiled it is small enough to fit into one floppy. So you can copy this into floppy. (I formatted floppy in ext2 format and then mounted in Linux system, copied netcat binary there.)

mkfs /dev/fd0
mount /dev/fd0 /mnt/floppy
cp nc /mnt/floppy
umount /mnt/floppy

So with 'linux rescue' mode and netcat binary on floppy you can use dd and netcat to clone your system over network. As we will see below.

Method [3] Modifying popular Linux distribution CDs and recreating your personal bootable ISO image:

If for some reason netcat won't fit in 1 floppy or you need more utility/binaries. Then you can change Linux distribution (SuSe/RedHat CD). This is a little hack but works.

NOTE: ISO images are read-only file systems. Even if you have an iso image (Say by using dd command )
dd if=/dev/cdrom of=redhat-boot-cd.iso
and if you try to mount this iso file using loopback device with option read/write (-o rw) (you need to have loopback device support (CONFIG_BLK_DEV_LOOP=y) compiled in kernel to do that)
mount -o loop -o rw ./redhat-boot-cd.iso /mnt/cdrom
This won't allow you to write/modify ISO filesystem.

I haven't found any good solutions to edit iso image directly , One such tool is winISO (http://www.winiso.com ) this is a shareware package so you have to pay for it. But you can use this to add more files in your ISO image and burn new image back to new CD. If you know any better solution let me know also :-)

Following steps are useful for adding additional files in RedHat bootable ISO image and burning a new CDs with additional files as of your choice.

· First mount RedHat CD (say RedHat 7.1). mount /dev/cdrom /mnt/cdrom

· Create a directory where you store stuff what goes on new CD. mkdir /home/cdburn

· cd /mnt/cdrom

· (tar cbf 20 - *) | (cd /home/cdburn | tar xvbf 20 -)

· This tar command will copy whole CD (~650MB to your Harddrive). In most cases you do not need all CD. Important stuff you need from directories, dosutils/, images/, RedHat/base. But if you do not want to play much simply copy the whole CD as above tar command and then you can delete RedHat/RPMS directory. These are simply RPM packages and since our intention is not to install OS from CD so we don't need that.

· Create subdirectory directory mkdir /home/cdburn/mystuff/ and add all your stuff such as static version of netcat binary etc. there.

· Now delete all TRANSLATION TABLES (TRANS.TBLS) files, otherwise mkisofs command will complain. find ./ -name "TRANS.TBL" -exec /bin/rm {} \;

· Make bootable iso image out of above distribution (/home/cdburn). Use mkisofs (part of mkisofs-1.9-6 package). The command below will create a bootable ISO image using initial boot image specified by -b, the -c option is for the boot catalog file. The -r option will make appropriate file ownership and modes. This iso image redhat-bootcd.iso is very similar to what is provided by RedHat except it will have our stuff also and we may have deleted any unnecessary contents such as RedHat/RPMS directory.

mkisofs -r -b images/boot.img -c boot.catalog -o /tmp/redhat-bootcd.iso ./

· Finally burn this iso image redhat-bootcd.iso using your cd-burner.

Now the Real drill:

Whatever method you choose to boot slave machine ( RedHat bootable CD + floopy or custom bootable RedHat CD), ultimate aim is to obtain shell, dd and netcat binary after 'linux rescue'. After you get shell you can access files stored on boot CD by changing directory to /mnt/sources/mystuff .
Hopefully your ethernet card has been detected by now. (as most Linux distributions allow OS install over network) if not then you have to load drivers for your ethernet card. Linux distribution documentation usually tells that how and sometimes they provide extra drivers floppy. In case of RedHat these floppy images are generally stored under directory images/ and you can copy these images to your floppy using commands like
dd if=<floppy-image> of=/dev/fd0

On Slave machine:
Run netcat command first on slave linux box (that to be cloned and booted using Linux boot CDROM as 'linux rescue' (See also Shell script case [1] in automation section below). Once ethernet card has been detected. (Use ifconfig -a command to check) assign IP address to this interface now on slave machine. Define loopback interface also. (You may choose different IP address for eth0). Also you may need to define /etc/hosts file before you can assign IP address. Use following commands to create your new /etc/hosts. (These are actually created in ram file system RAMFS).

rm /etc/hosts
echo "127.0.0.1 localhost" > /etc/hosts
echo "192.168.0.254 fakehost" >> /etc/hosts

ifconfig lo 127.0.0.1 up
ipconfig eth0 192.168.0.254 up

Assuming Master Linux box (from where you want to clone) is up and running with IP 192.168.0.1.

slave% nc -l -p 9000 | dd of=/dev/sda (Replace /dev/sda with actual drive on your slave machine)

This will listen at port 9000 and whatever it gets at port 9000 will hand over to dd command and dd will simply write that to on slave harddrive (sda) bit by bit. Here I am assuming dd and netcat (nc) are available either through floppy (/mnt/floppy/nc or through /mnt/sources/mystuff/nc). In case of floppy you need to mount floppy first using command:
mount /dev/fd0 /mnt/floppy

On Master machine:
Now Login on master linux box and run following command. (It is advisable that Master Linux box should be in calm state , i.e no major jobs running on the machine). This command below will read master disk bit by bit and throw this bit stream to netcat command which is connected to netcat command at port 9000 on <slave> box.

master% dd if=/dev/sda | nc 192.168.0.254 9000

That's it. You may have to wait for long time depending upon network speed and size of your harddrive. Typically 36GB drive may take 50 minutes over 100Mbps link. Again rather than cloning complete drive we can clone only relevant partitions and MBR only. That will make cloning much faster like we saw in above section.

Automating process and Case studies:

One of the primary reason for using dd and netcat way of cloning OS instead of using commercial software such as Ghost is we have a liberty to automate process as we like. Following scripts may help in automating cloning process.

Case [1]: Script for Slave machine (netcat and dd cloning) on the fly.

· Make sure you have netcat command available either /mnt/floppy or /mnt/sources/mystuff area.

=================================================
cloneme.sh :: Shell script for slave machine.
=================================================

#!/bin/sh
############### Edit variables below ######################
FLOPPY_PATH=/mnt/floppy
MYSTUFF_PATH=/mnt/sources/mystuff

# Uncomment only One of the options below.
#### OPTION ==> 1 if using floppy ################
#NC=$FLOPPY_PATH/nc
#### OPTION ==> 2 if using mystuff/ on CD #########
NC=$MYSTUFF_PATH/nc

LPORT=9000
DEST=/dev/sda
SRC=$DEST
############# No need to edit after this in general ###########

if [ $# -eq 1 ]
then
IPADDR=$1
echo "###############################################################"
echo " If there are no errors here. You need to run following"
echo " command on Master Box."
echo ""
echo "dd if=$SRC | nc $IPADDR $LPORT"
echo "###############################################################"

echo ""
echo "##>> Preparing /etc/hosts ##"
rm /etc/hosts
echo "127.0.0.1 localhost" > /etc/hosts
echo "$IPADDR fakehost" >> /etc/hosts

echo "#===================================================================="
echo "NOTE:: If you need to create routes"
echo " #route add -net <DEST_NET> netmask 255.255.255.0 gw $IPADDR metric 0"
echo "#===================================================================="

echo "##>> Preparing interfaces lo and eth0 ##"
ifconfig lo 127.0.0.1 up
ifconfig eth0 $IPADDR up

echo ""
echo ">>> Now start listening(at $LPORT) for traffic from Master :-)"
echo "$NC -l -p $LPORT | dd of=$DEST"
$NC -l -p $LPORT | dd of=$DEST

echo ""
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
echo " Cloning Process completed..... :-) Reboot Now"
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"

else
echo "Usage:: cloneme <IP_ADDR_OF_THIS_MACHINE>"
fi

Case [2] Saving Disk Images (Export Image for later use):

Although you can clone running machine over the network anytime. But it is sometimes desirable to store base installation as a reference image and you may want to clone from this pristine image later. With the help of dd you can image disks also. But let's discuss some issues first.
Most 32 bit operating system (Linux for IA32 , Windows etc.) will have physical limitation on max file size. In general practical limit is 2GB as a max. file size. 64 bit OS (Solaris8, HPUX 11.0, Linux for IA64, etc.) will not have this limitation. So if you use dd to copy harddrive image you can maximum image 2GB harddrive. That is pretty useless these days. Fortunately dd can image in chunks and you can specify start and end blocks, skip blocks etc. So idea here is to image your big harddrives in chunks of approx. 2GB files over network. Although I noticed RedHat 7.1 with Linux 2.4.x kernels will allow fie size even bigger than 4GB on ext2 FS.
Also if you want to store images in compressed format (to save space) it is desirable to have each image file size not too large.
Following perl script (export-image.pl) can be used to image local Linux harddrive /dev/hda to remote machine over NFS using dd. If you are not running NFS you can implement same thing using dd and netcat. For now that would be a manual process. If somebody knows a better way to run netcat and transfer multiple files automatically between two machines please let me know and I will cook up some automation script here.

This perl script is actually use dd command something as described below. This is imaging your big harddrive into chunks of 1950 MB files named (1, 2, 3, 4, .....) over NFS to remote machine.
($NFS is NFS destination directory on another server having plenty of space)

For 1st Image:
dd if=/dev/hda of=$NFS/1 bs=1024k count=1950 skip=0
For 2nd image: (Skipping the part of harddrive used for 1st image.)
dd if=/dev/hda of=$NFS/2 bs=1024k count=1950 skip=1950
For 3rd image: (Skipping the part of Harddrive used for 1st+2nd image)
dd if=/dev/hda of=$NFS/3 bs=1024k count=1950 skip=3900
and so on.

In case you want to use netcat you can simply pipe above dd commands manually to netcat and listen using netcat and dd on remote machine, just like we used netcat and dd to clone hardisks above. For example imaging harddrive on machineA and saving image on machineB.

For 1st image:
machineB% nc -l -p 9000 | dd of=1
machineA(master)% dd if=/dev/hda bs=1024k count=1950 skip=0 | nc machineB 9000
For 2nd image:
machineB% nc -l -p 9001 | dd of=2
machineA(master)% dd if=/dev/hda bs=1024k count=1950 skip=1950 | nc machineB 9001
For 3rdimage:
machineB% nc -l -p 9002 | dd of=2
machineA(master)% dd if=/dev/hda bs=1024k count=1950 skip=3900 | nc machineB 9002
and so on.

Once you have images (1, 2, 3, 4 ....) stored on network then you can boot your slave Linux box using bootable CD and pull these images to slave box as described in case [3].

========================================================
export-image.pl :: Perl script to image big harddrive using dd and NFS.
========================================================
#!/usr/bin/perl
#####################################################
#This script will run dd command (in serial) and dump
#1950 blocks (1.9GB) file for each.
#Run script as perl export-image.pl
#####################################################

################ Edit variables below #########################
#device is raw device name for harddrive to be cloned (imaged).
$device="/dev/hda";
#mount NFS file system with large space available which can hold images.
$nfs_path="/nfs/remote/home/tmp";
#Image name (read from user) (Make sure you have $nfs_path/$image directory)
#on remote machine.
$image="ob6000";
############################################################
$dd="/bin/dd";
#For compressing image
$bzip2="/usr/bin/bzip2";
$suffix=".bz2";
############## No need to edit after this #########################

$bs="1024k";
$block_count=1950;
$image_dir="$nfs_path/$image";
$compress=$bzip2;

$proceed=0;

if(!(-d $image_dir) )
{ die "\nOops!! Image Directory $image_dir must exist with chmod 777 permission\n"; }

system("clear");
print <<MSG1;
###########################################################
NOTE:: COMPRESSION TAKE TOO MUCH TIME(Many HOURS) OVER NFS.
So better compress manually latter on server itself.
###########################################################
\n\n Do you want to compress images using $compress [y/n] (Default n) = \t
MSG1

$compress_flag=<STDIN>;
if(($compress_flag eq "y") or ($compress_flag eq "Y"))
{ $compress_flag=1; }
else
{ $compress_flag=0; }

print "\n\n";
print "***************************************************\n";
print " Local Device = $device [SOURCE] \n";
print " Image Dir = $image_dir [TARGET] \n";
print "***************************************************\n\n\n";
print "Dude! I hope you understand what are you doing by pressing [y/Y] here :-) \n";
print " Press [y/Y] if you want to continue .. ";
$con=<STDIN>; chomp($con);

if(($con eq "y") or ($con eq "Y"))
{
$i=0;
$image_size=1; #Some fake value greater than zero.

print "\n\nDisk Imaging starts...\n";
system("date");
while($image_size > 0)
   {
$image_name="$image_dir/$i";
print "##############################################\n";
print "Creating Image $image_name\n";
print "##############################################\n";
$skip=$i*$block_count;
print "$dd if=$device of=$image_name bs=$bs count=$block_count skip=$skip \n";
system("$dd if=$device of=$image_name bs=$bs count=$block_count skip=$skip");
if($compress_flag)
{
   print "Compressing Image: $bzip2 $image_name => $image_name$suffix\n";
   system("$bzip2 $image_name");
   $image_name .= "$suffix";
}
++$i;
$image_size=(stat($image_name))[7];
system("date");
   }
}
else
{
print "Bye Bye ...\n";
}

Case [3] Importing Disk Images (1, 2, 3, 4 ...) created in Case [2] using netcat, dd and cat

This part is little tricky in the sense we want all images (1, 2, 3, 4, ...) to be imported on slave machine and use dd to write these images serially on slave drive. A very simple set of commands can be used as below.

On Slave machine: (booted through linux rescue). Run following netcat command to capture incoming data stream.

machineC(slave)% nc -l -p 9000 | dd of=/dev/hda

On machineB machine: (where images 1, 2, 3, 4 .... are stored). Run following cat and netcat command. Make sure you cat images in the same sequence as they were imported in case [2]. cat command will simply join these images and throw data stream to netcat which slave machine will pick up and copy bit by bit on slave harddrive.

machineB% cat 1 2 3 4 .... | nc machineC 9000

Case [4] Importing Disk images created in Case[2]:
Most likely 'linux rescue' system won't have NFS support. Which means when you boot slave box using such method you can not access resources over NFS. But if you cook your own CD and that has NFS support and perl the following perl script can be used to fetch images stored earlier from machineB using NFS. This script is actually doing:
($NFS is NFS source directory on another server machineB where you have images 1, 2, 3, 4, ... stored earlier)

For image 1:
dd if=$NFS/1 of=/dev/hda bs=1024k conv=notrunc seek=0
For image 2:
dd if=$NFS/2 of=/dev/hda bs=1024k conv=notrunc seek=1950
For image 3:
dd if=$NFS/3 of=/dev/hda bs=1024k conv=notrunc seek=3900

In any case if you are interested in using perl script below (if you have perl and NFS client support on slave linux box).

import-image.pl
#!/usr/bin/perl
#####################################################
#This script will run dd command (in serial) and dump
#and import image.
#####################################################

##############################################################################
#device is target raw device name for harddrive to be cloned.
$device="/dev/hda";
#mount NFS file system with large space available which can hold images.
$nfs_path="/mnt/images";
#Image name (read from user)
$image="ob6000";
###############################################################################
$dd="/bin/dd";
#$bzcat="/usr/bin/bzcat";
#$suffix=".bz2";

$bs="1024k";
$block_count=1950;
###############################################################################
$image_dir="$nfs_path/$image";

$proceed=0;

if(!(-d $image_dir) )
{ die "\nOops!! No Image Directory $image_dir\n"; }

system("clear");
print "***************************************************\n";
print " Local Device = $device [TARGET]\n";
print " Image Dir = $image_dir [SOURCE]\n";
print "***************************************************\n\n\n";
print "Dude! I hope you understand what are you doing by pressing [y/Y] here :-) \n";
print " Press [y/Y] if you want to continue .. ";
$con=<STDIN>; chomp($con);
print " Once Again!!! Press [y/Y] if you want to continue .. ";
$con=<STDIN>; chomp($con);

system("date");
if(($con eq "y") or ($con eq "Y"))
{
print "\n\nDisk Imaging import starts...\n";

$i=0;
$image_name="$image_dir/$i";
while(-f $image_name )
{
print "##############################################\n";
print "Importing Image $image_name\n";
print "##############################################\n";
$seek=$i*$block_count;
print "##############################################\n";
$seek=$i*$block_count;
print "$dd if=$image_name of=$device bs=$bs conv=notrunc seek=$seek \n";
#system("$bzcat $image_name | $dd of=$device bs=$bs conv=notrunc seek=$seek");
system("$dd if=$image_name of=$device bs=$bs conv=notrunc seek=$seek");
++$i;
$image_name="$image_dir/$i";
system("date");
}
}
else
{
print "Bye Bye ...\n";
}

Other Operating Systems Tips:

You can pretty much do same in other operating systems also. This section quickly list few tips that may be useful.

Windows:

· You can find GNU utilities ( http://unxutils.sourceforge.net/ ) for Win32 platforms which includes dd.exe command.

· dd.exe syntax is similar to as you use on Linux side. For physical partition you may have to use devicename something like \\.\PhysicalDrive0 etc. For example

dd if=\\.\PhysicalDrive0 of=<target>

· You can download netcat for windows ( http://www.l0pht.com/~weld/netcat/ )

· If you are cloning WinNT/2K system , you need to change SID for the new cloned system if machine is participating in Windows domain. You can use Ghostwalker program from Ghost distribution or can use newsid.exe from http://www.sysinternals.com

Solaris:

· Burning bootable CDROM for Solaris. ( http://www.lka.ch/projects/solcdburn/solcdburn.html )

· Sun Blue print : http://www.sun.com/software/solutions/blueprints/0301/BuildBoot.pdf

· To Make disk bootable (just like putting MBR in Linux) use command installboot(1M)

Others: (Make disk bootable)

· Irix : dvhtool

· HP-UX: mkboot

· Tru64: disklabel

Conclusion:

Few possible uses of netcat and dd shown in this document. Although methods presented here are very simple and easy to use but have few pros and cons also. This technique is very good for on the fly OS cloning. When we image the whole drive we need the equivalent harddrive space on other machine. This may not be very much practical. You can try compressing those images which will save lots of space. I noticed dd image can be compressed upto 30-80% depending upon real data on the drive using gzip/compress program. This cloning and imaging method may be very effective in forensic analysis where sometimes you need an exact snapshot of harddrive including swap space partitions. You can always break your images in small pieces (may be compress them) transfer over network to somewhere else and reproduce data. As mentioned above one of the great advantage here is to custom your own cloning scheme.

References:

1. GNU utilities for Win32. http://unxutils.sourceforge.net/

2. netcat for Windows. http://www.l0pht.com/~weld/netcat

3. First Attempt at Creating a Bootable Live Filesystem on a CDROM http://www.linuxgazette.com/issue54/nielsen.html

4. Good Site for Windows utilities such as newsid.exe: http://www.sysinternals.com

5. Modifying ISO image http://www.winiso.com

6. Solaris Bootable CD creation: http://www.lka.ch/projects/solcdburn/solcdburn.html

7. Sun Blueprint: http://www.sun.com/software/solutions/blueprints/0301/BuildBoot.pdf

8. Linux on Floppy: http://www.toms.net/rb/

9. Static binaries for Linux.

Take Command dd -- http://www.linuxjournal.com/article.php?sid=1320

The ' dd ' command is one of the original Unix utilities and should be in everyone's tool box. It can strip headers, extract parts of binary files and write into the middle of floppy disks; it is used by the Linux kernel Makefiles to make boot images. It can be used to copy and convert magnetic tape formats, convert between ASCII and EBCDIC, swap bytes, and force to upper and lowercase. For blocked I/O, the dd command has no competition in the standard tool set.

One could write a custom utility to do specific I/O or formatting but, as dd is already available almost everywhere, it makes sense to use it. Like most well-behaved commands, dd reads from its standard input and writes to its standard output, unless a command line specification has been given. This allows dd to be used in pipes, and remotely with the rsh remote shell command.

Unlike most commands, dd uses a keyword=value format for its parameters. This was reputedly modeled after IBM System/360 JCL, which had an elaborate DD 'Dataset Definition' specification for I/O devices. A complete listing of all keywords is available from GNU dd with

$ dd --help

Some people believe dd means ``Destroy Disk'' or ``Delete Data'' because if it is misused, a partition or output file can be trashed very quickly. Since dd is the tool used to write disk headers, boot records, and similar system data areas, misuse of dd has probably trashed many hard disks and file systems. In essence, dd copies and optionally converts data. It uses an input buffer, conversion buffer if conversion is specified, and an output buffer. Reads are issued to the input file or device for the size of the input buffer, optional conversions are applied, and writes are issued for the size of the output buffer. This allows I/O requests to be tailored to the requirements of a task. Output to standard error reports the number of full and short blocks read and written.

Example 1

A typical task for dd is copying a floppy disk. As the common geometry of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an optimized dd command to read a floppy is:

Example 1-a : Copying from a 3.5" floppy

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out

The 18b specifies 18 sectors of 512 bytes, the 2x multiplies the sector size by the number of heads, and the 80x is for the cylinders--a total of 1474560 bytes. This issues a single 1474560-byte read request to /dev/fd0 and a single 1474560 write request to /tmp/floppy.image, whereas a corresponding cp command

cp /dev/fd0 /tmp/floppy.image

issues 360 reads and writes of 4096 bytes. While this may seem insignificant on a 1.44MB file, when larger amounts of data are involved, reducing the number of system calls and improving performance can be significant.

This example also shows the factor capability in the GNU dd number specification. This has been around since before the Programmers Work Bench and, while not documented in the GNU dd man page, is present in the source and works just fine, thank you.

To finish copying a floppy, the original needs to be ejected, a new diskette inserted, and another dd command issued to write to the diskette:

Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out

Here is shown the stdin/stdout usage, in which respect dd is like most other utilities.

Example 2

The original need for dd came with the 1/2" tapes used to exchange data with other systems and boot and install Unix on the PDP/11. Those days are gone, but the 9-track format lives. To access the venerable 9-track, 1/2" tape, dd is superior. With modern SCSI tape devices, blocking and unblocking are no longer a necessity, as the hardware reads and writes 512-byte data blocks.

However, the 9-track 1/2" tape format allows for variable length blocking and can be impossible to read with the cp command. The dd command allows for the exact specification of input and output block sizes, and can even read variable length block sizes, by specifying an input buffer size larger than any of the blocks on the tape. Short blocks are read, and dd happily copies those to the output file without complaint, simply reporting on the number of complete and short blocks encountered.

Then there are the EBCDIC datasets transferred from such systems as MVS, which are almost always 80-character blank-padded Hollerith Card Images! No problem for dd, which will convert these to newline-terminated variable record length ASCII. Making the format is just as easy and dd again is the right tool for the job.

Example 2 : Converting EBCDIC 80-character fixed-length record to ASCII variable-length newline-terminated record
dd bs=10240 cbs=80 conv=ascii,unblock if=/dev/st0 of=ascii.out
40+0 records in
38+1 records out

The fixed record length is specified by the cbs=80 parameter, and the input and output block sizes are set with bs=10240. The EBCDIC-to-ASCII conversion and fixed-to-variable record length conversion are enabled with the conv=ascii,noblock parameter.

Notice the output record count is smaller than the input record count. This is due to the padding spaces eliminated from the output file and replaced with newline characters.

Example 3

Sometimes data arrives from sources in unusual formats. For example, every time I read a tape made on an SGI machine, the bytes are swapped. The dd command takes this in stride, swapping the bytes as required. The ability to use dd in a pipe with rsh means that the tape device on any *nix system is accessible, given the proper rlogin setup.

Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -

The dd runs on the SGI and swaps the bytes before writing to the tar command running on the local host.

Example 4

Murphy's Law was postulated long before digital computers, but it seems it was specifically targeted for them. When you need to read a floppy or tape, it is the only copy in the universe and you have a deadline past due, that is when you will have a bad spot on the magnetic media, and your data will be unreadable. To the rescue comes dd, which can read all the good data around the bad spot and continue after the error is encountered. Sometimes this is all that is needed to recover the important data.

Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image

Example 5

The Linux kernel Makefiles use dd to build the boot image. In the Alpha Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target issues the command:

Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1

This skips the first 512 bytes of the input bootimage file (skip=1) and writes starting at the second sector of the $(BOOTDEV) device (seek=1). A typical use of dd is to skip executable headers and begin writing in the middle of a device, skipping volume and partition data. As this can cause your disk to lose file system data, please test and use these applications with care.

Credits

The dd command has been around since the 1970s, ported to many systems, rewritten many times, and tested by time as a useful tool. The current Linux version is GNU dd GNU fileutils 3.12, written by Paul Rubin, David MacKenzie, and Stuart Kemp, Copyright © 1985, 1990, 1991 Free Software Foundation, Inc. GNU dd is found in the fileutils collection, with the current version at the URL ftp://prep.ai.mit.edu/pub/gnu/fileutils-3.12.tar.gz or a mirror near you.

Remote backup

It is often neccesary to backup into a remote machine's tape drive. Here are the commands that can be used to achieve this. Execute this command on the machine you want to backup.

$ tar cvf - $DIRNAME | rsh $SYS dd of=$TAPEDEV

Substitute

$DIRNAME with the directory to backup,

$SYS with the machine name with the tape drive,

$TAPEDEV with the tape device.

Note: You must be able to rlogin into the remote machine without a password. To do this add the name of your local machine with your user name in the .rhost file in your home directory on the remote machine.

To retrieve the backed up info...

rsh $REM dd if=$TAPEDEV | tar xvf - Tips For Linux - How and when to use the dd command In this article, Sam Chessman explains the use of the dd command with a lot of useful examples. This article is not aimed at absolute beginners. Once you are familiar with the basics of Linux, you would be in a better position to use the dd command.

For blocked I/O, the dd command has no competition in the standard tool set. One could write a custom utility to do specific I/O or formatting but, as dd is already available almost everywhere, it makes sense to use it.

Like most well-behaved commands, dd reads from its standard input and writes to its standard output, unless a command line specification has been given. This allows dd to be used in pipes, and remotely with the rsh remote shell command.

$ dd --help

In essence, dd copies and optionally converts data. It uses an input buffer, conversion buffer if conversion is specified, and an output buffer. Reads are issued to the input file or device for the size of the input buffer, optional conversions are applied, and writes are issued for the size of the output buffer. This allows I/O requests to be tailored to the requirements of a task. Output to standard error reports the number of full and short blocks read and written.

Example 1

A typical task for dd is copying a floppy disk. As the common geometry of a 3.5" floppy is 18 sectors per track, two heads and 80 cylinders, an optimized dd command to read a floppy is:

Example 1-a : Copying from a 3.5" floppy

dd bs=2x80x18b if=/dev/fd0 of=/tmp/floppy.image
1+0 records in
1+0 records out

cp /dev/fd0 /tmp/floppy.image

issues 360 reads and writes of 4096 bytes. While this may seem insignificant on a 1.44MB file, when larger amounts of data are involved, reducing the number of system calls and improving performance can be significant.

To finish copying a floppy, the original needs to be ejected, a new diskette inserted, and another dd command issued to write to the diskette:

Example 1-b : Copying to a 3.5" floppy
dd bs=2x80x18b < /tmp/floppy.image > /dev/fd0
1+0 records in
1+0 records out

Here is shown the stdin/stdout usage, in which respect dd is like most other utilities.

Example 2

Notice the output record count is smaller than the input record count. This is due to the padding spaces eliminated from the output file and replaced with newline characters.

Example 3

Example 3 : Byte Swapping with Remote Access of Magnet Tape
rsh sgi.with.tape dd bs=256b if=/dev/rmt0 conv=swab | tar xvf -

The dd runs on the SGI and swaps the bytes before writing to the tar command running on the local host.

Example 4

Example 4 : Error Handling
dd bs=265b conv=noerror if=/dev/st0 of=/tmp/bad.tape.image

Example 5

The Linux kernel Makefiles use dd to build the boot image. In the Alpha Makefile /usr/src/linux/arch/alpha/boot/Makefile, the srmboot target issues the command:

Example 5 : Kernel Image Makefile
dd if=bootimage of=$(BOOTDEV) bs=512 seek=1 skip=1

size="-2">Credits
size="-2">
The dd command has been around since the 1970s, ported to many systems, rewritten many times, and tested by time as a useful tool. The current Linux version is GNU dd GNU fileutils 3.12, written by Paul Rubin, David MacKenzie, and Stuart Kemp, Copyright © 1985, 1990, 1991 Free Software Foundation, Inc.

GNU dd is found in the fileutils collection, with the current version at the URL ftp://prep.ai.mit.edu/pub/gnu/fileutils-3.12.tar.gz or a mirror near you.

Other major versions include SYSV and BSD, with the BSD source version 5.16 4/28/93 derived from software contributed to Berkeley by Keith Muller of the University of California, San Diego and Lance Visser of Convex Computer Corporation, Copyright © 1991 The Regents of the University of California.

Scripts

#!/bin/sh

tar -cvf my.tar $(for i in `cat list`

do

       echo $i

   done)

exit

This script is backing up to tape using dump command. Logging date and all the messages

to a log file.

#!/bin/sh

echo "$DATE"backup.log

filenumber=`/usr/bin/mt stat|/usr/bin/grep "File Number"|/usr/bin/awk '{print $3}'`

echo "Backing up / to tape location: $filenumber"backup.log

/sbin/dump -0ua -f /dev/nrsa0 / &2backup.log

if [ $? -eq 0 ];then

   echo "/ backup successful"$HOME/log/backup.log

fi

Copy files ( even complete filesystem) from remote to local system

Note: You must be able to rlogin into the remote machine without a password. To do this add the name of your local machine with your user name in the .rhost file in your home directory on the remote machine.

#!/bin/sh

# Copies files from Remote System to the local current directory

name=`basename $0`

if [ $# -ne 2 ];then

echo "Usage: $name <remote-system> <dir-to-copy>"

exit

fi

system=$1

dir_to_cp=$2

rsh $system "cd $dir_to_cp; find . -print|cpio -ocB"|dd ibs=5k obs=5k|cpio -iducmvB

<img>

This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: December 18, 2004

Unix ln command and its usage for simplifing navigation

In Unix, what is a symbolic link, and how do I create one?

Also see:

Old News ;-)

9.2.1.2. Using the dd command to dump data

Recommended Links

Reference

dd The name dd stands for "copy and convert". Don't see it? Well, "cc" was already taken for the C compiler, so the author chose the next letter in the alphabet. The syntax has sort of an evil, JCL-like quality to it. According to The Jargon File, the interface was a prank.

Using dd

GNU dd

AIX dd

Solaris dd

Recommended Articles

Remote backup

Scripts