simultaneous copy to multiple media

[Claudio Grondi]

I would like to save time copying the same file
(>6 GByte) to various different target storage
media connected to the system via USB.

Is there a (Python or other) tool able to help me
to do this, so that I don't need to copy the
source file first to the first media, then to the
second, etc.?

Claudio

[Swaroop C H]

Re: simultaneous copy to multiple media

On Sun, 20 Mar 2005 11:41:10 -0000, Claudio Grondi
<claudio.grondi @freenet.de> wrote:[color=blue]
> I would like to save time copying the same file
> (>6 GByte) to various different target storage
> media connected to the system via USB.
>
> Is there a (Python or other) tool able to help me
> to do this, so that I don't need to copy the
> source file first to the first media, then to the
> second, etc.?[/color]

If you are using *nix, maybe you can use the `tee` command[1] and
redirect the file to different places.

For example,

cat <somefile> | tee file1 | tee file2 > file3

I haven't tried it out but it should work.

[1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?tee

HTH,
--
Swaroop C H
Blog: http://www.swaroopch.info
Book: http://www.byteofpython.info

[Jacek Trzmiel]

Re: simultaneous copy to multiple media



Claudio Grondi wrote:[color=blue]
>
> I would like to save time copying the same file
> (>6 GByte) to various different target storage
> media connected to the system via USB.
>
> Is there a (Python or other) tool able to help me
> to do this, so that I don't need to copy the
> source file first to the first media, then to the
> second, etc.?[/color]

Use 'tee' command from shell:
tee <file >/dest1/file /dest2/file

Or run two copies in parallel (one of them probably will read file from
cache):
cp file /dest1/file & cp file /dest2/file


-- Jacek.

[Claudio Grondi]

Re: simultaneous copy to multiple media

I am on a Widows 2000 box using the NTFS file system.
Both up to now suggested approaches as
- tee.exe (where I used the http://david.tribble.com/dos/tee.exe
DOS-port with redirecting stdout to NULL) and
- parallel copy (hoping caching does the job) are by far
slower than the consecutive copy:
single copy speed 12-15 MByte/s
which gives effectively
6-7 MByte/s,
tee.exe or twice copy in parallel
1-3 MByte/s.

Any other suggestions except writing an own
optimised version of tee.exe, because it is so
easy to come up with:
while(c=getchar ()){fputc(c,fp1 );fputc(c,fp2); }
?

Is there maybe a way to use a direct DMA
transfer to multiple target destinations
(I copy to drives connected via USB ports) ?

Can someone point me in the right direction?

Claudio

"Claudio Grondi" <claudio.grondi @freenet.de> schrieb im Newsbeitrag
news:3a52b3F66f ideU1@individua l.net...[color=blue]
> I would like to save time copying the same file
> (>6 GByte) to various different target storage
> media connected to the system via USB.
>
> Is there a (Python or other) tool able to help me
> to do this, so that I don't need to copy the
> source file first to the first media, then to the
> second, etc.?
>
> Claudio
>
>[/color]

[Grant Edwards]

Re: simultaneous copy to multiple media

On 2005-03-20, Claudio Grondi <claudio.grondi @freenet.de> wrote:
[color=blue]
> Is there maybe a way to use a direct DMA
> transfer to multiple target destinations
> (I copy to drives connected via USB ports) ?[/color]

Sure, but you'll have to throw away your OS, and write a
program that runs on the bare metal.
[color=blue]
> Can someone point me in the right direction?[/color]

I think they already did.

--
Grant Edwards grante Yow! I am NOT a nut....
at
visi.com

[Claudio Grondi]

Re: simultaneous copy to multiple media

means your message, that you think, that
the consecutive copy is the fastest possible
method if using Windows 2000?

Claudio

"Grant Edwards" <grante@visi.co m> schrieb im Newsbeitrag
news:113ra9iiqg 95edc@corp.supe rnews.com...[color=blue]
> On 2005-03-20, Claudio Grondi <claudio.grondi @freenet.de> wrote:
>[color=green]
> > Is there maybe a way to use a direct DMA
> > transfer to multiple target destinations
> > (I copy to drives connected via USB ports) ?[/color]
>
> Sure, but you'll have to throw away your OS, and write a
> program that runs on the bare metal.
>[color=green]
> > Can someone point me in the right direction?[/color]
>
> I think they already did.
>
> --
> Grant Edwards grante Yow! I am NOT a[/color]
nut....[color=blue]
> at
> visi.com[/color]

[Heiko Wundram]

Re: simultaneous copy to multiple media

On Sunday 20 March 2005 17:16, Claudio Grondi wrote:[color=blue]
> Is there maybe a way to use a direct DMA
> transfer to multiple target destinations
> (I copy to drives connected via USB ports) ?[/color]

Think about what USB stands for. Then reconsider whether you'll ever have the
chance of writing truly simultaneously to several devices connected via
USB... And then, as an extra exercise, think about why it takes so long when
several different jobs are done in parallel, writing to devices connected via
USB.

Bus design... Don't they teach anything at uni these days? ;)

--
--- Heiko.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBCPeeqf0b pgh6uVAMRAopNAJ 9fLMRkc3rCt6kfo wV4Nyn6gKvI2wCe Ndvd
giHebgCQT+7HPGD n6So99tU=
=UiN+
-----END PGP SIGNATURE-----

[Rolf Zwart]

Re: simultaneous copy to multiple media

Swaroop C H wrote:[color=blue]
> [...]
> If you are using *nix, maybe you can use the `tee` command[1] and
> redirect the file to different places.
>
> For example,
>
> cat <somefile> | tee file1 | tee file2 > file3[/color]
On Unixes I know, only 1 process is needed:
<infile tee outfile1 outfile2 outfile3 ... >/dev/null
It does work!

** Rolf[color=blue]
>
> I haven't tried it out but it should work.
>
> [1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?tee
>
> HTH,[/color]

[Claudio Grondi]

Re: simultaneous copy to multiple media

I don't know any deep details about USB, except that
I _know_ that it is a serial bus, but considering
following:

1) I can read/write 45 MByte/s from harddrive to harddrive
on the E-IDE bus (theoretically 100 MByte/s), so the
speed of the harddrive I read/write from/to is probably the
bottleneck.
2) if I understand it right, an USB controller is connected to
to the PCI bus and there can be many separate USB
controller on one PC
3) the theoreticall speed of USB (430 MByte/s?) is much
higher as the experienced 15 MByte/s, probably due to slow
controller on the side of the external storage media, so
maybe even on one USB line I will have the chance to
use it full capacity connecting many slow devices
(I can't imagine, that USB goes down with the speed to
the speed of the slowest component connected, does it?)
4) with 45 MByte/s I could theoretically simultaneously
supply three separate data streams to three USB
controller writing with 15 MByte/s each out to the
external storage media.
5) watching the low CPU usage while writing to USB
shows me, that CPU-time is not a problem.

I still see a chance of writing truly simultaneously to
several devices connected via USB.
Please let me know if I am wrong in any of the
points (1-5) or if my conclusion is based on
wrong assumptions or if I use a wrong way of
infering.

By the way: I was not thought about bus design at
university, so what I know about it is comes from
self-study :-).

Claudio
P.S. Your message appears in my Outlook Express
as one having no content, but an attachment, so I can't
read it directly.
[color=blue]
> Is there maybe a way to use a direct DMA
> transfer to multiple target destinations
> (I copy to drives connected via USB ports) ?[/color]
Think about what USB stands for. Then reconsider whether you'll ever have
the
chance of writing truly simultaneously to several devices connected via
USB... And then, as an extra exercise, think about why it takes so long when
several different jobs are done in parallel, writing to devices connected
via
USB.
Bus design... Don't they teach anything at uni these days? ;)
--
--- Heiko.

[Diez B. Roggisch]

Re: simultaneous copy to multiple media

> 2) if I understand it right, an USB controller is connected to[color=blue]
> to the PCI bus and there can be many separate USB
> controller on one PC[/color]

True.
[color=blue]
> 3) the theoreticall speed of USB (430 MByte/s?) is much
> higher as the experienced 15 MByte/s, probably due to slow
> controller on the side of the external storage media, so
> maybe even on one USB line I will have the chance to
> use it full capacity connecting many slow devices
> (I can't imagine, that USB goes down with the speed to
> the speed of the slowest component connected, does it?)[/color]

Nope. Or yes. Here comes into play why Heiko said think of what USB stands
for. While the devices appear to be responsive concurrently (the bus
arbitration and multiplexing/demultiplexing is tranparent to the user), it
still is a serial bus, so at a given point in time you can only write to
_one_ device. And if that device is limited in its speed, you're doomed.
The only thing I could imagine is that if you send small bursts that don't
make the write cache full, it _might_ be possible to switch fast between
the devices and thus write the data faster, as the devices can try and
write the data to disk while the others are feeded.

But I'm on no sure ground here with that assumption, and it certainly would
depend on how the usb-ata-controller deals with that. You said yourself
that you only had 15MB/sec, so it doesn't look to good.

To make this work would certainly be a deep lowlever driver hack and nowhere
in the scope of python per se.
[color=blue]
> 4) with 45 MByte/s I could theoretically simultaneously
> supply three separate data streams to three USB
> controller writing with 15 MByte/s each out to the
> external storage media.[/color]

That might be the case if you use three real separate controllers. But that
will depend on how the OS deals with these, if you really achieve the
theoretical throughoutput.
[color=blue]
> 5) watching the low CPU usage while writing to USB
> shows me, that CPU-time is not a problem.[/color]

That is certainly true - the bottleneck are the buses.

--
Regards,

Diez B. Roggisch

[Diez B. Roggisch]

Re: simultaneous copy to multiple media

>[color=blue]
> Nope. Or yes. Here comes into play why Heiko said think of what USB stands
> for. While the devices appear to be responsive concurrently (the bus
> arbitration and multiplexing/demultiplexing is tranparent to the user), it
> still is a serial bus, so at a given point in time you can only write to
> _one_ device. And if that device is limited in its speed, you're doomed.
> The only thing I could imagine is that if you send small bursts that don't
> make the write cache full, it _might_ be possible to switch fast between
> the devices and thus write the data faster, as the devices can try and
> write the data to disk while the others are feeded.[/color]


I had second thoughts of this, and have to admit that I don't make too much
senes here. So ignore that.
[color=blue]
> But I'm on no sure ground here with that assumption, and it certainly
> would depend on how the usb-ata-controller deals with that. You said
> yourself that you only had 15MB/sec, so it doesn't look to good.
>
> To make this work would certainly be a deep lowlever driver hack and
> nowhere in the scope of python per se.
>[/color]

--
Regards,

Diez B. Roggisch

[Heiko Wundram]

Re: simultaneous copy to multiple media

Am Sonntag, 20. März 2005 23:16 schrieb Claudio Grondi:[color=blue]
> 2) if I understand it right, an USB controller is connected to
> to the PCI bus and there can be many separate USB
> controller on one PC[/color]

Yes, there may be more than one USB-controller in a PC, but this doesn't
matter, they are all connected to the same bus (the PCI bus). This is a
parallel bus, but also a bus which can only serve a single endpoint at a
time. The frequency is normally 66 Mhz, so you get a theoretical 66*32 Mb/s
over the PCI bus.

But, this is only theoretical usage. Consider the EIDE-controller trying to
write data into DMA memory which it has read from disk. It has to do the
following to communicate with memory:

1) signal the DMA chip (which is also a device on the PCI bus) that it has
data. This implies grabbing the PCI bus, making sure noone else is talking at
the moment.

2) pass the data, 32 bit at a time, to the DMA chip, which places this data
into main memory.

3) you may not grab the bus too long, as other devices might also want to
communicate with some other device (e.g. the PIC, to signal an interrupt to
the CPU). Thus, you have to free the bus after some time, and you have to
forcibly give free the bus when some other device signals an interrupt. This
also means starting transmission again after the bus has been successfully
reacquired.

4) finally, after communication with the DMA-chip is done, signal the PIC via
an interrupt that you wish the CPU to be signaled that it should do a
software interrupt, so that the OS interrupt handler can be scheduled and can
take the data from the DMA region (and do something with it).

Now, even when the bus is fast, setting it up all still takes time, so you
never get the theoretical throughput you might wish to see.

Accessing the USB controller does the game the other way around; the OS uses
the DMA-chip to stream the data onto the PCI bus for the controller to
transmit.

So, overall, adding more USB controllers to your PC only adds extra throughput
until you reach the top limit of PCI (when the controllers start stealing
bandwith from each other).
[color=blue]
> 3) the theoreticall speed of USB (430 MByte/s?) is much
> higher as the experienced 15 MByte/s, probably due to slow
> controller on the side of the external storage media, so
> maybe even on one USB line I will have the chance to
> use it full capacity connecting many slow devices
> (I can't imagine, that USB goes down with the speed to
> the speed of the slowest component connected, does it?)[/color]

Well, 430 MB/s is only for USB 2.0. AFAIK, most devices (esp. storage devices)
are still only USB 1.1 compliant, which makes this rate go down to a mere 40
MB/s or something close.
[color=blue]
> 4) with 45 MByte/s I could theoretically simultaneously
> supply three separate data streams to three USB
> controller writing with 15 MByte/s each out to the
> external storage media.[/color]

No, you can't. As I said in response to point 2, it doesn't matter what kind
of bus you use, when the bus isn't meant to do real "broadcasts " or can
actually talk to several devices simultaneously because all wires are
duplicated a corresponding number of times, there's always overhead when
switching between the devices.

The bus must be acquired for communication with one device (so that the others
won't simply go blabbering over the communication), then the data must be
streamed, then the bus released again. This process takes time, and AFAIK,
USB is pretty expensive (compared to data transmission speed) here, as USB
guarantees that a packet reaches the endpoint when its transmitted over the
wire (in the sense that it won't get lost because someone else was also
talking). Now, when more devices come into play, this overhead becomes
significant, and takes away quite some bandwith.

For example, Ethernet takes another way. Ethernet simply says: when I have
data which is transmitted while someone else was talking, throw away the
resulting mess (a collision, remember those cool blinking lights on hubs?).
Someone will retransmit it if it was important.
[color=blue]
> 5) watching the low CPU usage while writing to USB
> shows me, that CPU-time is not a problem.[/color]

No, it most certainly isn't. That's why modern computers always use a DMA-chip
to do the heavy work of dealing with the hardware on external busses... ;)
[color=blue]
> P.S. Your message appears in my Outlook Express
> as one having no content, but an attachment, so I can't
> read it directly.[/color]

Err, that's pretty strange. I can read my KMail written emails in pretty much
any client that I have access to... So I guess this is a problem with Outlook
Express not understanding MIME correctly, and not a problem on my side... ;)
And, don't worry, I won't switch to <windows sucks>-mode now. ;)

--
--- Heiko.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQBCPgR0f0b pgh6uVAMRAvmQAJ 0UOF+foOuTktKgX T4C1Ghhz7loEACf TtQb
5qASZyT1uJfoMw4 qjS7/pSU=
=0Yv+
-----END PGP SIGNATURE-----

[Jacek Trzmiel]

Re: simultaneous copy to multiple media


Claudio Grondi wrote:[color=blue]
> I am on a Widows 2000 box using the NTFS file system.
> Both up to now suggested approaches as
> - tee.exe (where I used the http://david.tribble.com/dos/tee.exe
> DOS-port with redirecting stdout to NULL) and
> - parallel copy (hoping caching does the job) are by far
> slower than the consecutive copy:
> single copy speed 12-15 MByte/s
> which gives effectively
> 6-7 MByte/s,
> tee.exe or twice copy in parallel
> 1-3 MByte/s.
>
> Any other suggestions except writing an own
> optimised version of tee.exe[/color]

Try this:

http://mastermind.com.pl/multicopy/

This is small tool I've wrote, that does use large memory buffers with
asynchronous I/O to copy file.

Following command:
multicopy c:\testfile d:\testfile e:\testfile f:\testfile
will copy c:\testfile to d, e and f disks.

With four separate IDE disks I can copy file at about 30MB/s, which
means 120MB/s total I/O. You can give it a try, but I don't know if it
will work fast with USB drives.

HTH,
sc0rp.

[Diez B. Roggisch]

Re: simultaneous copy to multiple media

> Well, 430 MB/s is only for USB 2.0. AFAIK, most devices (esp. storage[color=blue]
> devices) are still only USB 1.1 compliant, which makes this rate go down
> to a mere 40 MB/s or something close.[/color]

I think it is 430 MBit(!), so about 50MB per second. The old usb 1.1 has
12MBits. So I don't think that Claudio could observe 15MByte/s with a 1.1
bus....
[color=blue]
>[color=green]
>> 4) with 45 MByte/s I could theoretically simultaneously
>> supply three separate data streams to three USB
>> controller writing with 15 MByte/s each out to the
>> external storage media.[/color]
>
> No, you can't. As I said in response to point 2, it doesn't matter what
> kind of bus you use, when the bus isn't meant to do real "broadcasts " or
> can actually talk to several devices simultaneously because all wires are
> duplicated a corresponding number of times, there's always overhead when
> switching between the devices.[/color]

Well, if the bus is faster, it can (and does) use multiplexing. All busses
do that, apart from crossbar switches which have n:n point2point
connectivity.[color=blue]
> The bus must be acquired for communication with one device (so that the
> others won't simply go blabbering over the communication), then the data
> must be streamed, then the bus released again. This process takes time,
> and AFAIK, USB is pretty expensive (compared to data transmission speed)
> here, as USB guarantees that a packet reaches the endpoint when its
> transmitted over the wire (in the sense that it won't get lost because
> someone else was also talking). Now, when more devices come into play,
> this overhead becomes significant, and takes away quite some bandwith.[/color]

USB uses a host-based bus arbitration scheme where only the controller
decides who can talk when. So there is very limited bus arbitration
overhead. But of course this also means that the host has grant each
attached device bus time every now and then, as the device itself has no
chance of signalling data availability itself. So - that adds again some
overhead :)

In the end, I'm not sure what throughoutput is actually possible. As I said
to Claudio in a private mail: Apart from multithreaded writing to the bus
and _hoping_ that things speed up one can't do much - at least not in
python, and not without deep driver fiddling or even writing drivers
oneself.
--
Regards,

Diez B. Roggisch

[Dennis Lee Bieber]

Re: simultaneous copy to multiple media

On Sun, 20 Mar 2005 22:16:21 -0000, "Claudio Grondi"
<claudio.grondi @freenet.de> declaimed the following in comp.lang.pytho n:
[color=blue]
> 3) the theoreticall speed of USB (430 MByte/s?) is much[/color]

USB 2.0... ~440 Mbit/sec

USB 1.1 is around 11Mbps

USB 1.0 <2Mbps

FireWire... 400Mbps

10Base-T Ethernet, 10Mbps (USB 1.1 just beats it in theory)
100Base-T Ethernet (aka EtherFast) 100Mbps
Gigabit Ethernet 1000Mbps

USB/Ethernet/Firewire are serial protocols, Your IDE (or SCSI)
drives are parallel
[color=blue]
> (I can't imagine, that USB goes down with the speed to
> the speed of the slowest component connected, does it?)[/color]

Yes, it does... Plug a 1.1 device into a 2.0 bus, and the bus
controller down-shift the whole thing to 1.1 speeds.

But if you mean data rate /within/ the device... You'll get a
polling situation. Data will transfer at the USB rate, but you may have
pauses while the device fills/empty internal buffers.
[color=blue]
> I still see a chance of writing truly simultaneously to
> several devices connected via USB.[/color]

The protocol is still serial... You may have submitted three
write commands, to three different devices, but the USB driver has to
format the data into packets for each device, sending one after the
other... Most likely, for such a drive, 512/1K bytes (4/8Kbits plus
overhead). If you are lucky, the first drive will still be processing
the first packet by the time the driver comes around to sending it its
second packet.


--[color=blue]
> =============== =============== =============== =============== == <
> wlfraed@ix.netc om.com | Wulfraed Dennis Lee Bieber KD6MOG <
> wulfraed@dm.net | Bestiaria Support Staff <
> =============== =============== =============== =============== == <
> Home Page: <http://www.dm.net/~wulfraed/> <
> Overflow Page: <http://wlfraed.home.ne tcom.com/> <[/color]