<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hey, Kevin. I guess I misled you last week when I suggested two
processes reading from the same pipe might work to implement a
queue. I thought the "read" calls in the script would do the right
thing, but that appears not to be the case.<br>
<br>
The key to avoiding race conditions between co-operating shell
processes is to find some sort of operation that's "atomic" - that
does its work all in one shot so it won't be interrupted
mid-stream. It appears that "read" is atomic at the single
character level, not at the single line level as would be needed for
this queue to work properly. I recall now that on Unix/Linux
systems, the link system call, or the ln command is good for that.
If two processes try to make a hard link, of the same name, to the
same file without forcing it (-f), only one of them will succeed.
So, instead of implementing a queue of file names to process, you
could make both processes use the same file list but for each file
check if a) the file has not already been processed, and b) you can
link to the file, and if both conditions are met the loop can go
ahead and process that file. If I recall, the processing was to
compress each tar file, and you wanted two simultaneous processes to
speed up the task on a multi-core system. Each process could be
implemented something like the following:<br>
<br>
ls /zonebackup/*tar | while read F; do<br>
if [ ! -e "$F.gz" -a ! -e "$F.lock" ] && ln "$F"
"$F.lock" 2>/dev/null; then<br>
gzip "$F"<br>
rm -f "$F.lock"<br>
fi<br>
done<br>
<br>
Theoretically, you should be able to run two (or more) of these and
they should compress all the files without getting in each other's
way. The loop won't even attempt the ln command if it sees the .gz
file or .lock link already there, but in the event of a race where
both processes decide almost simultaneously that the lock isn't
there, and both run the ln command, only one of the two ln's will
succeed, and only for that successful one will it go on to compress
the file and remove the lock link. The redirect to /dev/null is to
keep the losing ln command quiet. I haven't actually tested this
whole script, though. But I have successfully used ln in the past
for locking files in this manner.<br>
<br>
The next step would be to find the optimum number of processes to
run before you lose any speedup of the overall job. This would
likely depend on the number of files, their sizes, and the number
and speed of your CPU cores. The faster a CPU core is, the more the
compression task goes from being CPU-bound to being I/O-bound, and
the less gain you get from more parallel processes. Compressing
lots of small files, rather than a few larger ones, would likely
skew things towards being more I/O-bound too, because of the extra
overhead of each gzip startup.<br>
<br>
<div class="moz-cite-prefix">On 01/03/2014 4:48 PM, Kevin McGregor
wrote:<br>
</div>
<blockquote
cite="mid:CACBFUzioGnTZTKhrT5a9C=hrn8SnjxHp6xkWAkt1sMh-jED8=w@mail.gmail.com"
type="cite">
<div dir="ltr">Aw. I was afraid of that. In my 'production' script
there would be a long time between most reads, so it's unlikely
there would be a problem, but I still don't want rare random
failures. I'll find a work-around.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Sat, Mar 1, 2014 at 2:12 PM, Adam
Thompson <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:athompso@athompso.net" target="_blank">athompso@athompso.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb">
<div class="h5">Oh, it's obvious when I think about it -
the behavior of a pipe with multiple readers is
well-defined as being OS and clib-dependent.<br>
Each byte is only available to one reader, ever - if the
reading is interleaved, each reader will get garbage.<br>
You can use tee(1) to write to multiple FIFOs at once,
or just adapt the writing script to do so manually.<br>
-Adam<br>
<br>
On Mar 1, 2014 1:35 PM, Kevin McGregor <<a
moz-do-not-send="true"
href="mailto:kevin.a.mcgregor@gmail.com">kevin.a.mcgregor@gmail.com</a>>
wrote:<br>
><br>
> The writer is:<br>
> #/bin/ksh<br>
> PIPE=/tmp/backup.pipe<br>
> [[ ! -a $PIPE ]] && mkfifo $PIPE<br>
> # Start gzip processes<br>
> /opt/cronjobs/zip1 &<br>
> /opt/cronjobs/zip2 &<br>
><br>
> # Process files needing compression<br>
> let 'fc=0'<br>
> ls /zonebackup/*tar | while read F; do<br>
> echo $F >$PIPE<br>
> let 'fc=fc+1'<br>
> done<br>
><br>
> echo "end of list" >$PIPE<br>
> echo "end of list" >$PIPE<br>
> exit 0<br>
><br>
> The readers are:<br>
> #/bin/ksh<br>
> PIPE=/tmp/backup.pipe<br>
> NAME=zip1<br>
> if [[ ! -a $PIPE ]]; then<br>
> logger -p local0.warning "$NAME can't find
$PIPE -- exiting"<br>
> exit 1<br>
> fi<br>
><br>
> while (( 1 )); do<br>
> read F <$PIPE<br>
> if [[ "$F" = "end of list" ]]; then<br>
> break<br>
> else<br>
> echo "$NAME: $F"<br>
> fi<br>
> done<br>
><br>
><br>
><br>
> On Sat, Mar 1, 2014 at 1:29 PM, Kevin McGregor <<a
moz-do-not-send="true"
href="mailto:kevin.a.mcgregor@gmail.com">kevin.a.mcgregor@gmail.com</a>>
wrote:<br>
>><br>
>> I tried fiddling with IFS to no avail. I just
changed it like this:<br>
>> IFS='<br>
>> '<br>
>> And now the readers show all kinds of
gibberish! All lines have no whitespace, save for the
newline at the end. I'm assuming it's at the end. <br>
>><br>
>><br>
>> On Sat, Mar 1, 2014 at 12:47 PM, Robert Keizer
<<a moz-do-not-send="true"
href="mailto:robert@keizer.ca">robert@keizer.ca</a>>
wrote:<br>
>>><br>
>>> Have you tried setting IFS ( field sep )?
Also you could enable raw mode with -r.<br>
>>><br>
>>> Can you share the script?<br>
>>><br>
>>> Are the same lines failing repeatedly?<br>
>>><br>
>>> Rob<br>
>>><br>
>>> On 2014-03-01 11:55 AM, "Kevin McGregor"
<<a moz-do-not-send="true"
href="mailto:kevin.a.mcgregor@gmail.com">kevin.a.mcgregor@gmail.com</a>>
wrote:<br>
>>>><br>
>>>> I have a main script which writes to a
named pipe. Before it starts writing, it starts two
other scripts which read from this pipe. The reading and
writing is a list of file names, one per line. How do I
ensure that each script reads one complete line from the
pipe at a time (no more, no less)?<br>
>>>><br>
>>>> I have a test set up, and it usually
works, but sometimes a reader will get a blank line or
just a "/" (but not any other part of a line)!<br>
>>>><br>
>>>> Kevin<br>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Gilles R. Detillieux E-mail: <a class="moz-txt-link-rfc2396E" href="mailto:grdetil@scrc.umanitoba.ca"><grdetil@scrc.umanitoba.ca></a>
Spinal Cord Research Centre WWW: <a class="moz-txt-link-freetext" href="http://www.scrc.umanitoba.ca/">http://www.scrc.umanitoba.ca/</a>
Dept. Physiology, U. of Manitoba Winnipeg, MB R3E 0J9 (Canada)
</pre>
</body>
</html>