new h1 units not downloading
____________
kenlo

new h1 units not downloading

new h1 units not downloading

Here an excerpt from proxomitron log:

+++GET 30654+++
GET /download/38/h1_0205.0 HTTP/1.0
User-Agent: BOINC client
Host: einstein.astro.gla.ac.uk:80
Range: bytes=14736000-
Accept: */*
Connection: keep-alive

+++RESP 30654+++
HTTP/1.0 416 Requested Range Not Satisfiable
Date: Mon, 27 Jun 2005 23:41:50 GMT
Server: Apache/2.0.54 (Debian GNU/Linux) DAV/2 SVN/1.1.4 mod_python/3.1.3 Python/2.3.5 PHP/4.3.10-15 mod_ssl/2.0.54 OpenSSL/0.9.7e mod_perl/1.999.21 Perl/v5.8.4
Keep-Alive: timeout=15, max=89
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
+++CLOSE 30654+++

+++GET 30655+++
GET /download/38/h1_0205.0 HTTP/1.0
User-Agent: BOINC client
Host: einstein.astro.gla.ac.uk:80
Range: bytes=14736000-
Accept: */*
Connection: keep-alive

+++RESP 30655+++
HTTP/1.0 416 Requested Range Not Satisfiable
Date: Mon, 27 Jun 2005 23:41:54 GMT
Server: Apache/2.0.54 (Debian GNU/Linux) DAV/2 SVN/1.1.4 mod_python/3.1.3 Python/2.3.5 PHP/4.3.10-15 mod_ssl/2.0.54 OpenSSL/0.9.7e mod_perl/1.999.21 Perl/v5.8.4
Keep-Alive: timeout=15, max=88
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
+++CLOSE 30655+++

There is some filesize wrong!
____________
greetz, Uli

I had the same problem just now and I had to reset the project on that PC.


The reason :

It had two download tasks running on exactly the same file. (h1_0400.0)

One was downloaded successfully with the expected file size and the other downloader "wondered where those bytes all came from" and reported a file size error too with a retry every few seconds.

BOINC 4.19, Dual CPU P3s

After the reset it did download stuff successfully but still it shows is "download failed". Nothing missing but I guess I cannot allow BOINC to have two files with the same filename ;-)

There must be something damaged on server/scheduler side or in the WU XML config.

After a reset I got a H1_501.0

Same problem first - but then after successful(!) transfer of H1_501.0 BOINC got a request to delete H1_501.0 while it was still downloading H1_501.0 on the other download thread.

Of course the client didn't like that too much either - now there's a checksum error, 2 tasks are crunching and a few are still in "downloading" state

Very weird !

The story continues : After manually contacting the scheduler to report the error, it tried to delete H1_501.0

BOINC was very sad and told me it couldn't delete H1_501.0 .... but the work units are happy now and not trying to download H1_501.0 again (as it's still there of course)
___________

I guess it's the WU configuration that is wrong, the scheduler request which I saved after the first problem had this in it :

<file_info>
<name>H1_0400.0</name>
<report_on_rpc/>
</file_info>
<file_info>
<name>h1_0400.0</name>
<report_on_rpc/>
</file_info>

i.e. twice the same stuff


I would rate this as a critical problem

I would rate this as a critical problem[/quote]
Same here, just caught one machine in an endless loop here is an excerpt
6/27/2005 8:16:21 PM|Einstein@Home|Temporarily failed download of h1_0673.0: 416
6/27/2005 8:16:23 PM|Einstein@Home|Couldn't delete file projects/einstein.phys.uwm.edu/h1_0673.0
6/27/2005 8:16:24 PM|Einstein@Home|Started download of h1_0673.0
6/27/2005 8:16:25 PM|Einstein@Home|Temporarily failed download of h1_0673.0: 416
6/27/2005 8:16:27 PM|Einstein@Home|Couldn't delete file projects/einstein.phys.uwm.edu/h1_0673.0
6/27/2005 8:16:27 PM|Einstein@Home|Started download of h1_0673.0
6/27/2005 8:16:28 PM|Einstein@Home|Temporarily failed download of h1_0673.0: 416
6/27/2005 8:16:30 PM|Einstein@Home|Couldn't delete file projects/einstein.phys.uwm.edu/h1_0673.0
6/27/2005 8:16:30 PM|Einstein@Home|Started download of h1_0673.0
6/27/2005 8:16:31 PM|Einstein@Home|Temporarily failed download of h1_0673.0: 416
It went on till I aborted transfer which appears to have killed the issue. This machine running Einstein beta and 4.45 windows.
____________

I would rate this as a critical problem

Same here, just caught one machine in an endless loop here is an excerpt
6/27/2005 8:16:21 PM|Einstein@Home|Temporarily failed download of h1_0673.0: 416
6/27/2005 8:16:23 PM|Einstein@Home|Couldn't delete file projects/einstein.phys.uwm.edu/h1_0673.0
6/27/2005 8:16:24 PM|Einstein@Home|Started download of h1_0673.0
6/27/2005 8:16:25 PM|Einstein@Home|Temporarily failed download of h1_0673.0: 416
6/27/2005 8:16:27 PM|Einstein@Home|Couldn't delete file projects/einstein.phys.uwm.edu/h1_0673.0
6/27/2005 8:16:27 PM|Einstein@Home|Started download of h1_0673.0
6/27/2005 8:16:28 PM|Einstein@Home|Temporarily failed download of h1_0673.0: 416
6/27/2005 8:16:30 PM|Einstein@Home|Couldn't delete file projects/einstein.phys.uwm.edu/h1_0673.0
6/27/2005 8:16:30 PM|Einstein@Home|Started download of h1_0673.0
6/27/2005 8:16:31 PM|Einstein@Home|Temporarily failed download of h1_0673.0: 416
It went on till I aborted transfer which appears to have killed the issue. This machine running Einstein beta and 4.45 windows.

new h1 units not downloading

new h1 units not downloading

all i did after the bad download was to abort it and it seems to be running ok now.

This may be at least partly a screw-up on my side.

The "new" S4 data files are named l1_XXXX.X and h1_XXXX.X, in contrast to the "old" files which are named L1_XXXX.X and H1_XXXX.X.

Unfortunately I had not realized that on Win32, file names are case-insensitive.
So there may be some issues in the next few days if workunits which are supposed to use the file H1_0400.0 (which has a particular size and checksum) try to instead use the file h1_0400.0 (which has a DIFFERENT size and checksum).

Meanwhile, I'll see what I can do on the server side to ameliorate this issue.

Bruce
____________

The "download looping" problem is in boinc 4.43 ...

This may be at least partly a screw-up on my side.


Bruce

What about deleting all the uppercase or lowercase WUs on server side and then later reissuing them with new naming convention?

This should "convert" the temporary download error into a permanent one (with "giving up") so the computers break out of their download loop.

What about deleting all the uppercase or lowercase WUs on server side and then later reissuing them with new naming convention?

This should "convert" the temporary download error into a permanent one (with "giving up") so the computers break out of their download loop.

Unfortunately I had not realized that on Win32, file names are case-insensitive.

What about deleting all the uppercase or lowercase WUs on server side and then later reissuing them with new naming convention?

This should "convert" the temporary download error into a permanent one (with "giving up") so the computers break out of their download loop.

would this waste work that has already been done (even work that has been returned) on those wu?

After some discussions with David Anderson, I've taken the simple way out. I've cancelled the workunits with names that start "h1_" (NOTE: this is case sensitive, work starting "H1_" is NOT cancelled).

I've also removed the problematic h1_XXXX.X data files from the download servers. After these changes propagate to the data server mirrors (15 to 30 minutes) this should generate hard download errors for any client that attempts these WU.

I'll rename the workunits and files using "w1" (w for Washington state, where the Hanford detector is located) and reissue them.

Apologies to everyone for this fiasco. It's my fault. Hopefully we can recover quickly.

Please feel free to manually abort any h1_ workunits. My apologies for wasted CPU cycles. Fortunately these workunits have only been out there for a half-day so this shouldn't be too severe.

Bruce

____________

...Please feel free to manually abort any h1_ workunits. My apologies for wasted CPU cycles. Fortunately these workunits have only been out there for a half-day so this shouldn't be too severe.

Bruce

Any chance to reset the "daily quota" things too for today?

Any chance to reset the "daily quota" things too for today?

DONE!

I've reset the daily result quota for any host that received an h1 workunit.

Bruce

I've had a large number of boxes affected by this. I've only just noticed it a few minutes ago on one box. I stopped BOINC, deleted the h1 file (lower case h), restarted BOINC, forced an update and got a new file (l1 this time - lower case ell) and everything seems sweet again.

I've started looking at other boxes that I can't physically get to immediately and have found quite a number (probably about 10 so far) that have errored out work for no apparent reason today. Interestingly a number of these show signs of autorecovering in that fresh work is appearing in the list of results.

I'm not at all angry about this - c'est la vie, as they say. All I'd like to know is whether all affected boxes will autorecover now that the 8 per day has been reset, or will I physically have to go to each box and delete the offending h1 file?
____________
Cheers,
Gary.

@ Bruce Allen,

Why you didn't change yet the application from 4.79 to 0.03 (Windows)?

-only a Q.-

greetz littleBouncer

____________

I've had a large number of boxes affected by this. I've only just noticed it a few minutes ago on one box. I stopped BOINC, deleted the h1 file (lower case h), restarted BOINC, forced an update and got a new file (l1 this time - lower case ell) and everything seems sweet again.

I've started looking at other boxes that I can't physically get to immediately and have found quite a number (probably about 10 so far) that have errored out work for no apparent reason today. Interestingly a number of these show signs of autorecovering in that fresh work is appearing in the list of results.

I'm not at all angry about this - c'est la vie, as they say. All I'd like to know is whether all affected boxes will autorecover now that the 8 per day has been reset, or will I physically have to go to each box and delete the offending h1 file?

@ Bruce Allen,

Why you didn't change yet the application from 4.79 to 0.03 (Windows)?

-only a Q.-

greetz littleBouncer

OK, thanks very much for the reply. Let me get this straight. If I'm seeing repeated attempts to get a file and repeated checksum errors it's due to a clash between a H1_xxxx and a h1_xxxx and this results in rapidly errored out work.

However, if I see any box with work in its results list starting h1_nnnn then whilst it appears at the moment to be proceeding normally, I'm going to get a rude awakening when that work is finished and attempted to be reported so I'm going to be wasting cycles big time unless I go and delete all h1_ work on all boxes that have it.

Does that about sum it up in layman's terms? :).

If so, then AAAAAARRRRRRRRGGGGGGGGGGGHHHHHHHHHH!!!!!!! :).

Seriously, I'm still not at all mad about this. One redeeming feature is that because of my fetish for keeping small caches there is not a huge number of work units to be wasted even although (at a quick search) there are h1_ files on most of my boxes. In fact most of the wasted cycles have already occurred and if I spend the time it takes to get around every box, I'm probably not going to save very much anyway.

Would like to be assured that the basic layman's summary is correct though.
____________
Cheers,
Gary.

OK, thanks very much for the reply. Let me get this straight. If I'm seeing repeated attempts to get a file and repeated checksum errors it's due to a clash between a H1_xxxx and a h1_xxxx and this results in rapidly errored out work.

However, if I see any box with work in its results list starting h1_nnnn then whilst it appears at the moment to be proceeding normally, I'm going to get a rude awakening when that work is finished and attempted to be reported so I'm going to be wasting cycles big time unless I go and delete all h1_ work on all boxes that have it.

Does that about sum it up in layman's terms? :).

Yes!

I have CANCELLED all h1_ workunits. That means that any CPU time spent on them is entirely wasted. No credits, no glory, no good.

Shoot those workunits before they tire out your CPUs.

The interesting thing was that on at least three occasions BOINC claimed to be able to reget at least part of the hi_nnnn large file. I thought they were all supposedly deleted? Maybe BOINC was kidding itself :).

Ahhh OK... One one occasion, the re-download got to about 50K and then stalled. Maybe I was snagging the file just as the server was deleting it :). When I got the full download (about 8 megs) I'd just stop and delete again and that seemed to cure it. Bit of an eerie feeling when it's telling you it is getting a file that's not supposed to be there. Hopefully all servers are synced up now.

The interesting question is what is going to be the reaction of the silent majority out there who don't regularly follow the lists and are going to be mightily confused by these strange happenings. Is it possible to send a small email to all registered users to warn them to check if they have h1_nnnn style data files and if so check the web page for details? I can just imagine the complaints if someone has a couple of days of h1 work and they don't immediately notice that there is no credit. I watched one of mine do that and that spurred me into action :).
____________
Cheers,
Gary.

What about the "l1_xxx" WU's? I understand the point that lowercase h WU's are troublesome right now and should be aborted. What about lowercase l WU's?
____________

What about the "l1_xxx" WU's? I understand the point that lowercase h WU's are troublesome right now and should be aborted. What about lowercase l WU's?

The interesting question is what is going to be the reaction of the silent majority out there who don't regularly follow the lists and are going to be mightily confused by these strange happenings. Is it possible to send a small email to all registered users to warn them to check if they have h1_nnnn style data files and if so check the web page for details? I can just imagine the complaints if someone has a couple of days of h1 work and they don't immediately notice that there is no credit. I watched one of mine do that and that spurred me into action :).

I found a script that I have used before, which I can use to grant credit to users/hosts/teams for workunits which I have cancelled. I am going to use this to grant credit to people who have had the misfortune of getting and doing work then having it cancelled.

I found a script that I have used before, which I can use to grant credit to users/hosts/teams for workunits which I have cancelled. I am going to use this to grant credit to people who have had the misfortune of getting and doing work then having it cancelled.

that is a nice touch, Bruce.

Fortunately it does not affect me, but I'm pleased to see the swift way the problem has been dealt with.

Yes!

I have CANCELLED all h1_ workunits. That means that any CPU time spent on them is entirely wasted. No credits, no glory, no good.

Shoot those workunits before they tire out your CPUs.

In anticipation of that answer I've just finished deleting h1_nnnn work on about a dozen boxes that I can actually get physical access to. Bit of a struggle for V4.19 as it doesn't have the nice abort button that the later CCs have. Here's basically what I had to do.

1. Stop BOINC
3. Delete the large h1_nnnn file in the the einstein subdir of the projects dir
4. Restart BOINC. It would complain about missing files and would try to reget them.
5. The current WU would error out and the reget would mostly fail but occasionally it seemed to succeed.
6. Stop BOINC and repeat the procedure. The next h1_nnnn would then seem to error out.
7. I think on all second passes, BOINC would then get an l1_nnnn data file and I knew I was winning.
8. I'd throw in the odd "update" which occasionally seemed to help. I also had to stop and start BOINC to get processing started.

The interesting thing was that on at least three occasions BOINC claimed to be able to reget at least part of the hi_nnnn large file. I thought they were all supposedly deleted? Maybe BOINC was kidding itself :).

I took the easy way out. :-)

.......

So all is well in the world again. :-)

Claude

Real science is VERY error prone. In fact one of the distinguishing characteristics of real research is that (especially the first and second time) one gets it wrong more often than not. The only saving grace in all of this is that with other scientists you get 99.9% forgiveness for being brutally honest about what happened and why. That's the one thing that I can promise Einstein@Home participants that they will get 100% of the time.

[Edit added 30 min later]
I found a script that I have used before, which I can use to grant credit to users/hosts/teams for workunits which I have cancelled. I am going to use this to grant credit to people who have had the misfortune of getting and doing work then having it cancelled.

Bruce

However, it's not my day today :). I took your advice and cancelled running work that was in many cases 80-90% complete!!! And I'm still not mad at you in the slightest :). I'd rather lose the credits than hold up the science by doing work that will only have to be repeated anyway so my cancelling the partly completed work was still the right thing to do.

Edit: Hmm.. 4.19.. Was there a abort/cancel-button on those?
Hope they got reported.(Haven't read all posts here.(too long))

[Edit added 30 min later]
I found a script that I have used before, which I can use to grant credit to users/hosts/teams for workunits which I have cancelled. I am going to use this to grant credit to people who have had the misfortune of getting and doing work then having it cancelled.

Bruce

I'm very pleased that you have done that and it will be good for the silent majority who probably aren't even aware of the problem yet.

However, it's not my day today :). I took your advice and cancelled running work that was in many cases 80-90% complete!!! And I'm still not mad at you in the slightest :). I'd rather lose the credits than hold up the science by doing work that will only have to be repeated anyway so my cancelling the partly completed work was still the right thing to do.

It must have been one of those nightmare days (and nights) for you :).

I just aborted "h1_0118.0__0118.1_0.1_T00_S4ha_0" from my machine, 06/28/2005 08:11:06 PM|Einstein@Home|Starting result l1_0315.5__0315.9_0.1_T00_S4lA_0 using einstein version 4.79.

____________
Greg

I'd also like to say thanks for keeping us informed. Screw-ups happen, and I'm quite happy as long as I'm reasonably well informed.
____________

I confess to being in a pretty foul mood for most of the day today!

I agree, good work from the country of cheese and packers :-)

Especially for the good communication I'll give an A++

Actually you deserve heaps of praise for the way you handled everything. I don't think you could have done more

It is this kind of professionalism that makes me proud to give my full support to this project. Well done, and many thanks for all your efforts!!

it wasn't till I saw this wu that I realised just how much Bruce had done to defuse anger: he has set things up so that people get credit for the part worked wu they cancel part way through - at least I think that is what this wu is telling us

it wasn't till I saw this wu that I realised just how much Bruce had done to defuse anger: he has set things up so that people get credit for the part worked wu they cancel part way through - at least I think that is what this wu is telling us

Your interpretation is entirely correct. I am giving credit for partial/aborted/failed/completed h1_* workunits. Note that this is not instantaneous and may take a few hours. I have to run the script by hand and only do it a few times per day.

Bruce

it wasn't till I saw this wu that I realised just how much Bruce had done to defuse anger: he has set things up so that people get credit for the part worked wu they cancel part way through - at least I think that is what this wu is telling us

Your interpretation is entirely correct. I am giving credit for partial/aborted/failed/completed h1_* workunits. Note that this is not instantaneous and may take a few hours. I have to run the script by hand and only do it a few times per day.

Bruce

it wasn't till I saw this wu that I realised just how much Bruce had done to defuse anger: he has set things up so that people get credit for the part worked wu they cancel part way through - at least I think that is what this wu is telling us

Your interpretation is entirely correct. I am giving credit for partial/aborted/failed/completed h1_* workunits. Note that this is not instantaneous and may take a few hours. I have to run the script by hand and only do it a few times per day.

Bruce

Gary has pointed out to me that credit is not granted for wu that are killed by stealing their files. On consideration this makes sense if the xml that held the cpu time has gone. If the client re-starts the download when the files vanish, presumably it also deletes/overwrites the file that remembers the cpu time so far?

My thought is that it may be better, if running 4.19, to kill those wu from the operating system while BOINC is actually crunching them. This assumes the OS has some kind of task manager (eg not Win-98).

On win-XP for example, hit ctrl-alt-del and the task manager comes up. Highlight the Einstein task, right click, and kill process. The wu will report to BOINC that it ended with some error code that means killed. I think that this means that BOINC will report it back with a 'client error' message and they will get credit.

On linux: you probably already know how to use top or ps to get the pid, and how to use kill to abort. If not, I recommend the man pages on top, ps, kill.

Note: I have tried the win-xp method in the past, but not on these wu. If my suggestion won't work, please say so!