David Anderson and I made modifications to the BOINC scheduler which are designed to resend WU to hosts which have lost them. This only works if you are running a recent client (>=4.45 I think).

Currently any WU which are supposed to be on your machine and which are NOT reported as being there are resent. This is accompanied by a message of the form:

Resent lost result w1_0399.5__0399.6_0.1_T09_S4hA_0

Currently any 'missing' results are sent, even if they are close to deadline.

Please report good and/or bad experiences with this feature in this thread.

Bruce


____________

In the case where a resent WU is close to its deadline, will the client recognize this and go into EDF mode? Or do you have any data on this possibility yet?

Jim

David Anderson and I made modifications to the BOINC scheduler which are designed to resend WU to hosts which have lost them. This only works if you are running a recent client (>=4.45 I think).

Currently any WU which are supposed to be on your machine and which are NOT reported as being there are resent. This is accompanied by a message of the form:

Resent lost result w1_0399.5__0399.6_0.1_T09_S4hA_0

Currently any 'missing' results are sent, even if they are close to deadline.

Please report good and/or bad experiences with this feature in this thread.

Bruce

David Anderson and I made modifications to the BOINC scheduler which are designed to resend WU to hosts which have lost them. This only works if you are running a recent client (>=4.45 I think).

Currently any WU which are supposed to be on your machine and which are NOT reported as being there are resent. This is accompanied by a message of the form:

Resent lost result w1_0399.5__0399.6_0.1_T09_S4hA_0

Currently any 'missing' results are sent, even if they are close to deadline.

Please report good and/or bad experiences with this feature in this thread.

Bruce

David Anderson and I made modifications to the BOINC scheduler which are designed to resend WU to hosts which have lost them. This only works if you are running a recent client (>=4.45 I think).

Currently any WU which are supposed to be on your machine and which are NOT reported as being there are resent. This is accompanied by a message of the form:

Resent lost result w1_0399.5__0399.6_0.1_T09_S4hA_0

Currently any 'missing' results are sent, even if they are close to deadline.

Please report good and/or bad experiences with this feature in this thread.

Bruce

Bruce, it works real well (thanks!) but for one thing. Maybe an unintentional side effect based on what you wrote - when it sends the WU again, it assigns a new complation date. So when the result is close to deadline, you can "reset" the project and get the same work back but with another week to go.

Heres a line from my results list:

Before WU was resent:
6906875 1643286 28 Jul 2005 20:14:59 UTC 4 Aug 2005 20:14:59 UTC In Progress Unknown New

After WU was resent:
6906875 1643286 28 Jul 2005 20:22:29 UTC 4 Aug 2005 20:22:29 UTC In Progress Unknown New

[EDIT]
The original deadline was like 28 JUL 2005 14:42:32. I noticed after testing that the deadline showed one week to the second after the re-download. Tried again, but first saved off the "old" deadline as reported by the server.

Walt

Walt,

Good catch -- I'm going to have to change your status to 'Developer'!!

Now that you point this out it's obvious that this is how our code works. But it wasn't what I intended. I'll have to fix it, else results will never time out for misconfigured hosts that never get the work.

Any reason that I shouldn't fix this?

[EDIT 10 minutes later]
Walt, I've fixed this. Now when results are resent the 'sent_time' and 'report_deadline' in the database are left unchanged.

[EDIT 5 minutes later]
I wonder if I should update 'sent_time' but NOT 'report_deadline'. This way the result will still time out OK but it'll be obvious from the database that it has been re-sent one or more times. Thoughts??

I just got a huge batch of these lost results resent to my host.

Now I'm a kind of dilemma wether I like the deadline being reset (I got them before Bruce changed the behaviour). Originally they were due in 2 to 7 hours after they were resent so they would time out anyway (as did some 50). Now I have opportunity to crunch them down. Some of them will time-out anyhow as I have about 12-days worth of them ...
Perhaps I'm better off just to abort them?

As to why I have so many: BOINC started to misbehave about a week ago. Eventually I detached the project and re-attached. Host got a new Id ... so far so good. Then I obviously made a mistake to merge the two records together.
____________
Metod ...

Perhaps I'm better off just to abort them?

Perhaps I'm better off just to abort them?

My heart was bleeding when I saw the pile of WUs ... I aborted all that already have quorum and validated.

The pile is a bit shorter now. I just have to wait for BOINC to report them back to the server as aborted.

I've made an additional change as Walt and I discussed.

For results that are re-sent, the REPORT DEADLINE is left unchanged. However I update the SENT TIME when the result is reset. Thus if

(REPORT_DEADLINE-SENT_TIME) is less than 7 days

it means that the work was resent one or more times.
____________

I haven't had any resent to me (I think) but from looking at the posts here and the change notes I think there is a check missing. If the project is reset causeing the lost workunits they should not be resent. Probably this should apply to merged hosts as well.

I think this needs to be added because in either of those cases there was a problem that may have even been caused by the workunit that is being resent. If so that workunit will most likely cause the same problem again and we get into a cycle of resetting and resending.
____________
BOINC WIKI

BOINCing since 2002/12/8

I haven't had any resent to me (I think) but from looking at the posts here and the change notes I think there is a check missing. If the project is reset causeing the lost workunits they should not be resent. Probably this should apply to merged hosts as well.

I think this needs to be added because in either of those cases there was a problem that may have even been caused by the workunit that is being resent. If so that workunit will most likely cause the same problem again and we get into a cycle of resetting and resending.

I haven't had any resent to me (I think) but from looking at the posts here and the change notes I think there is a check missing. If the project is reset causeing the lost workunits they should not be resent. Probably this should apply to merged hosts as well.

I think this needs to be added because in either of those cases there was a problem that may have even been caused by the workunit that is being resent. If so that workunit will most likely cause the same problem again and we get into a cycle of resetting and resending.

I see the point. But I'm not sure about this. After all a user can always ABORT a workunit that is problematic, to get rid of it.

I haven't had any resent to me (I think) but from looking at the posts here and the change notes I think there is a check missing. If the project is reset causeing the lost workunits they should not be resent. Probably this should apply to merged hosts as well.

I think this needs to be added because in either of those cases there was a problem that may have even been caused by the workunit that is being resent. If so that workunit will most likely cause the same problem again and we get into a cycle of resetting and resending.

I see the point. But I'm not sure about this. After all a user can always ABORT a workunit that is problematic, to get rid of it.

Even if (s)he doesn't abort it (not everyone babysits their BOINC installations), it will eventually pass the dead-line iff it is not re-set after WU is re-sent.

I just got a pile of these on one of my hosts. However, the deadline is set to tomorrow. I'm not sure how 48-70 hours of work is supposed to get done in 36 hours or so. Shouldn't the deadlines be reset on any of these resent units such that the host has a chance of catching up?
____________

I just got a pile of these on one of my hosts. However, the deadline is set to tomorrow. I'm not sure how 48-70 hours of work is supposed to get done in 36 hours or so. Shouldn't the deadlines be reset on any of these resent units such that the host has a chance of catching up?

I suggest that you abort the workunits which can't be finished in time. Then do 'update project' to report the aborted WU to the server. This way, new WU can be issued and your computer won't spend a long time doing work that's overdue.

Any idea how this work got lost??

By the way, this new machine has CC 4.45, and that's the only client it's ever had.
____________

Any idea how this work got lost??

On a slightly different topic: is it possible that the DL server has slight problems from time to time?

Just today I installed BOINC on another cruncher and attached to E@H project. It downloaded all the needed files fine except for science app (exe and pdb). Due to that it trashed two WUs. Next try yielded in assigning two more WUs and DLing exe fine, but DLing pdf file failed, therefore trashing another two Wus. The pdb file transferred fine just a moment later but at that time, that host used up it's daily quota (4 as it is a new host) leaving it without E@H work until tomorrow.
____________
Metod ...

I just looked over my second host that got these units, and realized it had 8 WU's all due in 7 hours. I ended up aborting all but the currently executing unit, since none of them will finish on time.

If you're going to resend these ghost units to their original hosts, there either needs to be a much longer deadline, or a throttle on how many get sent. Otherwise, you're just causing more missed deadlines and aborted units.

How about just marking the ghost units as aborted/comp error/not returned/etc, and then throwing them back in the queue for the next user to pick up in the normal course of business? I know other projects resubmit units that for whatever reason never got a quorum. Shouldn't this be handled the same way?
____________

Then reading through this tread one suggestion comes to my mind. If results already have quorum and validated, is there a point in resending that result? Isn't it better to automatically mark those results with an error so the result can be removed from the database faster.
____________
Then you're really interested in a subject, there is no way to avoid it. You have to read the Manual.

All these close deadlines may not be the norm. Since this feature has just been turned on, it is resending all the stuff that has been setting there for a while. In future the ghost work units should generally get resent well before the deadline; unless the host has been disconnected from the internet for some reason.

But I was thinking if a host should ask for xxxxx seconds of work, the server should subtract the amount of time of any ghost units and resend them. Then only send new work if the ghost units total time is less then that which was requested.

For Example:
Request 20,000 seconds of work. Have 5,000 seconds of ghost units. You would only need to send 15,000 seconds of new work.
____________


You like Myst? Uru Live returns! www.urulive.com

The deduction of a resend from the pending work makes sense. I'd also like to see the deadline recalculated from TODAY, for another 7 days. Lastly, I'd like it not to resend them if the client is already at capacity. Just wait on them until there's room, or mark them out as errored and resend to another client if need be.

____________

The deduction of a resend from the pending work makes sense. I'd also like to see the deadline recalculated from TODAY, for another 7 days.

Lastly, I'd like it not to resend them if the client is already at capacity. Just wait on them until there's room, or mark them out as errored and resend to another client if need be.

Two WUs resent were added to two WUs in "Ready to run" by which I experience two bad things:

1. The task "No new work" for Project was ignored.
2. The deadline of the WUs resent are earlier by one day than that of the WUs in "Ready to run." 4WUs in total must be finished within four days. (Besides EAH, I also participate in SAH and PAH with 1 pc)

Please report good and/or bad experiences with this feature in this thread.
Bruce

Please report good and/or bad experiences with this feature in this thread.
Bruce

Juli 26th 2005, I posted a thread to report about 'Spooky WU's.'
These 16 missing 'Ghost WU's' were resent a couple days later.
First of all, I never asked for 16 WU's; Since I am running 5 different Boinc-projects on my computer the "connect every X days" is set to 0.1 days.
Anyway, I got them resent and started running them. Endlessly:

I get the following error on a resent Ghost WU:
30/07/2005 13:30:27|Einstein@Home|Result l1_1480.5__1480.5_0.1_T00_S4lA_1 exited with zero status but no 'finished' file
30/07/2005 13:30:27|Einstein@Home|If this happens repeatedly you may need to reset the project.
30/07/2005 13:30:27||request_reschedule_cpus: process exited
This WU has been running for over 20 hours now (even though the Boinc-manager contradicts, and claims that CPU time is only 9 ours.)

I tend to abort all 16 resent Ghost-WU's.
Somehow this feels like a waste of time and effort.

The deduction of a resend from the pending work makes sense. I'd also like to see the deadline recalculated from TODAY, for another 7 days.

This would have a bad consequence. A host which had a proxy problem and never received a work unit, but which kept contacting the scheduler, would cause that workunit to never finish.

Lastly, I'd like it not to resend them if the client is already at capacity. Just wait on them until there's room, or mark them out as errored and resend to another client if need be.

I don't know how to make this determination.

However I have just made the following changes. IF
- Work within 25% of deadline (42 hours for Einstein@Home), OR
- Work no longer needed (Canonical result already exists), OR
- Work unit has error flag set (something wrong), THEN
the scheduler no longer resends the workunit, but instead marks it as timed out in the database. The scheduler will then send an informational message to the client reporting that this WU has been 'expired'.

I'll test this over the next few hours, and see if it has undesirable side effects.

Bruce

got no work and every time i try to update i get this message group

01/08/2005 12:10:32|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:33|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:33|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:34|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:34|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:35|Einstein@Home|Unrecoverable error for result l1_1083.0__1083.1_0.1_T12_S4lA_2 (CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20))
01/08/2005 12:10:35||request_reschedule_cpus: start failed
01/08/2005 12:10:35||request_reschedule_cpus: process exit

anyone help ?

already tried resetting


____________

Based on the feedback in this forum, I've made some additional modifications to the scheduler policy on resending lost workunits. Details may be found here:
deadline_proposal.txt. This extends the deadlines (up to a total of an additional week) for machines that did not get the work when it was originally sent.

Bruce
____________

got no work and every time i try to update i get this message group

01/08/2005 12:10:32|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:33|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:33|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:34|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:34|Einstein@Home|CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20)
01/08/2005 12:10:35|Einstein@Home|Unrecoverable error for result l1_1083.0__1083.1_0.1_T12_S4lA_2 (CreateProcess() failed - The process cannot access the file because it is being used by another process. (0x20))
01/08/2005 12:10:35||request_reschedule_cpus: start failed
01/08/2005 12:10:35||request_reschedule_cpus: process exit

anyone help ?

already tried resetting

I can't be sure if the problem I am experiencing is related to the scheduler changes or not. I have experienced an inundation of workunits on a machine that seems to have this problem with regularity. My post "Overcommitted"
in Problems and Bug reports goes into the background. Overcommitted

Since that time I have removed the application and the boinc directory and reinstalled boinc 4.45.

Machine # 383671 has currently 47 work units and it appears that several old ones were submitted per this lost work unit update. The units at the top of the list are due Aug 8 and are preempted because the machine received a large batch of units dated Aug 5, Aug 6, and Aug 7. I aborted the 8/5 units since they would not get credit (today is Aug 5).

I'm open to suggestions. Since I have 3 other machines identical to this configuration that operate normally, I may ghost one of their drive images on to this machine and replace BOINC for another attempt at a clean install.
____________

I can't be sure if the problem I am experiencing is related to the scheduler changes or not. I have experienced an inundation of workunits on a machine that seems to have this problem with regularity. My post "Overcommitted"
in Problems and Bug reports goes into the background. Overcommitted

Since that time I have removed the application and the boinc directory and reinstalled boinc 4.45.

Machine # 383671 has currently 47 work units and it appears that several old ones were submitted per this lost work unit update. The units at the top of the list are due Aug 8 and are preempted because the machine received a large batch of units dated Aug 5, Aug 6, and Aug 7. I aborted the 8/5 units since they would not get credit (today is Aug 5).

I'm open to suggestions. Since I have 3 other machines identical to this configuration that operate normally, I may ghost one of their drive images on to this machine and replace BOINC for another attempt at a clean install.

Walt, Thanks! your advise qualifies as "sticky". I did merge the host after the last reinstall so that is likely what caused all the lost units coming back. For now, I have the "no new work" button selected and I will trim the work que to an achievable level. When the que runs down, I will see if this machine contines to over commit.
____________

Could this patch have any side effects on the upload handler?

I can still receive work with a 4.19 on Linux going through a squid proxy with PW - but I cannot deliver results anymore.

resultid=7800129 ist stuck for quite some time now, upload always gives me a -127, temporarily failed upload.

As downloads still work, it cannot be a proxy or rights problem, all files belong to the user running BOINC, uploads did work before, I can "wget" the upload handler reply so there's nothing blocked either.

BOINC runs with "-return_results_immediately" because it is a slow machine.

I'm not 100% sure but windows clients seem not to be affected (I can check that on Monday)

Interesting effect : when the next result was ready, it uploaded the one that was stuck with -127 before without any trouble - now the new one (resultid=8035146) keeps giving me -127.

[color=dark blue]Please refer to post at:
http://einstein.phys.uwm.edu/forum_thread.php?id=2789

I have no work units for one machine.[/color]
____________

I have read this thread with interest. I have been trying to figure out what is going on with my results for some time. It seems as if thing go fine for a few days and then all of the sudden I get a large "patch" of results that I have never seen on my machine, and never get completed. It is almost as if they were never sent to me at all but are now in my results list. I know my machine does not request work often enough to receive that many WUs.

I have just discovered that WUs may be delivered in "packets", and that the system draws on these packets for a local supply of WUs. If this is true, I wonder if what I am seeing could be the result of these packets being too large, and containing WUs that have too short deadlines. for the system to complete in the time allotted?

In any case I would like to have some idea how to prevent these large gaps of incomplete WUs that I have never seen from developing in the first place. My results are here - http://einstein.phys.uwm.edu/results.php?userid=97084

Regards
Phil

____________

We Must look for intelligent life on other planets as,
it is becoming increasingly apparent we will not find any on our own.

...
In any case I would like to have some idea how to prevent these large gaps of incomplete WUs that I have never seen from developing in the first place. My results are here - http://einstein.phys.uwm.edu/results.php?userid=97084

Regards
Phil

...
In any case I would like to have some idea how to prevent these large gaps of incomplete WUs that I have never seen from developing in the first place. My results are here - http://einstein.phys.uwm.edu/results.php?userid=97084

Regards
Phil

Try upgrading your BOINC client to 4.45 from http://boinc.berkeley.edu/download.php. This should fix the problem (details in this thread), and also download those results which you are currently missing.

I have been running 4.45 since I first installed BOINC.
...

I have been running 4.45 since I first installed BOINC.
...

Are you sure about that? Forgive the arrogance, but unless I can't read the results page properly, you are using version 4.43. (example http://einstein.phys.uwm.edu/result.php?resultid=8290642 under stderr out) I must retract my advice, however, as there is no version 4.45 for a Mac available for download (4.43 is the latest release version). I am afraid I can't comment on what will happen with the alpha release of BOINC, but I'd stick with the regular release version if I were you, as the ghost WU don't cause significant harm.

Nick

I am experiencing the same problem with my Mac as well (running 4.43). It is not isolated to Einstein either. I'm seeing similar behavior with ProteinPredictor as well. I'm hoping this does get fixed because I have run into situations where I can't retrieve more work units since I have met my daily quota (in this case all phantom units). This also unfortunately results in delayed project results since those work units will not be resent until after the deadline is past. I have not been able to get in a situation where I can get these lost results sent to me and I am not sure how one accomplishes that (i've done the reset, detach,etc. routines with no luck).

I am experiencing the same problem with my Mac as well (running 4.43). It is not isolated to Einstein either. I'm seeing similar behavior with ProteinPredictor as well. I'm hoping this does get fixed because I have run into situations where I can't retrieve more work units since I have met my daily quota (in this case all phantom units). This also unfortunately results in delayed project results since those work units will not be resent until after the deadline is past. I have not been able to get in a situation where I can get these lost results sent to me and I am not sure how one accomplishes that (i've done the reset, detach,etc. routines with no luck).