Project downtime

Oliver Behnke
Oliver Behnke
Moderator
Administrator
Joined: 4 Sep 07
Posts: 949
Credit: 25167626
RAC: 7

RE: Time for beer! I

Quote:
Time for beer!

I updated 19 servers and gateways in two data centers in the past seven hours, fixing the Ghost CVE, upgraded kernels, updated our master/slave databases and deployed pending package updates - so yes, for sure, time for two beers!

Oliver :-)

 

Einstein@Home Project

Holmis
Joined: 4 Jan 05
Posts: 1118
Credit: 1055935564
RAC: 0

RE: RE: Time for beer! I

Quote:
Quote:
Time for beer!

I updated 19 servers and gateways in two data centers in the past seven hours, fixing the Ghost CVE, upgraded kernels, updated our master/slave databases and deployed pending package updates - so yes, for sure, time for two beers!

Oliver :-)


Then you've really earned it! Cheers!

Phil
Phil
Joined: 8 Jun 14
Posts: 579
Credit: 228493502
RAC: 0

Just in the nick of time.

Just in the nick of time. Finished building a new cruncher this afternoon.

Thanks Oliver.

Phil

Mike Hewson
Mike Hewson
Moderator
Joined: 1 Dec 05
Posts: 6543
Credit: 287356645
RAC: 76962

What service ! I slept right

What service ! I slept right through that. Is it Beer-O-Clock already ? :-)

Cheers, Mike.

I have made this letter longer than usual because I lack the time to make it shorter ...

... and my other CPU is a Ryzen 5950X :-) Blaise Pascal

Pooh Bear 27
Pooh Bear 27
Joined: 20 Mar 05
Posts: 1376
Credit: 20312671
RAC: 0

RE: Ok, it was indeed a

Quote:
Ok, it was indeed a missing firewall rule. For some reason it wasn't yet made persistent and thus got lost due to the system reboot.


Mr. Oliver,
Those firewalls will get you every time if you do not push the policy. I can recall several times losing data because a push was missed, or the CMA lost sync and the primary went down and someone pushed old policy on the secondary losing months worth of changes. I loved reading the last couple of outage statics from your team. It reminds me of what I have gone through in over 30 years of IT. The hardest part is always the customer barking to get it fixed, when you are busy trying to do just that. Lots of extraneous noise.

I am glad to see your team was very professional during the whole time. Makes me proud to crunch for this project. Some admins may have blown up at one point or another during the heat of the moment, but the team there kept their cool to the customers, even though you were probably all overtired and worn out during a good portion of the repairing the issues. My hats off to you and the team.

Oliver Behnke
Oliver Behnke
Moderator
Administrator
Joined: 4 Sep 07
Posts: 949
Credit: 25167626
RAC: 7

Thanks guys. We appreciate

Thanks guys. We appreciate that.

Oliver

 

Einstein@Home Project

MAGIC Quantum Mechanic
MAGIC Quantum M...
Joined: 18 Jan 05
Posts: 1710
Credit: 1081886589
RAC: 1122430

Never skipped a beat here


Never skipped a beat here Oliver ( PST )

AllparDave
Joined: 7 Jan 15
Posts: 8
Credit: 171011
RAC: 0

Am I the only person not

Am I the only person not getting new tasks since yesterday afternoon? Communication works but says "0 new tasks."

Fri Feb 20 13:02:05 2015 | Einstein@Home | Scheduler request completed: got 0 new tasks

AllparDave
Joined: 7 Jan 15
Posts: 8
Credit: 171011
RAC: 0

Just got two jobs…out of

Just got two jobs…out of eight….

AllparDave
Joined: 7 Jan 15
Posts: 8
Credit: 171011
RAC: 0

Back to normal. Sorry for

Back to normal. Sorry for hijacking.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.