Conversation

Hello everyone,

All the services we offer are down and unreachable. We are currently investigating the cause. For the time being, it appears that all three servers are without power.

25
6
0

@codebergstatus Let me guess, the datacenter is testing its power failover. Can you tell I've been through that before?

0
1
0

@codebergstatus

A site that big runs on only three servers?? I'm impressed!

I mean, I knew was efficient—I run an instance myself—but I didn't know it was *that* efficient.

1
0
0

@codebergstatus

well, on the bright side, if your services were down and reachable that would just be weird

0
0
0

@codebergstatus It's relatable. Sometimes I too am unresponsive when I'm without power.

<3

0
0
0

@codebergstatus thanks for the information and your services. for all working on the issue.

0
0
0

@codebergstatus Best of luck getting to the bottom of the issue! (And thanks for running an awesome community service!)

0
0
0

@codebergstatus Could possibly be the thunderstorms heading throughout Europe?

0
0
0

@codebergstatus I think I found the culprit behind the power outage. Hope you get everything detangled and back online soon!

0
0
0

@codebergstatus
sorry everyone, I just wanted to register but that might have been like the last mint leaf.
It might be not very helpful but I obtained an 500 Internal Server Error when I followed the activation link and only shortly afterwards, the connection could not be established anymore.

Might just be a result of a backend service going down slightly earlier than the frontend.

1
0
0

@codebergstatus
Dann geh' ich mal ins Bett …
Aber ihr habt ja noch lange Zeit, bis eure Uptime auch nur in die Nähe von GitHub … sinkt.
Viel Erfolg auch von mir bei der Fehlersuche.
Gute Nacht.

0
0
0
codeberg is down, post ponies (since @dpk doesn't seem to be around right now to post this)

RE: https://social.anoxinon.de/@codebergstatus/116824443563698727
0
0
0

We do not expect the issue to be resolved soon. We are waiting for a fix from the datacenter operator and will monitor the situation in order to restore service as soon as possible after power returns.

Currently, one server is with power again, but the others remain unreachable. Our investigation shows that network equipment is also affected.

6
2
0

@codebergstatus

Is this the part where we wildly speculate on the cause of the power outage?

I'll start: aliens abducted the entire power plant! They were all like, “yoink this is mine now.”

2
0
0

@argv_minus_one maybe it was a thunderstorm or the heatwave. People need to understand that climate change is here to stay and that what we took for granted can be taken away in seconds...
@codebergstatus

0
0
0

@codebergstatus Oof, thatcs tough, it happens. Sending HugOps and hoping the servers get well soon!

0
0
0

@olonam @codebergstatus

Someone found the perfect socket to plug the vacuum cleaner into.

0
0
0

@argv_minus_one @codebergstatus
Portable AC plugged into their server power strip… 😂

0
0
0

@codebergstatus I hope it gets resolved soon, and I hope it doesn't cause y'all too much stress in the meantime.

0
1
0

@codebergstatus
I think it's mostly up again :3
At least the status page is getting greener :3
Yay

2
0
0

@codebergstatus
Just slept the time Codeberg was down → good problem solving; sleep whilst the problem exists :3

0
0
0

We have recovered services after power returned, leading to a total downtime of about 7 hours.

We apologize for the impact and we are in touch with our datacenter operator to discuss the incident.

14
3
0

@codebergstatus A 7 hour outage due to power delivery failure to the servers is really a bad thing. I hope there will be a postmortem on how this was possible, and, more importantly, which steps are being taken to make sure this doesn't happen again. Anyway, thank you all for getting things working again!

0
0
0

@mauk_y @codebergstatus It's quite brave of you to admit to this, given the context ☺️

0
0
0
bro was my pr that bad
0
0
0

Someone else I know in the same DC reported short power outage plus inlet temperatures between 36c and 45c, so I hope all the equipment is fine 😬

2
0
0

@gsuberland sounds like it!

Thankful our own uplink goes through the other DC in the building (which has its own cooling et al)

0
0
0

@codebergstatus This is why everyone should have a mirror for their code. I hope the lights can get turned back on.

0
0
0

whats new bro

0
0
0

@argv_minus_one @codebergstatus
I'm running it on an N200 for me and my friends and it had less problems than many other services takes up almost nothing of the limited resources and still serves pages real fast; it is really just well optimised (and not written in JS neocat )

1
0
0

@argv_minus_one
Yes, it pulls like nothing in terms of power which is why I wanted a NUC with it for my little homelab so it doesn't get to expensive.

1
0
0

@argv_minus_one
Yes it isn't socketet, but it is much slower than most laptop CPUs but also more power efficient. It is basically what a Raspberry Pi with x86 would be; performance wise somewhere between a Pi and an average laptop processor. It just pulls like ~4W in my config + some watts for SSD und HDD as well as Mainboard and USB stuff totalling 11W with a peak of like 14W.
The N200 only has like 4 cores with each a single thread but that is fine as long as I don't compile anything on it and for Forgejo I just get the precompiled binaries :3

0
0
0

@codebergstatus

Welcome back !

Do keep us updated, would you? I'd certainly like to hear what happened, out of curiosity if nothing else.

0
0
0

@codebergstatus last time I saw a data centre have a power cut was when someone dropped a spanner in Redbus, and it was not just a power cut but a bridged phase which fried a lot of kit. Data centres don’t normally have power outages!

Best of luck.

0
0
0

@codebergstatus that’s honestly a really amazing recovery time for a power outage! Thanks for your hard work, came back up by the time I woke up!

0
0
0

@codebergstatus You'll probably won't see this Codeberg staff but thanks for the effort of keeping it up! Keep up the good work guys. <3

1
0
0

@codebergstatus It's great to see this level of transparency and I look forward to a potential post-incident report.

0
0
0

@codebergstatus thanks to all the folks on the infra team, you all are awesome! 💜

0
0
0

@codebergstatus Great news !

I'll be able to take advantage of the return to normal temperatures to get some work done this Sunday.

Thanks for everything you do.

0
0
0

@codebergstatus Maybe (just speculating) it's because the electical grid can't keep up with the demand because of the heat? Had local blackouts last week, but lasted 2h in total. Anyway, thanks for hosting my mirrors 👍

0
0
0

@micdan @codebergstatus yeah, I can only agree to that:

I only started to actively use codeberg recently and was doing stuff tonight. Suddenly, codeberg didn't work anymore, so I checked the status page and mastodon. I genuinely didn't think there would be any reaction so quickly in the night. But there it was: a new toot, made 5 min ago (with information on what happened!).

I took it as a sign to go to bed, so at least I got some sleep out of this outage.

0
0
0