GitLab Outage

 (status.gitlab.com)

50 points | by kunley 2 hours ago

8 comments

  • davb 2 hours ago
    Coincidentally (?) we’ve seen a huge uptick in failed login attempts to our MFA-protected admin accounts on our self-hosted Gitlab CE instance today. They’re coming from thousands of IPs across hundreds of networks (VPS providers and, apparently, residential proxies).
    [-]
    • deckar01 1 hour ago
      There was a relative lull in GitHub bot activity the last few weeks, but they seem to be back today. I assume the promo token flood gates opened for the new models.
      [-]
      • physicallyIllfr 1 hour ago
        I dont think bots need bonus/trials. There are lots and lots of ways to get free, or virtually free tokens. Hell half the vibe coded AI chrome extensions send store their openrouter keys client side.
    • TZubiri 25 minutes ago
      It's probably related to the issue published by aikidolabs where leaked per-user gitlab issued emails were being used as a sort of credential.

      https://www.aikido.dev/blog/gitlab-email-push-to-main

  • delf 52 minutes ago
    You can use GitSocial to mirror to your own bucket for situations like this: `gitsocial mirror https://gitlab.com/owner/repo s3://s3.example.com/mybucket/repo`

    https://gitsocial.org/

  • b0afc375b5 2 hours ago
    Interesting that the status page was updated immediately. Is this the norm now? I remember back in the day when status pages showed all green even when servers were clearly down for a long time already.
    [-]
    • stack_framer 2 hours ago
      It was not updated immediately. I was seeing 503s for a solid 15 minutes while the status page was all green. This is also their second outage today:

      https://status.gitlab.com/pages/history/5b36dc6502d06804c083...

      [-]
      • nicholasjarnold 2 hours ago
        Yep, status page was certainly all green when I started noticing it.
    • seabrookmx 2 hours ago
      As others mentioned, it wasn't that quick.

      It's also not accurate. After nearly a half hour the page shows "git operations" as green but I'm not able to push/pull from my repos with SSH or HTTPS (and haven't been able to the whole time).

    • thesh4d0w 2 hours ago
      It took at least 5 minutes, closer to 10. I think the outage is so bad, it got pushed fast.
  • ChrisArchitect 15 minutes ago
    Incident link(s):

    https://status.gitlab.com/pages/incident/5b36dc6502d06804c08...

    https://status.gitlab.com/pages/incident/5b36dc6502d06804c08...

  • lampcord 50 minutes ago
    Another day, another reminder why we self-host our GitLab instance. Cloud convenience always has these kinds of trade-offs.
    [-]
    • zer0x4d 43 minutes ago
      We self host ours too. 100% uptime ;)
      [-]
      • TZubiri 17 minutes ago
        what host are you using that gets 100%uptime? A hyperscaler like AWS?
  • nabeards 2 hours ago
    Just went to push code and found this out.
    [-]
    • nicholasjarnold 2 hours ago
      Code review for me... Hit the 503. The status page lagged the event by ~10 mins or so it seems.

      git+ssh also seems affected, as I'm unable to push/pull right now.

  • smalltorch 2 hours ago
    Gasp! immediately builds a complete git alternative to never expirence a single second of downtime ever again
    [-]
    • JonChesterfield 1 hour ago
      Well, git is already distributed, so you could use _any other computer_ instead of gitlab.com...
      [-]
      • smalltorch 1 hour ago
        I'm only kidding, although it could be a good excersise to learn. I think a pure TUI git front end / back end project would be fun to make.
      • gblosser 26 minutes ago
        or go back to Mercurial...

        Do you remember Mercurial?

        Pepperidge Farm remembers.

  • lemonish97 59 minutes ago
    Right, on to another alternative now