I used to host a Gitlab instance at work. It was dog slow so I started digging into it and discovered they had a serious memory leak in some of their “unicorns,” aka Ruby tasks. Instead of fixing the source of the leak they tacked on a “unicorn killer” that periodically killed tasks. The tasks were supposed to be atomic anyway, so this is technically fine (and maybe a good thing in the long run for correctness a la Netflix’s Chaos Monkey) but I found myself kind of disgusted by the solution. I dropped it and went for a much sparser Git repo web server.
I don’t think memory leaks could ever amount to a security vulnerability, but it just feels yucky. I guess I shouldn’t cast stones, I write C++ at work.
I don’t think memory leaks could ever amount to a security vulnerability
In theory it could, after all there are technically denial-of-service vulnerabilities (not DoS/DDoS attacks, that is something different) according to CVE Numbering Athorities.
According to the CVE Numbering Athorities, there can be vulnerabilities that result in service being denied, and they refer to them as a denial-of-service vulnerability. For example, there can be a bug in a program that causes it to crash if you perform a certain set of steps/actions, thus resulting in the service being denied. Whereas traditionally, a DoS/DDoS attack is simply flooding a target with more bandwidth than they have available downstream bandwidth. Sending massive amounts of data to overwhelm a service is not the same thing as finding a unique set of actions to cause the program to crash.
So in theory, yes, a memory leak could amount to and result in a security vulnerability, like if the memory leak is reproducible and so severe it causes a service to crash.
Aha, I didn’t realize compromising availability was sufficient for the CVE definition of security vulnerability. Projects I’ve worked on have typically excluded availability, though that may not be the norm.
And I see your point about some exploits being highly asymmetric in the attacker’s favor, compared to classic [D]DoS.
They do have a ton of tests actually. In their defence, if this task is doing Git things then just killing it when it goes badly is probably the best you can do. Git itself is quite buggy if you stray from the most basic setup. I’ve had it almost completely destroy my .git directory in the past when using submodules.
On the other hand, Gitlab itself is an enormous entirely untyped Ruby monster, with extremely difficult to follow code. Not in terms of individual functions - except for the lack of types mean you can’t really know what they do, they are quite clear and well written. The issue is the control flow between parts of the system. It’s difficult to know what calls what, so I’m not surprised they occasionally have to give up.
I had a play with Deno’s Fresh web framework recently (Typescript/TSX but mainly server rendered). IMO it’s light years ahead of other solutions.
You get full amazing Typescript typing, including in templates (unlike Go for example), but unlike React you don’t have to deal with JavaScript tooling or complex client side state management. It’s a real breath of fresh air. (Ha that wasn’t even intentional.)
I used to host a Gitlab instance at work. It was dog slow so I started digging into it and discovered they had a serious memory leak in some of their “unicorns,” aka Ruby tasks. Instead of fixing the source of the leak they tacked on a “unicorn killer” that periodically killed tasks. The tasks were supposed to be atomic anyway, so this is technically fine (and maybe a good thing in the long run for correctness a la Netflix’s Chaos Monkey) but I found myself kind of disgusted by the solution. I dropped it and went for a much sparser Git repo web server.
lmao! Man that’s hilarious!
“We have a memory leak that could lead to a security issue.We should do something about it.”
“I made a process that periodically kills those tasks. No one will notice the problem now.”
The unicorn killer will have a memory leak as well. 💀
I had to look it up to check my memory. Yup! https://about.gitlab.com/blog/2015/06/05/how-gitlab-uses-unicorn-and-unicorn-worker-killer/
I don’t think memory leaks could ever amount to a security vulnerability, but it just feels yucky. I guess I shouldn’t cast stones, I write C++ at work.
In theory it could, after all there are technically denial-of-service vulnerabilities (not DoS/DDoS attacks, that is something different) according to CVE Numbering Athorities.
Maybe I’m misunderstanding you, but DoS is exactly the same thing as “denial of service”.
My point is that memory leaks can only degrade availability; they are categorically distinct from security vulnerabilities.
I think you might be misunderstanding me.
According to the CVE Numbering Athorities, there can be vulnerabilities that result in service being denied, and they refer to them as a denial-of-service vulnerability. For example, there can be a bug in a program that causes it to crash if you perform a certain set of steps/actions, thus resulting in the service being denied. Whereas traditionally, a DoS/DDoS attack is simply flooding a target with more bandwidth than they have available downstream bandwidth. Sending massive amounts of data to overwhelm a service is not the same thing as finding a unique set of actions to cause the program to crash.
So in theory, yes, a memory leak could amount to and result in a security vulnerability, like if the memory leak is reproducible and so severe it causes a service to crash.
Aha, I didn’t realize compromising availability was sufficient for the CVE definition of security vulnerability. Projects I’ve worked on have typically excluded availability, though that may not be the norm.
And I see your point about some exploits being highly asymmetric in the attacker’s favor, compared to classic [D]DoS.
That’s disappointing. They are pretty consistently choosing the wrong thing. I don’t think they know what they’re doing.
Unicorn killer does sound great for testing. If they wrote tests around anything I’d be surprised, though. LOL.
If you don’t need all the user management and whatever else it definitely doesn’t make sense to run their junk.
They do have a ton of tests actually. In their defence, if this task is doing Git things then just killing it when it goes badly is probably the best you can do. Git itself is quite buggy if you stray from the most basic setup. I’ve had it almost completely destroy my .git directory in the past when using submodules.
On the other hand, Gitlab itself is an enormous entirely untyped Ruby monster, with extremely difficult to follow code. Not in terms of individual functions - except for the lack of types mean you can’t really know what they do, they are quite clear and well written. The issue is the control flow between parts of the system. It’s difficult to know what calls what, so I’m not surprised they occasionally have to give up.
I had a play with Deno’s Fresh web framework recently (Typescript/TSX but mainly server rendered). IMO it’s light years ahead of other solutions.
You get full amazing Typescript typing, including in templates (unlike Go for example), but unlike React you don’t have to deal with JavaScript tooling or complex client side state management. It’s a real breath of fresh air. (Ha that wasn’t even intentional.)