r/pop_os • u/jackpot51 System76 Principal Engineer • 2d ago
COSMIC projects will no longer accept LLM-generated content in PRs
There are various reasons, and I don't want to hurt anybody's feelings. The primary reason is simply our team's load for reviews. In particular, we would like to prioritize working on contributions from our own team and regular contributors, while we have been receiving far more changes from first-time contributors using LLMs. These changes are often unplanned and have had a low acceptance rate.
The policy can be seen as part of the revised PR template here: https://github.com/pop-os/cosmic-epoch/blob/master/.github/PULL_REQUEST_TEMPLATE.md.
There is an exception for cosmic-flatpak, where each project manages its own flatpak manifest, pointing to its own code. These manifests are reviewed by our team for the correct sandboxing prior to accepting changes.
20
12
u/LatinaKaren 2d ago
Please never change, you guys are awesome!!!
It's becoming increasingly difficult to find companies that don't blindly hop on the AI hype train. đ
25
u/Frosty-Storage-9359 2d ago edited 2d ago
I once have generated code pushed during a project in university and allowing a teammate to constantly push generate code was the worst mistake ever. It looked good at the beginning, but when it become times to adjust or just correct some behaviour, it become an absolute nightmare to debug and find the code to correct. It was then I have absolutely understood why low effort generated code is a coder worst enemies. I truly do support this because I know by experience just how horrific and nightmarish it can become fast, especially since it come from someone else and just navigating is an absolute nightmare. The best prevention is to not allow it in the first place.
1
u/phobrain 1d ago
In the downhill segment of my career, people in the dev team that I did QA for were vocal in requesting my code reviews when my manager questioned the value since it was informal and PRs didn't accumulate. I had stopped counting lines of production code I'd written and supported at 640K nonblank lines, in mostly C and Fortran iirc, and it seemed like I could see inside the personality of the programmer and, Feynman-like, finger their bugs directly. Setup for my prediction that you'd need an LLM to unscramble what an LLM has wrought. Please share if you know someone who is affected.
15
19
u/NotesFromYourElf 2d ago
You just keep delivering. Thank you for being so good. Cosmic is the best desktop experience I've had, ever. Going back to win 95 era. Better than KDE, better than Gnome, better than Hyprland, and so on.
7
u/1tan_freed0m 2d ago
Yeah... It's about time.. As PM I did all from reqs to dev to publishing software myself made using AI. I'm not a professional developer, but my role was AI PM And, had to take up Dev & QA cuz the company asked to.And I can say it's like 10% good vs 90% garbage. Takes so much time to QA & check and remove the garbage. đĽ˛
9
u/mikhailvalerie 2d ago
As someone who had a bunch of PRs incorrectly closed as "LLM generated" I agree and support this stance. ^
My work requires AI usage and there's a noticeable drift that's putting more work on reviewers. AI generated stuff "works" so there's less effort into doing things "correctly" or "consistently" - the extra steps that make it easier to review and maintain the work. And that's for an internal department, I can imagine it's much worse with a public repository.
AI tools do bring a clear benefit as tools, but they also have a lot of baggage and liabilities (copyright?) with them. I can understand that the risk is not worth the reward for a small team with a big vision.
^ The COSMIC team have thankfully reopened my PRs after my confirmation - which I'm grateful for. I'll be updating them to use their new declaration template.
7
u/mmstick Desktop Engineer 2d ago
There will likely be some accidental closures as we clear out our backlog of generated PRs.
4
u/mikhailvalerie 2d ago
I'm okay with that. I'm on the other side of the internet; you have no idea who (or what) I am, so a judgement call was made to suit your team.
If I was a drive-by submittor then closing unsupported PRs was the right call.
When I responded you quickly reopened the PRs and I'm very grateful for that.
3
u/crocodus 1d ago
Iâll be honest and say that I never really cared about PopOS and I also never really cared about Rust. But considering the current AI slop epidemic this is a breath of fresh air, and Iâll treat you guys to a beer.
I love some god damned good ass engineering, and Iâm more than happy to put my money where my mouth is. Cheers!
2
u/Automatic_Friend3744 1d ago
cool! does system76 themselves push LLM code? i'm sure it's low priority, but if not it would be a good listing for open-slopware :)
23
u/mmstick Desktop Engineer 1d ago edited 4h ago
No, we have never needed to use LLMs. You can make a lot of progress with Rust in a short time with a small team of talented developers. Everyone on the team has many years of experience with Rust and many of us have been with Rust since the early Rust 1.0 days.
The entirety of COSMIC was built from the ground up within three years, which would not be possible without the human talent which no LLM can replace. So when I see comments about "falling behind", or "there won't be any new contributors without LLMs", these are very unserious comments vastly underestimating human capabilities.
You need only look at how quickly the open source Rust ecosystem was able to iterate with Rust since the 1.0 release in 2015; despite the lack of mature libraries to build with and the initial ergonomics issues that made the borrow checker 10x harder. Rust will continue to allow humans to accelerate software development like never before, regardless of some trying to attribute all of Rust's productivity gains recently to LLMs.
3
0
u/gajop 1d ago
I mean, I've been writing C++ since I was in high school some 20+y ago and I'd rather have AI write it. Same for Rust, Python and other languages. Are you seriously claiming productivity is the reason you're not using LLMs, like at all?
You do you, but I'd rather make an ideological/moral claim than productivity - for sound argument sake.
5
u/mmstick Desktop Engineer 16h ago edited 4h ago
It would be easy to argue from a point of a moral superiority but that would imply that there is no practical reason for us to avoid using a LLM. Productivity is often cited as the reason to use a LLM but we are already productive enough because of our Rust/Linux experience. I've been writing Rust code for 12 years to date so I've never had the personal motivation to start using it.
For a project like COSMIC, there is a lot of greenfield work in areas where there is a very high risk of regression so it is imperative that we maintain a high degree of quality in our output. Maintaining quality is more important than brute forcing quantity, which is part of the reason why we've been using Rust for all of these years.
Rust was designed and optimized for humans. We excel at reasoning but we make mistakes occasionally that the type system and borrow checker are able to guard against at compile-time. Memory and thread safety vulnerabilities are the costliest and most common of these mistakes which Rust is able to completely eliminate with its static code analysis.
The memory safety vulnerabilities are the root cause of most of the security issues that LLMs are discovering in the Linux kernel. So while I can understand the Linux kernel needing LLMs to discover hidden memory safety issues in their C code, building software with Rust from the start prevents them from ever reaching your code. Quality is maintained because the Rust compiler forces us manage our memory and threads correctly. And as long as we write the code by hand, we will continue to be able to understand our entire codebase.
LLMs are great at making predictions (which was their original purpose) but they are unable to reason. So there is a very high risk of regression if the person at the prompt providing the reasoning loses their ability to reason about the code it generates. MIT recently published a study where using a LLM to assist with writing assignments caused measurable cognitive decline within four months. The EEG scans for those who used LLMs for assisting their writing tasks had significantly reduced brain connectivity compared to those who did not. This is not what you want to happen to yourself when the quality of your work depends on your ability to think.
I've seen a lot of comments on social media from other Rust developers that are losing their ability to write code without a LLM after using LLMs for a while. So I personally don't want to risk that. Social media already gives enough brain damage.
2
u/gajop 16h ago
I think if you use LLMs to generate code, you will certainly lose the ability to write code by hand as well. That is not a big problem and I don't think you're arguing it either.
However I do agree with you that there's a risk in both the quality of the output to drop, and more importantly, your understanding of the project as well. Perhaps you also become stupider, but I'm not sure about that.
If you're worried about that, you would indeed need to be very disciplined in your use, and it's very easy to decide to vibecode just this one little feature because the deadline is pushing or you want to see it deployed asap... So I can agree with your worry there, it's a slippery slope.
I am personally not as worried. I think things have been going pretty well overall in both my personal and work projects (it's thanks to AI that I'm able to introduce Rust to work for example), and while there are drops in understanding here and there, those can be (and have been) recovered, and usually, the alternative would've been month long delays at work or often just a lack of will to start doing hobby projects after long work days.
One very nice aspect of it, it's very easy to get myself to work on a hobby project with AI, the starting barrier is small as opposed to having to spent half an hour just getting started trying to remember where you left off. Once you start, you can always do less vibing and more engineering, much easier if you're in the seat already. Maybe not something you want for COSMIC, dunno.
2
u/ExistingSelection180 14h ago
I think you hit on a crucial point: understanding the code you're writing. You can develop an applet without fully understanding everything behind the scenes, but when youâre building the foundations of an operating systemâin this case, Cosmicâwhich controls everything and interacts with everything, itâs vital that you understand the code, and thereâs no other way to do that except by writing it yourself. LLM might help with secondary tasks (such as tools), but the code must be written, audited, and reviewed by someone who understands the entire process.
It's the lesson of quantity versus quality. And in our case, having the least amount of efficient code is best so that it can interact with other components of the PC and other code. In this case, predictably.
The vast majority of peopleâmyself includedâdon't fully grasp this paradox of AI use. In design, it helps me review drafts and brainstorm ideas, but it will never deliver a polished, optimized design.
2
u/gplgang 8h ago
"there is a lot of greenfield work in areas where there is a very high risk of regression so it is imperative that we maintain a high degree of quality in our output."
This has been what keeps me from using LLMs much in my compiler. They are excellent at locations and explaining what caused a bug, spotting small issues I miss but would have to debug, and giving library info directly in the context where I need it.
Almost every time I've let one touch some core thing connected to a lot of invariants of state in the compiler, it will get something subtly wrong I never would have working through it myself. It ends up costing time
A good example was a AST visitor with early termination, very common and routine to implement. It just didn't have the ability to reason about specifically what some of the invariants of the tree are and how it should do termination. It worked well until edge cases hit and I had to rewrite it
-2
u/Dev-in-the-Bm 1d ago
I assume for ideological reasons?
4
u/mrtruthiness 16h ago
No. It's to save developer time. They've been overloaded with poor quality PRs. Did you read the link???
There are various reasons, and I don't want to hurt anybody's feelings. The primary reason is simply our team's load for reviews. In particular, we would like to prioritize working on contributions from our own team and regular contributors, while we have been receiving far more changes from first-time contributors using LLMs. These changes are often unplanned and have had a low acceptance rate.
1
u/Dev-in-the-Bm 13h ago
I was talking about your core team using LLMs.
From what I'm hearing from many I know, if you do it right, commanding the implmentation, and reviewing all generated code, it can be a real productivity multiplier, allowing to ship more faster, without having to deal with slop.
1
u/mrtruthiness 12h ago
I see.
I don't think they prohibit their own devs from using AI tools. There's quite a difference between having the AI generate code vs. AI used for debugging or AI used for code refactoring.
2
u/Easy-Truck4725 13h ago
I don't want to hurt anybody's feelings
LLM boosters don't have feelings, you're fine to say what you want
1
u/Individual-Angle1311 2d ago
As a software engineer, I find it really interesting to see how different open source projects react and adapt to the evolving world of LLM-crazed software industry.
1
5
1
-3
u/Status-Afternoon-425 2d ago
I think what I missing here is this. Open source is not a product it's a process. In most cases people participate because they love craft. It's a huge difference from corporate software. Where main focus is on customers and on features. It will be interesting to see how it all converge.
1
-5
u/MaxxxV 1d ago
Please don't forget many contributors spend hours if not days to fix issues themselfs in a software project they love and use on a daily basis.Â
Closing their work that was made possible with AI may make sense to the team. There is also a community member with positive energy behind this.Â
5
u/Careless_Subject8377 1d ago
If you don't know how to cook, you shouldn't be trying to make a holiday feast.
If you don't know how to write code, you shouldn't be trying to contribute to projects.3
u/Ghost_x_Knight 1d ago
Seems targeted toward vibe code, where the contributor never learned how to understand the code they are submitting, or developed the skill/knowledge to potentially write an equivalent code themself.
2
u/vancha113 8h ago
Pretty sure denying these pull requests is not at all personal, just reactionary to an amount of pull requests that's too much for the size of their team, at least that's how it comes across. If this solution solves that problem, and gets rid of generally (not always) the more low-quality submissions, that sounds like a win, but I definitely wouldn't doubt the positive energy is appreciated.
-5
u/MaxxxV 1d ago
Lol get downvotes for having another opinion. Thanks guysÂ
5
u/taleorca 1d ago
Must be quite shocking for you that the system used to express disagreement is used to express disagreement.
-2
u/MaxxxV 22h ago
Well this community is even more hostile then I thought. Will leave this place.Â
1
1
-20
u/AssaultClipazine 2d ago
I get it. You and the team are being flooded with AI slop PRs, and I understand the need to filter aggressively.
That said, I spent several days researching, testing, and validating some PRs that were closed, and I fully understand and stand behind the code I submitted.
A blanket ban on AI-assisted contributions feels like a sledgehammer. I hope the policy eventually leaves room to distinguish low-effort AI output from well-researched, tested contributions.
Either way, itâs your project, and youâre well within your rights to run it as you see fit.
Keep up the good work.
31
u/mmstick Desktop Engineer 2d ago edited 2d ago
You are permitted to use it only for local research/testing. It is only the generative part that is disallowed. So if you are competent enough to write code by hand and understand it well enough to explain the changes in your own words then there won't be any problems.
11
u/AssaultClipazine 2d ago
Thanks for the reply, I do highly appreciate all of the work you and the team are doing.
-2
u/binarypie 2d ago
just curious how do you separate generated code with clear human understanding from non generated code with clear human understanding?
If the code quality is high and the pr is well documented by a human who can defend the work. does it matter?
4
u/ChaiTRex 2d ago
If the code quality is high and the pr is well documented by a human who can defend the work. does it matter?
They already answered that:
The primary reason is simply our team's load for reviews. In particular, we would like to prioritize working on contributions from our own team and regular contributors, while we have been receiving far more changes from first-time contributors using LLMs. These changes are often unplanned and have had a low acceptance rate.
Having to decide whether an overload of pull requests each constitute high-enough quality code is still an overload. The overload is there because of LLM code generation. Without LLM code generation, the overload is no longer there.
-3
u/binarypie 1d ago
my point is more subtle.Â
Any competent programmer can make a well structured well coded PR with the help of AI that isn't slop in the end.
While the intent of the rule ks obvious the enforcement is not. Since LLM used correctly wouldn't lead to overload the question remains.
How will they objectively know?Â
5
u/mmstick Desktop Engineer 1d ago edited 18h ago
Any competent programmer can make a well structured well coded PR with the help of AI that isn't slop in the end.
If you cannot write code, comments, or descriptions without a LLM then you should not create a pull request.
While the intent of the rule is obvious the enforcement is not.
Enforcement is the easiest part. The policy provides an unbiased justification for closing a pull request that generated code, comments, or descriptions with a LLM.
Since LLM used correctly wouldn't lead to overload the question remains.
This policy change is happening because allowing it has caused overload. The bar is very low for creating a pull request or issue with a LLM.
How will they objectively know?
Wrong question. If you are competent then you will not need to ask this question. If you must lie when agreeing to the policy, you are not competent and will be banned from contributing to COSMIC when caught.
A person that is genuinely interested in contributing is not going to break the rules. They will spend time learning if need be. This is how open source has always worked.
-2
u/binarypie 1d ago
like the linux kernel i think a lot of people are about to start lying. basically anyone using vscode or similar is using an llm for auto completeÂ
2
u/mmstick Desktop Engineer 1d ago edited 14h ago
LSPs such as rust-analyzer are not LLMs. The Linux kernel needs LLMs to scan 20+ million lines of C code for memory and thread safety violations in drivers that aren't being actively maintained and tested outside of automated sanitizers. COSMIC was developed from scratch in Rust by a small team so it does not have that problem.
0
1
u/brad-ml 2d ago
I agree. But I think a reasonable rule is "if it looks AI-generated, we won't check whether it's good code because the cost of checking any such code generally outweighs the benefits" along with "you must be able to explain your code". If you pass both of those tests, they have no basis to reject it. In practice, this means you're going to have to have a heavy hand in writing the code.
0
u/moobini 4h ago
the cost in terms of time and effort have shifted heavily to the maintainers. AI has diminished the general value of contributors because code is now easy to generate.
AI is now a legit tool for coding. But drive-by PRs made possible by AI, submitted by contributors who are unfamiliar with the project, scope, style, etc are killing maintainers on many projects.
I think the âcrowdsourcingâ model of open source development will evolve into smaller teams wielding AI-powered tools themselves to generate code that fits the project better.
-12
-16
u/Status-Afternoon-425 2d ago
I hoped they would leverage AI to quickly catch up with the competition. I really like the foundation they have built so far. But iced and COSMIC DE have so many gaps at the moment...
7
u/LatinaKaren 2d ago
quickly catch up with the competition
At the cost of what?
-6
u/Status-Afternoon-425 2d ago
At the cost of what? Money? Well nobody expect them to pay, but why reject folks who are willing to pay? Yes. Quality bar is not negotiable, but I expect it to be the same bar for human PRs.
5
u/mmstick Desktop Engineer 1d ago edited 1d ago
It is not the same bar. There's a reason why the most prolific contributors are getting their code approved quickly without the help of a LLM while all but the simplest LLM-generated pull requests fail to get past the initial code and QA reviews.
If you have money to fix a problem that matters to you but are unable to do it yourself then create a bug bounty and hire a human developer to fix it. There are plenty of Rust developers looking for work.
-7
2d ago
[deleted]
34
u/mmstick Desktop Engineer 2d ago edited 2d ago
It would be quicker and more reliable for a trusted contributor who understands the code to take on the assignment of writing the code for new features and complex fixes. The only LLM-generated content that was able to meet our review standards over the last year were small changes that would have been better to do ourselves; or at least reserved for human contributors interested in contributing to familiarize themselves with our code, develop some Rust experience, and gain the confidence and trust with maintainers.
13
u/sammy0panda 2d ago
or just reject LLM generated content in PRs causing too much low quality work, as they are doing
-31
u/sb6_6_6_6 2d ago edited 2d ago
Cosmic is already falling behind :( Why not just make a proper AI policy? A few weeks ago, we got a Niri style scrolling addition that is pretty cool and was made entirely by AI, while the request for this feature has been sitting on the wishlist for a long time.
18
u/M4SK1N 2d ago
the author said himself he's new to Rust, he will not support it and hopes someone will review it, so kinda bad example lol
10
u/Aware-Librarian-3749 2d ago
And that it was a proof of concept. There was no intention of it actually being added.
9
u/mmstick Desktop Engineer 2d ago edited 2d ago
There was already an AI policy for 7 months that many complained was too strict. Yet leaving the door ajar slightly for generated code of any kind opens the floodgates wide open for abuse.
A few weeks ago, we got a Niri style scrolling addition that is pretty cool and was made entirely by AI,
They would not have been able to generate this demo had COSMIC been developed and maintained by LLMs. LLMs are only able to adapt it because the foundation already exists with well-written code by hand. So you have to ask yourself if you want something done quickly with reckless abandon or steadily with care.
while the request for this feature has been sitting on the wishlist for a long time.
Anyone from the general public can create a feature request. It does not automatically mean that there is a plan to implement this in COSMIC. This is not on the roadmap so there is no plan to do so currently.
0
u/Shynzon 1d ago
This is not on the roadmal so there is no plan to do so currently.
That's kind of a bummer. It was definitely my number 1 most desired feature.
Fully agreed on the No AI, though. Slow but reliable is the way.
5
u/mmstick Desktop Engineer 1d ago edited 1d ago
The system's keyboard shortcuts are heavily influenced by the tiling system that it was designed around so adding an entirely different paradigm would require a lot of design discussions with keyboard shortcut behaviors being a big part of that.
COSMIC is intentionally built to be modular with a multi-process applet architecture so the intended path to support scrolled tiling is to swap cosmic-comp for Niri. Just as it is possible to swap out applets (ie: GlowBerry and Snappea), the reverse is also possible to use the COSMIC Session with a different compositor.
Niri fundamentally shares a lot of code with COSMIC by virtue of being developed with Smithay and collaborating with COSMIC on features in Smithay+COSMIC so that would be the primary target for integration. There could be a way to provide a separate login session when Niri is installed that runs COSMIC with Niri for scrolled tiling.
That would likely be better than overloading cosmic-comp with a different tiling paradigm since Niri is explicitly designed around scrolled tiling. This would just need some work in the session/settings to integrate with Niri better.
-3
u/ba2ba4e 15h ago
You are closing PRs done before your so called policy change. When a policy changes it's not retroactive. Your GitHub issues are full with 2 years tickets never treated, you refused tested correct PRs for no reason except you do not like AI, AI is mostly translating UIs and fixing your typos.
"If you are using an LLM, and do not fully understand the changes it is making to the code base, do not create a PR." Yes we opened PRs after fully doing understanding the changes, testing it.
Your company promotes this code as "built for your AI stack" you refuse simple PRs addressing 2 years old issues you never address, built with an AI stack that are perfectly in order and would not be better done by you.
And above all you do that with laconic closing comments, you do not even take 5 minutes to test the PR or look at the changes. In 2026 refusing PRs to be checked and improved by AI is ridiculous. "Low acceptance" rate because you refuse the PR by principle, not because there are issues in the code.
3
u/queen-adreena 5h ago
If you canât be bothered to write it, why should they be bothered to read it?
103
u/Professional-Ant5498 2d ago
The COSMIC team is doing an exceptional job. As someone who reviews PRs every day, I completely understand and support them. Reviewing AI-generated PRs can be frustrating, overwhelming, and exhausting, especially when there are so many of them.