So I have been working in the industry for around 2 years now. 1.5 years as an embedded firmware engineer and half a year as an embedded linux engineer at the same company(startup). Lately I have been working a lot with BSP and device driver development. I have noticed that Canonical has a junior linux kernel developer position open and I wanted to know is it smart to get into this field as a linux device driver developer and also to go work for a company like Canonical.

Hey everyone, especially kernel/OS developers,

I had a weird thought and I'm curious how you'd approach it from a systems perspective:

If we imagined the human brain as a kernel, and neurons, brain regions, memory, sensory systems, etc. as kernel subsystems, what kind of kernel architecture would make the most sense? hybrid kernel microkernel or monolithic or something else?

Traditional Python profilers work in userspace using sampling (py-spy, Pyroscope). They miss what happens at the kernel level. GIL contention, off-CPU waits, scheduler delays are all invisible to them.

I built a profiler using BCC that attaches eBPF uprobes to CPython's take_gil/drop_gil and gc_collect_main. It walks the Python frame chain directly from BPF to capture full Python stacks at the exact moment of GIL contention or GC pause. No sampling. Event-driven. Every event has exact timestamps.

It also uses finish_task_switch tracepoint for off-CPU analysis, combining kernel + userspace + Python stacks in a single event. Plus tcp_set_state kprobe for TCP handshake latency and retransmit tracking.

What we found in production: URL resolution was the biggest GIL holder, added caching, cut GIL wait by 80%. GC gen-2 was pausing 1.27s walking 1.5M objects, fixed with gc.freeze(), down to 88ms. Off-CPU stack traces revealed a Kafka push was blocking threads for 130ms per request, completely invisible in OpenTelemetry because the off-CPU time happens below the instrumentation layer. Found a stale nginx upstream IP causing 504s using TCP loss tracking.

The interesting technical challenge was walking CPython's PyFrameObject chain from inside BPF. I get the base address from /proc/<pid>/maps, then walk tstate -> cframe -> current_frame. Each frame points to a code object which has the filename and function name. I read this using bpf_probe_read_user, walking the linked list until NULL or max depth.

CPython 3.11 only. Struct offsets are hardcoded. BCC + Python. Apache 2.0.

GitHub: https://github.com/deepanshu406/python-ebpf-profiling

I ran a red team on a project of mine, Governance Kernel.

Governance Kernel is a deterministic governance layer

for AI agents. It sits beneath the agent and decides

permissions, trust levels, and resource bounds before

any action is executed.

Repo: https://github.com/mohamedaitzaouit84-hue/governance-kernel

Context on the project:

- V0.4: 54/54 adversarial prompts blocked, PRI = 1.0000

- V0.5: 5/5 gates

- V0.6: 5/5 gates

- V0.7: 19/19 gates

- 2 dependencies (cryptography, pyyaml)

- 3 environments verified (CI, Colab, Termux)

Now the red team.

I did a self red team first (V0.7.1): 20 attacks,

18 blocked, 2 documented. The report itself said:

"Self-red-team is NOT equivalent to external red

team. External red team remains the strongest

missing validation."

So I invited external review. Two AI systems

(Claude and Kimi) plus a manual verification pass

in Termux produced the following:

9 findings total.

CRITICAL (6):

- J-0.8.50 — audit chain can be rewritten by

recomputing hashes (hash chain, not signature

chain).

- J-0.8.51 — signed checkpoints can be re-signed

by the same attacker, because the signing key

is readable.

- J-0.8.52 — kill switch can be disabled by

deleting control/kill.flag.

- J-0.8.53 — identity/owner_key.priv is stored

unencrypted (0o600 only).

- J-0.8.57 — resource_governor.reset() runs

without authorization. Any process can erase

the resource state.

- J-0.8.58 — rotation_manifest.jsonl is unsigned.

PARTIAL (3):

- J-0.8.54 — bootstrap re-signs tampered policies

silently.

- J-0.8.55 — subjects.json is unsigned.

- J-0.8.56 — audit log tail can be truncated

without detection.

Root cause: the trust base is the file system.

Every guarantee the kernel makes on top of a

file that a same-user process can modify is

advisory.

I am posting this before fixing the findings.

The transparency is more valuable to me than

the appearance of safety.

The kernel itself (V0.5-V0.7) passed all its

gates. Nothing in the red team invalidated the

kernel loop, the agents, the consensus, or the

delegation mechanism. The findings are in the

layers around the kernel: the trust base, the

audit trail, the control plane, and the policy

distribution.

The fix plan is documented (FREEZE_v0.7.19),

about 6 working days of work.

Full report:

https://github.com/mohamedaitzaouit84-hue/governance-kernel/blob/main/docs/RED_TEAM_v0.7.18_AI_ASSISTED.md

Threat model:

https://github.com/mohamedaitzaouit84-hue/governance-kernel/blob/main/docs/SECURITY_MODEL.md

Fix plan:

https://github.com/mohamedaitzaouit84-hue/governance-kernel/blob/main/docs/FREEZE_v0.7.19.md

The protocol I followed for the red team is at:

https://github.com/mohamedaitzaouit84-hue/governance-kernel/blob/main/docs/RED_TEAM_PROTOCOL.md

If you see an attack I missed, I want to hear it.

I have no budget; recognition for valid findings

is academic (credit in JOURNEY.md, optional

co-authorship on a preprint), not financial.

Traditional Python profilers work in userspace using sampling (py-spy, Pyroscope). They miss what happens at the kernel level. GIL contention, off-CPU waits, scheduler delays are all invisible to them.

I built a profiler using BCC that attaches eBPF uprobes to CPython's take_gil/drop_gil and gc_collect_main. It walks the Python frame chain directly from BPF to capture full Python stacks at the exact moment of GIL contention or GC pause. No sampling. Event-driven. Every event has exact timestamps.

It also uses finish_task_switch tracepoint for off-CPU analysis, combining kernel + userspace + Python stacks in a single event. Plus tcp_set_state kprobe for TCP handshake latency and retransmit tracking.

What we found in production: URL resolution was the biggest GIL holder, added caching, cut GIL wait by 80%. GC gen-2 was pausing 1.27s walking 1.5M objects, fixed with gc.freeze(), down to 88ms. Off-CPU stack traces revealed a Kafka push was blocking threads for 130ms per request, something completely invisible in OpenTelemetry because the off-CPU time happens below the instrumentation layer. Found a stale nginx upstream IP causing 504s using TCP loss tracking.

The interesting technical challenge was walking CPython's PyFrameObject chain from inside BPF. Each frame has a pointer to the code object, which has the filename and function name. You walk the linked list until you hit NULL or max depth.

CPython 3.11 only. Struct offsets are hardcoded. BCC + Python. Apache 2.0.

GitHub: https://github.com/deepanshu406/python-ebpf-profiling

thought this was a pretty neat idea. didn't seem that hard so i took a stab at it with an llm. turned out there was a pretty short path to a clean implementation. at first, i started with (more or less) re-implementing NFS from scratch on top of bcachefs, then the concurrecy/resiliency/durability quickly became too much to manage, so i decided to pivot and attempt it as an extension on NFS. that worked out surprisingly well.

supports btrfs and bcachefs for the backend. built on LTS so it should be easy to maintain. I'll probably automate that bit (the rebase should be very mechanical).

the test coverage is pretty extensive, and i'm currently testing it at work to replace a 10x larger lustre cluster, so i think it's probably worth sharing despite being mostly vibe coded.

About 7 months ago I posted here about OpenGoodixSPI, an experimental open-source project to bring Linux support to Goodix SPI fingerprint sensors found in some Huawei laptops.

That post unexpectedly became the starting point for the project getting its first contributors and hardware testers, so I wanted to come back with an update.

GitHub: https://github.com/PeshalaDilshan/OpenGoodixSPI

Where we are now

The project has moved quite a bit beyond the original proof of concept.

Current work includes:

• Linux kernel SPI driver infrastructure

• ACPI device matching

• SPI communication/debugging

• IRQ handling

• reset handling

• character-device interface for experimentation

• protocol investigation

• hardware-specific findings for GXFP51A0

The project is still experimental, and we do NOT have working fingerprint capture yet.

The interesting part: GXFP51A0

One of the devices we're investigating is the Goodix GXFP51A0.

We've found that the ACPI resources expose a reset GPIO separately from the SPI device. Hardware investigation indicates a reset sequence approximately like:

RESET LOW

10 ms

RESET HIGH

100 ms

SPI initialization

That fixed one important piece of the hardware integration.

But then we hit the real problem.

The protocol

After reset, we're still not getting a useful response from the sensor in the current Linux implementation. In some states we're seeing responses such as:

FF FF FF FF ...

We don't yet know exactly what that means.

I don't want to make the usual reverse-engineering mistake of deciding that \`0xFF\` must mean "firmware missing" and then building the entire driver around that assumption.

We need to understand what Windows actually sends to the sensor.

We've found interesting information about the device and firmware through Windows-side investigation, but the complete initialization transaction is still missing.

What would help the most

At this point, the single most valuable thing would probably be a logic-analyzer capture of a compatible Goodix SPI fingerprint sensor while Windows initializes it.

Something like:

RESET

↓

SPI initialization

↓

Wake / handshake

↓

Device identification

↓

Configuration / firmware

↓

Sensor initialization

↓

Ready

A capture from a real GXFP51A0 would be especially useful.

We're also looking for people who have:

• a Huawei laptop with a Goodix SPI fingerprint sensor

• GXFP51A0 / GXFP5187 / GXFP3287 hardware

• experience with SPI protocol reverse engineering

• a logic analyzer

• experience with Linux kernel SPI drivers

• experience with libfprint

• Goodix documentation or legitimate developer documentation

We're NOT looking for leaked source code or anything confidential that shouldn't be shared.

Public documentation, protocol information, hardware observations, logs, or legitimate developer resources would all be useful.

I also contacted Goodix

I've contacted Goodix to ask whether they can provide developer documentation or point the project toward the appropriate technical/developer contact.

Hopefully that leads somewhere.

Until then, we're continuing with hardware investigation and reverse engineering.

What's still missing

• [ ] Fully understand GXFP51A0 initialization

• [ ] Confirm SPI framing

• [ ] Understand the \`0xFF\` responses

• [ ] Reproduce the Windows initialization sequence

• [ ] Understand firmware loading/update behavior

• [ ] Capture fingerprint images

• [ ] Understand enrollment/matching

• [ ] Integrate with libfprint

• [ ] Test additional Goodix SPI devices

If you have one of these sensors, even if you don't know anything about kernel development, you can still help.

A laptop model, ACPI information, kernel logs, Windows driver information, or simply being able to test experimental builds can be useful.

Thanks again to everyone who helped with the original post. That's how this project got its first contributors in the first place.

Hopefully we can find the missing piece together.

Hi everyone,

I'm 23 and I'm from Argentina. I have a 3-year tertiary degree in Software Development, and I'm also studying Systems Engineering, which is a 5-year degree here. If everything goes well, I should finish it in about 2 years.

My English is pretty good, and one of my main goals is to eventually work remotely, since most of the opportunities I see in the areas I'm interested in seem to be outside Argentina.

Lately I've been getting more interested in low-level / systems areas: Embedded Linux, firmware, bare metal, RTOS, Linux kernel / drivers, and systems programming in C, C++ or Rust.

I'm trying to figure out how realistic this path is for someone with my background. I come more from a software development background than an electrical engineering one, so I'm not sure which area would make the most sense to focus on first.

For example, would Embedded Linux be a more practical entry point than bare-metal firmware or kernel development? Are remote jobs in these areas realistic from Latin America, or do most companies expect you to be on-site because of hardware access? Also, how much does not having the engineering degree finished yet matter for these kinds of roles?

I'm not looking for a shortcut. I know this stuff takes strong fundamentals and a lot of hands-on work. I just want to choose a specialization carefully before investing the next few years into it.

If you work in embedded, firmware, Linux kernel, RTOS, or systems programming, I'd really appreciate your perspective. What would you focus on first? What skills or portfolio projects would actually make someone employable in this market?

Thanks in advance.

My motherboard uses this driver for WiFi and Bluetooth the WiFi works flawlessly but I’m having to use a patch to get Bluetooth working does anyone know the status of the MR for the firmware for that chipset? Last I heard anything about it was months ago im referring to these for those interested in reading: https://github.com/samutoljamo/bazzite-mt7927/pkgs/container/bazzite-mt7927

https://jetm.github.io/blog/posts/mt7927-wifi-making-it-work/

A lot has happened since the last time I posted.

I bought the book "Operating Systems: 3 Easy Pieces". I wish I had read this before starting my PMM. I highly recommend it to anyone else starting from the beginning.

The PMM has had yet another rewrite. Hopefully the last major one.

My kernel crossed the 3M boundary and I had to revisit my boot.S I suspect my PMM and VMM data structures are too big, but we will see.

I decided I should let the people that might benefit from my posts here should see my struggles, and there have been many. In that light I have decided to make my repository public, you can find it here: https://github.com/tedavids/JakelynnOS

I am currently working on my VMM and the module documentation is already up to 8 pages, and growing. you can find it in the "Module doco.doc"

There is nothing really interesting on the screen yet, but here it is:

It does also run on VirtualBox, but I haven't tried bare metal, as I don't want to spring for a CD writer at this point. I have a couple of old Lenovo's that might support USB Boot, but I haven't investigated those yet. I want to get to user space before I start checking that out.

For all your AI haters, in my README I do have the AI policy I'm following.

Guess that's all for now. I hope this helps some newbie (like me) that is struggling.

Thanks for reading

Does anyone know if there is any support, alpha, experimental or per manufacturer that would support some api to determine current system power usage?

I'm assuming it would require some data channel between the PSU and the motherboard. Does any power supply on the market support this and does linux offer any support if something like that did exist? Is anything close, or is it something that is workable or is this just some completely unsupported pipe dream?

Also, what about for usb powered devices like a pi? I assume whatever support would exist for that would have to be entirely on chip, not related to the psu.

libark

libark is a modern, secure, Swift-native foundational layer over Linux POSIX primitives.

While originally developed as the core runtime layer for ArkOS, libark is completely decoupled and designed to be used on any Linux distribution. It provides highly secure, object-oriented abstractions over low-level system calls, memory management, and file system interactions, without the burden of manual C interoperability.

Features

• Object-Oriented POSIX: Wraps file descriptors, paths, and directory operations in safe, RAII-compliant Swift types.
• Secure Syscall Boundary: Routes kernel interactions through strict, audited Syscall primitives.
• Terminal Capabilities: Built-in abstractions for TTY detection and window sizing.
• Advanced Formatting: Standardized formatters for human-readable byte sizes (SI and IEC), ISO 8601 timestamps, permissions, and classifications.
• Declarative Sorting: Powerful generic SortEngine for metadata sorting across multiple criteria.

Also ARK-OS has switched to github: https://github.com/orgs/ARK-OS-Swift-and-Linux/repositories

These errors appeared on the console of one of my lab's servers. We've already changed the CPU. I'm not familiar with server management, so if you could explain it to me in the most didactic way, I would be very grateful.

kernel:[Hardware Error]: Corrected error, no action required.

kernel:[Hardware Error]: CPU:0 (19:8:2) MC18_STATUS[Over|CE|MiscV|AddrV|-|-|SyndV|CECC|-|-|-]: 0xdc2040000000011b

kernel:[Hardware Error]: Error Addr: 0x0000000017eb09c0

kernel:[Hardware Error]: PPIN: 0x02b0ba6d78974068

kernel:[Hardware Error]: IPID: 0x0000009600150f00, Syndrome: 0x083fb50b0a800410

kernel:[Hardware Error]: Unified Memory Controller Ext. Error Code: 0

I'm trying to do some bpf programming. Helpful folks elsewhere have pointed me at clang rather than gcc. Ok, sweet.

So:

hyperlynx@Osiris ~/Projects/MfdFix/repo (_clang) $ which clang
/usr/bin/clang
hyperlynx@Osiris ~/Projects/MfdFix/repo (_clang) $ clang --version
clang version 22.1.8
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

but for some reason, make just cannot access it. Huh?

[make all]
/usr/bin/clang -nostdinc -emit-llvm -Wall -isystem /usr/lib/clang/22/include -O2 -D__KERNEL__ -I/usr/lib/modules/7.2.3-1-cachyos/build/arch/x86/include/uapi -I/usr/lib/modules/7.2.3-1-cachyos/build/arch/x86/include/generated/uapi -I/usr/lib/modules/7.2.3-1-cachyos/build/include/generated/uapi -I/usr/lib/modules/7.2.3-1-cachyos/build/include/uapi -I/usr/lib/modules/7.2.3-1-cachyos/build/include -I/usr/lib/modules/7.2.3-1-cachyos/build/arch/x86/include -c ./fixup.c -o ~/Projects/MfdFix/build/fixup.o
/bin/sh: line 1: /usr/bin/clang: No such file or directory
[Finished in 0 seconds with code 2]

whaaaat's going on? I can run that command from the shell myself and it works (well, I mean, I have problems in my code so no it doesn't, but clang at least gets invoked).

All I've been able to find when searching for "clang: No such file or directory" is people who do not have clang installed, and I definitely do!

Huh?

Edit: Doh. I realised the next morning what's gone wrong. The change from gcc to clang wasn't the only change I made in this time period, I also switched from system vim to a flatpak, and flatpaks by default can't access most of the filesystem!

I just happen to be running make from vim.

That's why I couldn't find any help for this elsewhere - it's my own particular weird setup that's to blame. Presumably I can use flatseal to fix the problem.

Sorry for wasting people's time, and thanks for taking a look nonetheless. Doh.

Edit edit:
Huh. I've just got home from work, used flatseal to grant Vim all the permissions, and it still can't see /usr/bin/clang. In fact, I can open a vim window to /usr/bin, and it's simply not listed in there.

Well, okay, either way this seems obviously a flatpak'd vim problem, not a make problem. Oh well.

Got tired of opening htop, checking logs, docker, etc. every time something felt slow, so I made this little linux debug overlay.

It sits on screen and shows the current apps CPU and RAM + warnings when something looks off slowly adding docker + kubernetes stuff to it too.

still building it, but honestly pretty fun seeing it come together lol.

[https://github.com/codeafridi/Debug-Overlay-App]()