Bortlog

2021-05-12 13:50:46 I have a great article for you about the challenges and problems of writing software for real-time audio generation.
https://glowcoil.com/posts/basedrop/

That's where we cannot do without low-level programming languages ​​with absolute control over what exactly will happen at any moment. Because you can't allow any unpredictable (non-constant) operations in the thread that generates the audio, it turns out that you can't do I/O, allocate or free memory, use mutexes or blocking queues to synchronize threads, etc.
With such restrictions, even the use of Rust looks too high-level because its RAII can play a bad joke on you and start to free some buffers when they lose their owner. So it turns out that even with Rust, you may, in special cases, have a memory leak or OOM.

It seems that for such software Zig with its explicit use of allocators and explicit defer, would be a little more convenient, but also not without difficulties. Открыть/Комментировать

2021-04-08 14:02:09 Great article that shows what will happen if you do not design asynchronous programming features in low-level language from scratch https://tomaka.medium.com/a-look-back-at-asynchronous-rust-d54d63934a1c

It turns out that everything is kind of works, but there are a handful of annoying nuances that you have to put up with or fix with some dirty hacks. And it is similar in most languages and most runtimes. It seems to me that the industry is trying to solve the problem of expensive OS threads and context switches, not where it needs to be solved. Why can't we make OS threads more lightweight? That would be great, and we already know how to use treads in our streams and work with queues, semaphores, and mutexes. All our tooling content understands streams, then what's the problem?

In 2013, people from Google delivered this talk


Where they demonstrate how to reduce the overhead of context switch 100 times or more.
If I understand correctly, in 2013, they already used this solution for services written in C ++ using "regular" threads.

In 2020, they even offered 3 patches for Linux, but didn't finish the discussion and did not merge these patches, and it may take another 5-10 years to get back to this idea. Открыть/Комментировать

2021-04-06 22:42:07 Well, if we are talking about all sorts of dynamic analysis to detect memory safety issues, then here is the news that I could not miss:
Linus merged to 5.12 the following mechanism https://lwn.net/Articles/835542/
And here are some explanations of how it works https://github.com/google/kasan/blob/master/KFENCE.md
In general, KFENCE is a kernel memory safety error detector with such a low overhead that it can be used in prod. Открыть/Комментировать

2021-04-02 13:04:40 We had a really good conversation with @insert_reference_here in the comments for "Memory tagging" post here https://t.me/bortlog/81
And I think the arguments presented there are really compelling, but to some degree, they are based on the assumption that memory safety features based on type system and static analysis are essentially free.

But there are actual proofs that this is not true, and here is one of them:
As you may know, Microsoft is working on a C language extension that will bring memory safety features to it https://t.me/bortlog/23
They added 3 new pointer types, with a whole bunch on statical analysis, and at some points where it is not possible to prove safety staticaly, they add runtime null and bounds checks.

Microsoft Research team recently presented their results


and they showed the effects of the conversion of some C codebases to Checked C. On average, they change 18% of code during conversion, code size increases by 7%, and runtime overhead is almost 9%, and compile-time increased by 25%.

Of course, we can not draw any conclusions for only one research team work, but I'd say that it is safe to admit that in real-world examples, even type system and static analysis assisted memory safety is not free, and hardware support of some runtime checks with only <5% overhead is definitely needed. Открыть/Комментировать

2021-04-02 12:27:30 In case if you missed yesterday's announcement: Rustaceans are actively working to bring garbage collection to Rust.
Proof Открыть/Комментировать

2021-04-01 00:12:16 Many people post a recent announcement of the ARMv9 architecture, but the most remarkable feature (in my opinion) has been announced for a long time now, and its support is even ready in compilers and operating systems. "Memory tagging" can be a revolutionary leap that will ensure the security and correctness of programs written in languages ​​with manual memory management.

For less than 5% CPU memory bandwidth, you get a 93% chance of intercepting global-buffer-overflow, heap-buffer-overflow, use-after-free, use-after-return, use-after-scope, stack-buffer-overflow. If your program works long enough and you have really good test coverage, you will catch all these bugs without any problems at all. In debug mode, you can pay 20% of bandwidth and get a full stack trace of where exactly, who and how violated the integrity, and so on.

In such a world, the benefits of big runtimes, overcomplicated programming languages, etc., partially disappear, and we again, solve everything on the silicon level))

In short, I highly recommend this talk; there are more details on how exactly this is implemented at the LLVM compiler and memory allocator in Android.



Открыть/Комментировать

2021-03-31 00:21:37 Here is a 2 years old article, which explains why the architecture of all modern operating systems is obsolete. The advent of new hardware and very high-speed network and disk interfaces, makes the strategy of running the kernel on all CPU cores and work with interfaces via system calls, disadvantageous due to the large number of context switches and the presence of shared memory.
To write really high-performant software, people are now doing kernel bypass, using eBPF or XDP.
And this is just one of the 5 points of criticism listed by the article authors.

The authors propose a new operating system architecture which they call parakernel. And this architecture will be as complementary as possible to all sorts of thread-per-core frameworks such as Seastar (http://seastar.io). Parakernel allocates all the necessary DRAM resources, network RX / TX queues, and disk NVMe queues when starting or initializing the process and guarantees their isolation and unique ownership by the process. It is proposed to implement network protocols in user space. Discard almost all POSIX APIs, especially various blocking operations, get rid of virtual address space, or leave it as a security feature, but ensure that the OS will make page replacements, swap memory pages, etc.
It turns out that the in such architecture kernel will only deal with resource allocation, handle process failures, and work as an adapter in case we run on some legacy hardware that does not support resource allocation at the driver or hardware level.

In short, it is a really cool article, and if you are interested in details, you can come here: https://penberg.org/parakernel-hotos19.pdf Открыть/Комментировать

2021-03-31 00:21:29 Channel name was changed to «Bortlog» Открыть/Комментировать

2021-02-21 12:18:55 Just a couple of hours ago, Oleg and I recorded a REST vs gRPC video.
As I promised, I am posting an extension video with a description of my thoughts on JSON indexing.
Do not take it too seriously, I just described some ideas, that haven't been verified in any way.
I did a little introduction there, but I suggest you watch the main podcast first.



Открыть/Комментировать

2021-02-17 21:18:09 If you want to relax and watch how someone is codding, take a look at this channel https://youtube.com/c/AndreasKling
This person has been writing his operating system for a couple of years, but the last video is quite simple to watch



To understand how TLB works and how exactly CPU and kernel agree on which pages are available for writing, or read-only, or execution, I advise https://open.spotify.com/episode/2jF3PrFnnyaeRYfteLTYeW?si=DyU6iHJDQ7aiYQ8rfq2jrA

Maybe Andreas himself has a video on the channel where he impements it Открыть/Комментировать