Bortlog

2021-02-17 02:06:44 The new "This week in Rust" has an interesting link https://www.reddit.com/r/rust/comments/lg0a7b/benchmarking_tokio_tasks_and_goroutines/ to a discussion of how Go runtime very confidently wins over Tokio's when working with blocking system calls.
The top comment on Reddit explains why it is like that, and how to tweak Tokio's settings to almost eliminate the difference, and why these settings are not applied by default.
That is performance domination we deserve Открыть/Комментировать

2021-02-12 20:57:16 Michael Knyszek has rolled out a new GC pacer proposal for Go https://github.com/golang/go/issues/44167

Nothing magical is offered there, but the new algorithm will allow:
- Consider the number of goroutine stacks and global variables when choosing when to start Mark Phase.
- It will be possible to make a heap even less than 4MB.
- Well, there will be much fewer cases when the heap outgrows the planned limit.

Note: the detailed proposal contains interesting graphs with simulations of the behavior of the current and new pacer. Открыть/Комментировать

2021-02-11 17:39:41 An interesting article (https://blog.cloudflare.com/why-we-started-putting-unpopular-assets-in-memory/) about how the team from Cloudflare divided their hybrid (disk + RAM) cache into separate generations to get rid of cases when a resource is requested once, gets into the cache, and then no one touches it there.
They have a very interesting idea, and a side effect in the form of a 20-25% reduction of disk writes.

But this is all done by the engineering specialists in the field)))
Meantime, researchers one year ago created a neural network for such hybrid caches (

), which guesses what to add and where by keys.
And researchers managed to downgrade the records 6-25 times, depending on the load profile. Открыть/Комментировать

2021-02-10 13:48:20 The author of Tile38 (https://github.com/tidwall/tile38), Josh Baker (tidwall) still does not abandon his idea that on modern hardware the Quadtree structure for spatial search should work better than R-tree. He first wrote about it 2 years ago. But somehow, we have to deal with the fact that Quadtree has a worst-case has search and insert with time complexity of O(n).
He has since come to a 16x16 tree structure at https://github.com/tidwall/ptree
Which, in the case of a uniform distribution of points in the tree, works 3 times faster than https://github.com/tidwall/rtree

Well, and then he has an interesting option with a hybrid structure where we can specify a subspace in which we know for sure that the distribution of points will be uniform, and use ptree for it, and everything else will be indexed using rtree: https://github.com/tidwall/prtree

Another interesting question is why he doesn't use compression in his 16x16 tree as shown here
https://arxiv.org/abs/1411.2785
Perhaps in real projects and maps, the nesting of a 16x16 tree is not so big, and overcomplication simply doesn't worth the effort Открыть/Комментировать

2021-02-09 15:25:23 I listened to the podcast https://media.handmade-seattle.com/the-race-to-replace-c-and-cpp/
In it, developers Jai, Odin, and Zig talk about their compiler development experiences, language design principles, and an understanding of simplicity. It is quite interesting to listen doing some routine activities in the background. Открыть/Комментировать

2021-01-08 14:23:09 Based on a post on Oleg's blog: https://t.me/oleg_log/3989

Thread leaks or goroutines on blocking calls are a fairly common problem in many languages ​​and can be made even by experienced programmers, so in all large projects, they use tools to detect them.
The problem with such leaks is that they are quite difficult to detect when reviewing the code, and you can't just catch them with tests because often everything works correctly, just resources are leaking.

I will not talk here about how to fix such problems - it will depend on the language and idiomatic approaches in each of them. But here are some ways how to detect such leaks faster, even during the testing phase.
1) You can use pools of goroutines or threads with a limited maximum size, which, if leaked, will lead to a deadlock. Deadlock is already easier to detect in tests because everything will stop working.
2) Dump the stacks of all threads before running tests and after they finish and then look for suspicious increases in their number.

An example with a dump can be found here: https://github.com/cockroachdb/cockroach/blob/b02c788d51ecf0b544450653db32e310561b53a5/pkg/util/leaktest/leaktest.go Открыть/Комментировать

2021-01-07 14:03:05 Nice article on the benefits of RISC-V as an instruction set.

https://erik-engheim.medium.com/the-genius-of-risc-v-microprocessors-b19d735abaa6

It explains why RISC-V won't have much of an instruction cache size problem by incorporating instruction compression into the set itself from scratch. Compression allows you to pack 2 instructions into 32 bits for the most frequent instructions.

It turns out that the CPU fetches the code in a packed form from memory and decompresses them before execution using only 400 logical ventels (which is essentially nothing for modern processes).

There are other improvements as well, such as splicing several operations into one during execution.

As a result, it turns out that RISC-V can remain a very simple processor if it needs to be made such (for educational purposes or IoT), but with the addition of optional improvements, it can compete on par with similar processors for ARM and x86 for which such overcomplications are simply a necessity and a part standard. Открыть/Комментировать

2020-12-28 00:28:15 Here the people reverse engineer the BioNTech / Pfizer SARS-CoV-2 vaccine code. It turns out this is ~ 1KB of a rather interesting code printed on a DNA printer, which penetrates a human cell, deceives it, and forces it to produce "needles" identical to those of the coronavirus at a very fast pace, then our immune system detects such a security breach and learns to detect these needles to neutralize them in the future... There are many interesting things the code headers, metadata, modified nucleotides to deceive the cell's defense mechanism, pretend to be a "local" code, and so on.

https://berthub.eu/articles/posts/reverse-engineering-source-code-of-the-biontech-pfizer-vaccine/ Открыть/Комментировать

2020-11-20 18:24:01 We continue to look at hacks from interesting projects. This time we have a whole subsystem for monitoring memory consumption in TiDB. It works globally for the entire database and for each request separately.
But the authors themselves mention that this control method was taken from Apache Impala.

Here is the component itself in which large allocations are registered when a request is made https://github.com/pingcap/tidb/blob/abc8f1665f2b94d6d030977f0e1f0f970e6538d6/util/memory/tracker.go
Tracker itself is thread-safe and allows you to make branches and child Trackers with a separate memory limit if you need to start several threads to execute a request.

Tracker contains a registry of various actions that must be triggered if the limit was exceeded, and then, throughout the code, such actions are registered in different components that execute the request.

The most commonplace examples are to log something on limit reach and/or panic in a goroutine that has exceeded the limit, but there are smarter strategies, for example:
- merge intermediate query results to disk https://github.com/pingcap/tidb/blob/0c822d13a151d80ddc1906fb18e8262e097f9155/util/chunk/row_container.go
- lower the rate limits of asynchronously running processes https://github.com/pingcap/tidb/blob/611d24911ec1653c456a196bd986ac1b263b3b1f/store/tikv/coprocessor.go#L1354

Cleaning of all sorts of caches also sounds like a viable option, but I didn't find such an example in the codebase.

#hacks Открыть/Комментировать

2020-11-18 16:08:53 Here's a selection of hacks from the Tailscale repository:

1) https://github.com/tailscale/tailscale/blob/main/syncs/syncs.go
WaitGroupChan - A WaitGroup that issues a channel so you can wait for it to close. It is mainly used if you want to make one select from a context, and from a waitgroup. Here's an example: https://github.com/tailscale/tailscale/blob/acade77c86741b0100b3a1fc9497c002a523449e/net/netcheck/netcheck.go#L891

2) https://github.com/tailscale/tailscale/blob/main/syncs/watchdog.go
A tool for monitoring of contention on a mutex.

3) https://github.com/tailscale/tailscale/blob/main/syncs/locked.go
Methods for checking whether the mutex is locked or not. Used primarily in tests.

4) https://github.com/tailscale/tailscale/blob/main/atomicfile/atomicfile.go
"Atomic" write to file.

5) https://github.com/tailscale/tailscale/blob/main/wgengine/watchdog.go
"Offensive" monitoring and checking latency.

#hacks Открыть/Комментировать