​​Logging is one of the three pillars of observability. One of | CatOps

​​Logging is one of the three pillars of observability. One of the Palantir Blog posts outlines some pretty nice ideas that may help you to build a better observability solution.

Structured logging. Of course, all of us know that logs are easier to analyze if they are structured in commonly across applications. Palantir improved the way "message" field usually looks like:


{
"message": "Processing event from user",
"params": {
"event": "foo",
"user": "bar"
},
"level": "DEBUG",
"logger": "com.logger.name",
"time": "2019-07-28T09:31:04.46165-07:00"
}


Note that these structured logs avoid a common logging practice of inserting parameters into the log message via string formatting, creating a variable message ("Processing event foo from user bar"). Such an approach will prevent you from building complex log queries to filter out variable messages. Instead, you can search for an exact match and add additional filters by other fields (e.g. params.user == "bar")

Logs from 3rd party services. Usually, logs from 3rd-party components do not fit your unified logging structure. To overcome this, Palantir engineers built tooling that analyzes Golang source code and creates regular expressions which convert original messages to the ones with an expected structure.

You can read more about it in the blog post itself. I hope these ideas will help you to build a better observability system!

P.S. Thanks Valerii Tatarin for this post. If you'd like to share something with community too, feel free rich @MaxymVlasov or @grem1in.

#logging #observability