Posted on 2022-05-09
We have been using and working on Skogul extensively for the last few years. The "killer feature" is probably support for Juniper telemetry, but for us, Skogul is much more than that.
You can use Skogul to receive time series data on a number of different formats and using a number of different protocols, and transform it, filter it and selectively redirect it, e.g., to permanent storage. For very simple statistics-gathering setups, it might not be needed - you can just use your data collector to store things directly. For more advanced setups, it's a big benefit.
Today, v0.15.2 is released, which, together with v0.15.0, brings enrichment of metadata to the table. The enrichment in the v0.15-series is still experimental, but I believe it's stable and works well, but the specific configuration details might change somewhat, as they are a bit cumbersome.
PS: Scroll to the bottom for configuration examples.
TL;DR: Adding metadata to incoming time series data.
An example can be if you are using Skogul to receive Junos telemetry and you want to add a description field to the interfaces, which the telemetry doesn't contain. You can now achieve this by loading a database of your interfaces and description.
An other example can be adding a customer ID.
Skogul enrichment takes place in the enrichment Transformer, which is optional. All enrichment exists in a database that lives in memory. Why not make it on-demand, e.g.: Look things up in a database on demand? The main reason I want to avoid that is speed and simplicity. I did some minor tests, and the memory footprint is not significant, assuming the loading is sensible. E.g.: If you actually have 14 million interfaces, skogul will handle it fine, but it will need a few GB of memory.
I want Skogul to work regardless of scale.
One of the nicer things about the enrichment transformer, is that it comes paired with an enrichment sender. This sender can be used to update the enrichment database live, using any mechanism available to Skogul. E.g.: If Skogul can parse and receive data, it can use that to update the enrichment database - granted, many methods don't make sense. You can, in theory, use Juniper telemetry-data to update the enrichment, but that just doesn't make much sense imo.
Because I honestly was feeling lazy, I reused Skoguls definition of a Metric to define enrichment. The way this works is that whatever you specify in the Metadata section will be the look-up key, and whatever is specified in the data section will be added. Now the slightly confusing part is that data-section in the enrichment-data is added to metadata, and there is no way to directly add enrichment-data to the data-section of the "target data". If this is confusing, just bear with me until the examples at the end.
Some of these are acceptable trade-offs, some are more in the "we'll see how it works out before I decide if it's worth addressing".
Run-time speed is basically unaffected by enrichment. If skogul can handle your load without enrichment, adding enrichment does not slow it down. This is because the enrichment is done by just looking up a key in a map and adding it, which is lightning fast. I don't have bench-marks, but it's running fine on some of our more considerable skogul-instance.
The size of the enrichment database affects memory footprint. Currently, there is a flaw where the actual JSON encoding takes up a considerable amount of overhead, and I'm looking into this and hope to resolve it in the v0.15.x series.
The current bottle neck is actually initial loading, but again, you need a serious amount of data to notice. I do intend to use this for data sets with roughly 14 million entries at present time.
Sorry for the delay.
This is from tester_to_stdout_enrich.json:
{
"receivers": {
"test": {
"type": "test",
"handler": "json",
"metrics": 3,
"values": 2,
"delay": "1s",
"threads": 1
},
"updater": {
"type": "http",
"address": "[::1]:8080",
"handlers": { "/": "update" }
}
},
"handlers": {
"json": {
"parser": "json",
"transformers": ["enrich"],
"sender": "print"
},
"update": {
"parser": "json",
"sender": "updater"
}
},
"transformers": {
"enrich": {
"type": "enrich",
"source": "docs/examples/payloads/enrich.json",
"keys": ["key1"]
}
},
"senders": {
"updater": {
"type": "enrichmentupdater",
"enricher": "enrich"
}
}
}
This sets up TWO receivers, the primary receiver is the test-receiver - replaces this with whatever you already have (e.g.: udp/protobuf), the second one, the "updater" is used to update the enrichment data live using HTTP on localhost.
The two handlers are similar. Note that I'm using the "print" sender for the regular data stream which just prints the data to stdout. Since it supports "automake" you don't have to explicitly define it.
The transformer named "enrich" is the one that does the actual enrichment. It has two configuration options, but one might disappear. One option is the list of keys to look up, the other is the source path.
The source past is a json file that is read at start-up, this option was added BEFORE the enrichment updater was written, and I will probably remove it since it is mostly redundant now. But for now, it can be used to set up the initial database.
The "updater" sender, of type "enrichmentupdater" (or just "eupdater") has a reference back to the transformer. I'm sorry for the somewhat confusing names, but you can have multiple independent enrichment transformers with unique names.
We re-use the Metric format. For a file on disk, use:
{
"metadata": { "key1": 1 },
"data": {
"example": "hei"
}
}
{
"metadata": { "key1": 2 },
"data": {
"customer": "nei",
"serial": 123123123
}
}
{
"metadata": { "key1": 0 },
"data": {
"potato": "tomato",
"tree": {
"hi": "ho",
"lol": "kek"
}
}
}
Note: This is NOT an array. It's a stream. Also note: You probably should add a timestamp, even though it's not used. The reason? Because if you try to load that through the file receiver instead, it will fail to validate without SOME timestamp. Sorry for the inconsistency.
If you seek to update this live, you can use something like:
kly@thrawn:~/src/skogul$ cat foo.json
{
"metrics": [
{
"timestamp": "2022-05-09T18:59:00Z",
"metadata": { "key1": 0 },
"data": {
"potato": "lumpe"
}
}
]
}
kly@thrawn:~/src/skogul$ POST -USse http://[::1]:8080/ < foo.json
POST http://[::1]:8080/
User-Agent: lwp-request/6.52 libwww-perl/6.52
Content-Length: 134
Content-Type: application/x-www-form-urlencoded
204 No Content
Connection: close
Date: Mon, 09 May 2022 18:02:46 GMT
Client-Date: Mon, 09 May 2022 18:02:46 GMT
Client-Peer: ::1:8080
Client-Response-Num: 1
kly@thrawn:~/src/skogul$
Here's a complete example:
kly@thrawn:~/src/skogul$ cat docs/examples/tester_to_stdout_enrich.json
{
"receivers": {
"test": {
"type": "test",
"handler": "json",
"metrics": 3,
"values": 2,
"delay": "15s",
"threads": 1
},
"updater": {
"type": "http",
"address": "[::1]:8080",
"handlers": { "/": "update" }
}
},
"handlers": {
"json": {
"parser": "json",
"transformers": ["enrich"],
"sender": "print"
},
"update": {
"parser": "json",
"sender": "updater"
}
},
"transformers": {
"enrich": {
"type": "enrich",
"source": "docs/examples/payloads/enrich.json",
"keys": ["key1"]
}
},
"senders": {
"updater": {
"type": "enrichmentupdater",
"enricher": "enrich"
}
}
}
kly@thrawn:~/src/skogul$ cat docs/examples/payloads/enrich.json
{
"metadata": { "key1": 1 },
"data": {
"example": "hei"
}
}
{
"metadata": { "key1": 2 },
"data": {
"customer": "nei",
"serial": 123123123
}
}
{
"metadata": { "key1": 0 },
"data": {
"potato": "tomato",
"tree": {
"hi": "ho",
"lol": "kek"
}
}
}
kly@thrawn:~/src/skogul$ ./skogul -f docs/examples/tester_to_stdout_enrich.json
WARN[0000] The enrichment transformer is in use. This transformer is highly experimental and not considered production ready. Functionality and configuration parameters will change as it matures. If you use this, PLEASE provide feedback on what your use cases require. category=transformer transformer=enrich
{
"metrics": [
{
"timestamp": "2022-05-09T20:04:14.938349097+02:00",
"metadata": {
"id": "test",
"key1": 0,
"potato": "tomato",
"tree": {
"hi": "ho",
"lol": "kek"
}
},
"data": {
"metric0": 5577006791947779410,
"metric1": 8674665223082153551
}
},
{
"timestamp": "2022-05-09T20:04:14.938349097+02:00",
"metadata": {
"example": "hei",
"id": "test",
"key1": 1
},
"data": {
"metric0": 6129484611666145821,
"metric1": 4037200794235010051
}
},
{
"timestamp": "2022-05-09T20:04:14.938349097+02:00",
"metadata": {
"customer": "nei",
"id": "test",
"key1": 2,
"serial": 123123123
},
"data": {
"metric0": 3916589616287113937,
"metric1": 6334824724549167320
}
}
]
}
In a second terminal, updating the enrichment:
kly@thrawn:~/src/skogul$ cat foo.json
{
"metrics": [
{
"timestamp": "2022-05-09T18:59:00Z",
"metadata": { "key1": 0 },
"data": {
"potato": "lumpe"
}
}
]
}
kly@thrawn:~/src/skogul$ POST -USse http://[::1]:8080/ < foo.json
POST http://[::1]:8080/
User-Agent: lwp-request/6.52 libwww-perl/6.52
Content-Length: 134
Content-Type: application/x-www-form-urlencoded
204 No Content
Connection: close
Date: Mon, 09 May 2022 18:04:22 GMT
Client-Date: Mon, 09 May 2022 18:04:22 GMT
Client-Peer: ::1:8080
Client-Response-Num: 1
Back in the first terminal with Skogul running:
WARN[0007] Hash collision while adding item &{2022-05-09 18:59:00 +0000 UTC map[key1:0] map[potato:lumpe]}! Overwriting! category=transformer transformer=enrich
{
"metrics": [
{
"timestamp": "2022-05-09T20:04:29.939744465+02:00",
"metadata": {
"id": "test",
"key1": 0,
"potato": "lumpe"
},
"data": {
"metric0": 605394647632969758,
"metric1": 1443635317331776148
}
},
{
"timestamp": "2022-05-09T20:04:29.939744465+02:00",
"metadata": {
"example": "hei",
"id": "test",
"key1": 1
},
"data": {
"metric0": 894385949183117216,
"metric1": 2775422040480279449
}
},
{
"timestamp": "2022-05-09T20:04:29.939744465+02:00",
"metadata": {
"customer": "nei",
"id": "test",
"key1": 2,
"serial": 123123123
},
"data": {
"metric0": 4751997750760398084,
"metric1": 7504504064263669287
}
}
]
}
Notice how the "potato"-metadata field updated. You can ignore the warning, since we _know_ it was a "collision": We wanted to update existing data, and that warning is meant for initial loading mostly.
I will be trying to put this blog-post into more proper documentation, add an SQL receiver, probably _remove_ the "source"-setting and instead provide good examples for common use cases. We have a bit too many examples right now.
Any feedback is more than welcome, preferably in the form of issues on github (questions or general discussion on issues are fine - there are not that many issues).
Particularly if you find this useful or if it doesn't solve your enrichment-related problems I'd love to hear from you.