Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log
In order to consume
Security Datasets the Kafka way, I recommend to use a tool named Kafkacat to act as a Kafka producer and send data to Kafka brokers.
In producer mode, Kafkacat reads messages from standard input (stdin) or a file. This means that you can send data back to any other Kafka broker that you are using as part of your pipeline.
You can just grab the logs from this repo and re-play them as if they were being ingested in real-time.
Kafka Broker : A distributed publish-subscribe messaging system that is designed to be fast, scalable, fault-tolerant, and durable
Install Kafkacat following the instructions from the official Kafkacat repo
If you are using a debian-based system, make sure you install the latest Kafkacat deb package.
If you are using Ubuntu 19, you might need to run the following commands (Thank you Jason Yee)
sudo dpkg -i libssl1.0.0_1.0.2n-1ubuntu6_amd64.deb
You can also install it from source following the Quick Build instructions.
Download Security Datasets¶
Security-Datasets repo and choose your technique:
$ curl -LJO https://raw.githubusercontent.com/OTRF/Security-Datasets/master/datasets/small/windows/lateral_movement/host/covenant_wmi_wbemcomn_dll_hijack.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 605k 100 605k 0 0 3522k 0 --:--:-- --:--:-- --:--:-- 3522k
Decompress the specific security dataset
$ unzip covenant_wmi_wbemcomn_dll_hijack.zip Archive: covenant_wmi_wbemcomn_dll_hijack.zip inflating: covenant_wmi_wbemcomn_dll_hijack_2020-10-09173318.json
Ship Data to Kafka Broker¶
Send the data to your own kafka broker via Kafcakat with the following flags:
-b: Kafka Broker
-t: Topic in the Kafka Broker to send the data to
-P: Producer mode
-l: Send messages from a file separated by delimiter, as with stdin. (only one file allowed)
$ kafkacat -b <Kafka Broker IP>:9092 -t mordortopic -P -l covenant_wmi_wbemcomn_dll_hijack_2020-10-09173318.json