PetaByte per day load generation

cribl architecture

Cribl Stream helps you process machine data — logs, instrumentation data, application data, metrics, etc. — in real time, and deliver them to your analysis platform of choice. It allows you to:

• Add context to your data, by enriching it with information from external data sources.
• Help secure your data, by redacting, obfuscating, or encrypting sensitive fields.
• Optimize your data, per your performance and cost requirements.

To generate peta byte scale load

100 workers, 10 worker group, 1 leader

25 syslog source-in per worker group

Load gen

20k containers: 200 ec2 c5.4xlarge each running 100 containers of syslog

(in+out) 6.93*3600*24*2 = 1.1 peta byte / day

3.5 billion events per min