An example to use it for any log file.

Hi, Thanks for making such an amazing project.

I have trying to use it for my log files. I could parse log files to its equivalent csv files using Logparser by LogPAI, but I have no idea how to convert logs to sequence of number as you have in you ~/data/hdfs/ directory. Also, then how to use it for inference real time log file.

Could please you help me with it?

[donglee-afar]

I will update the code and documentation on how to generate the sequence of number in the next few days.It will include two methods which depends on your own logs: Time sliding window and hard disk ID sequence window.

[kartikeyporwal]

Thanks alot for sharing your expertise.

I’ve gone through your code, I get your idea as (I am putting minute details so that it could be helpful to someone in future.)

1. First gather all the logs which was obtained from normal execution of application, i.e., logs without errors.
2. Combine these logs and convert to _structured.csv and _template.csv file using drain from logpai.
3. Train the model using obtained _structured.csv from step 2.
4. After successful training and saved model, it’s time to test the model’ s accuracy using a abnormal log file (log file with anomaly) and normal log file followed by inference of the model for the new logs files.
5. To implement step 4, since log files will be different in the sequence of events, so obtaining _structured.csv and _template.csv file using drain will not make any sense as randomly generated event_id will be completely different for an event from generated event_id for same event from the log file used for training. So, you proposed structure_bgl.py, using which I can generate event_id for completely new logs based on the event_id of the logs used for training using the generated event_template. Further, sample_bgl.py will convert the structured log into sequence of event_id which can further be replaced by its equivalent integer and thus testing can be performed.
6. Further to inference the model, new log line or logs lines in particular time window can be mapped with training file’s event_template to obtain event_id.

Could I figured it out correctly?

Please feel free to correct me if I failed to describe your approach.

Thanks for your time.