View unanswered posts | View active topics It is currently Mon Sep 16, 2019 2:45 pm



Post new topic Reply to topic  [ 2 posts ] 
 Decoding A Continuous Network Stream 
Author Message

Joined: Thu Feb 02, 2012 3:25 pm
Posts: 26
Reply with quote
Post Decoding A Continuous Network Stream
I am trying to figure out the best way to go about decoding a continuous network stream. I would like the results to be displayed via Xplico's web interface. I have a few goals in mind. The first is that the newly decoded data should be displayed as quickly as possible and preferably within an hour of the capture time. The second goal is that there should be some way to view only recent data. For example, view data within a 6 hour time frame or within a 24 hour time frame.

The ideal solution would be to use the "Live acquisition" mode in Xplico that constantly decodes data and updates the web interface in real-time and adding an option to only display the previous (user-defined) amount of time. There would also ideally be an option in the web interface to view past data at specific time frames. Xplico would also ideally have some way to "archive" old network traffic and Xplico data so that it could be removed from the server and saved in a storage system. If one wants to view history that has been archived they would need to copy the data back over to the main system. This "archive" piece isn't required, it is more of a long term / side goal. A more realistic solution would be to continue to store network traffic and Xplico data until a (user-defined) limit is reached and then just continue to delete old data as new data comes in such that the amount of data stays below the limit.

However, there are currently several problems with this ideal solution:
1) The "Live acquisition" mode currently drops packets and is not the recommended way to input data into Xplico (uploading .pcap files is)
2) There is currently no way to specify a time interval to view only a range of time (this may get tricky as some protocols last for long periods of time)
3) There is no way to delete old data

Since "Live acquisition" mode is not reliable, I started looking into ways to accomplish a similar result via uploading .pcap files. However, I found that when splitting up a .pcap file and uploading it in separate pieces in the same Xplico session, Xplico was not able to decode protocols that spanned across two files. This is a major issue for trying to accomplish the goal using .pcap file uploading.

Is there a way to accomplish my goal? Does anyone have any other ideas in how it can be done?


Thu Mar 29, 2012 5:57 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Decoding A Continuous Network Stream
Quote:
The first is that the newly decoded data should be displayed as quickly as possible and preferably within an hour of the capture time.

This is possible but its depends from:
  • the type of your network traffics the
  • the average (in hours) amount network traffic
  • the HW resources of your server/s
Quote:
The second goal is that there should be some way to view only recent data. For example, view data within a 6 hour time frame or within a 24 hour time frame.

In Xplico (with XI) the "sessions" have this goal, and you can use the python script (useful for your cli script) session_mng.pyc

Quote:
The ideal solution would be to use the "Live acquisition" mode in Xplico

Very bad idea. You can use net-sniff-ng and it is developed by Daniel Borkmann . From the last release (from some day) this tool support our patch.

Quote:
that constantly decodes data and updates the web interface in real-time and adding an option to only display the previous (user-defined) amount of time. There would also ideally be an option in the web interface to view past data at specific time frames. Xplico would also ideally have some way to "archive" old network traffic and Xplico data so that it could be removed from the server and saved in a storage system. If one wants to view history that has been archived they would need to copy the data back over to the main system. This "archive" piece isn't required, it is more of a long term / side goal. A more realistic solution would be to continue to store network traffic and Xplico data until a (user-defined) limit is reached and then just continue to delete old data as new data comes in such that the amount of data stays below the limit.

This goal is not so complicated to reach. The session can do want you want, and a single session can be removed. Then with a few improvements you can achieve all your goals. Even the store/archive.
Quote:
However, there are currently several problems with this ideal solution:
1) The "Live acquisition" mode currently drops packets and is not the recommended way to input data into Xplico (uploading .pcap files is)

Ok... there are SW and HW solution. Xplico is not a probe.
Quote:
2) There is currently no way to specify a time interval to view only a range of time (this may get tricky as some protocols last for long periods of time)

You can specify (with session) any type of time interval. With net-sniff-ng (with a small patch) and session_mng.pyc you can have a probe with the features that you needs.

Quote:
3) There is no way to delete old data

It is not true... but only because you do not know what can do DeMa for you with a simple script (a lite help: function delete() of file sols_controller.php from XI ;) )

Quote:
Xplico session, Xplico was not able to decode protocols that spanned across two files.

This is not true. But you must know and configure DeMa/Xplico. When (2007) we have designed Xplico (the decoder) we have considered this issue, and inside Xplico and Dema there is the solution (some part of DeMa are not (also) public).

Quote:
Is there a way to accomplish my goal?

Yes!
Quote:
Does anyone have any other ideas in how it can be done?

Yes! write me an email.

Ciao.
Gianluca


Sat Mar 31, 2012 11:06 am
Profile WWW
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 2 posts ] 


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by Vjacheslav Trushkin for Free Forums/DivisionCore.