View unanswered posts | View active topics It is currently Wed Oct 16, 2019 4:21 am



Post new topic Reply to topic  [ 41 posts ]  Go to page 1, 2, 3, 4, 5  Next
 Questions about PEI format/purpose; extracting info from PEI 
Author Message

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Questions about PEI format/purpose; extracting info from PEI
My goal is to export the information generated by Xplico and output it in an SQLite database, using my own schema. It seems to me that this involves creating a new dispatcher. I have looked at the current dispatchers (cli, none, lite, etc.) and none output things in the way that I want. The closest is the disp_lite dispatcher, since it outputs to SQLite - but, of course, not to the tables that I would like.

I looked into writing a dispatcher, and it looks like it comes down to implementing 3 methods: DispInit, DispEnd, and DispInsPei. It seems like the DispInit() method is used to initialize IDs, for use later when DispInsPei is called.

---

What does a PEI (protocol element information) represent? It seems that the only information that a dispatcher is given is a bunch of PEIs, inserted/given as the parameter to the DispIns() function.

I have come to the conclusion that the structure of this PEI seems rather arbitrarily descriptive. With FTP for example, you can run "./xplico -i ftp" to get a protocol description for FTP, and it displays 9 fields (namely, url/user/password/cmd/file_in/file_out/offset/down_n/up_n). My question is, does the protocol description for FTP HAVE to look like that, and have those fields? Can any more be added/removed to/from those fields? Why is it just those 9 fields? My hunch is that it doesn't need to just be those 9 fields, but can actually be any properties/attributes that one likes. Where is the PEI FTP format defined?

My point is that I'm trying to write a dispatcher to export info in a certain way, but I am finding that I cannot find the needed information from what is given by the passed-in PEIs. A proposed solution can then be, is there a way to add the needed information to the PEI before it is passed to the DispIns() function? I imagine that this would involve modifying a part of the code that occurs before the dispatchers (like, the dissectors).


Mon Jul 26, 2010 6:04 pm
Profile

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
I found the answer to my question while I was at the end of my post's completion, but thought it would be good to post anyway for others to see.

---

I had a look at the xplico-src/dissectors/ftp directory, and found the answer to my previous post. Namely, the 9 PEI fields that FTP uses/defines are set in the DissecRegist() function of dissectors/ftp/ftp.c . In that function, I can clearly see how each of those 9 PEI fields are constructed and registered.

I am still looking things over to see how I can put the needed information into the PEI format, but this is putting me on the right track I think.


Mon Jul 26, 2010 6:10 pm
Profile

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Is it possible to export results in a format similar to Wireshark's display, in that packets are displayed in a time-dependent (and not protocol-oriented)? I have found Xplico's output formats to be oriented in some way that I could not use (like, directory per IP address output, or directory per protocol output) - what would be the easiest way to get output that is arrival time oriented, like Wiresharks default display? Is that doable by just writing a new dispatcher, or will I have to create/modify a dissector to construct appropriate PEIs (for a dispatcher)? Is an output like that already achievable by some other means (and am unaware of it)? Thank you


Mon Jul 26, 2010 10:42 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Quote:
Is it possible to export results in a format similar to Wireshark's display, in that packets are displayed in a time-dependent (and not protocol-oriented)?

Yes, every PEI (and also components of a PEI) has a time (start and end), you can use this information to order (by time) the data extracted.
Xplico is multi-thread, the temporal synchronicity of the output data is not guaranteed. There are mechanisms (too complex) to ensure (on request) the temporal synchronicity of input data, for example to decode H323 protocol that it is composed of many streams (and protocol: TPKT, Q931, H225, H245, RTP, RTCP, ...) these streams must be synchronous to extract the data correctly.

Quote:
I have found Xplico's output formats to be oriented in some way that I could not use (like, directory per IP address output, or directory per protocol output) - what would be the easiest way to get output that is arrival time oriented, like Wiresharks default display?

Xplico output formats is not oriented in IP or directory or etc, Xplico output formats is oriented to data extracted, between the data there is also the time (capture time). Maybe you're confusing the dispatcher with decoding/dissector.

Quote:
Is that doable by just writing a new dispatcher, or will I have to create/modify a dissector to construct appropriate PEIs (for a dispatcher)?

Yes, Xplico was designed also for this purpose. You have many degrees of liberty in order to achieve what you want. If you can achieve your goal with the dispatcher, then the compatibility of your dispatcher, with future versions of Xplico is easier to maintain.

Quote:
Is an output like that already achievable by some other means (and am unaware of it)?

Yes, we have develop a custom dispatcher (GPL) but it is not public.
Another solution is to use the data in the DB and create a new timeline view. Maybe we implement it for the next release.

Would you like to write what you learn (discover) in the Wiki so that it is useful to others?

Ciao.
Gianluca


Tue Jul 27, 2010 5:37 am
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Thank you for the reply - it was informative.

Could you confirm my beliefs with regards to the timing of when DispPei() is called (for any dispatcher)? Are you saying that, since Xplico is multi-threaded, the DispPei() functions will be called in no particular order? This would mean that I should use the pei's timing information (from inside the DispPei() function) for my ordering purposes?

I will likely document my findings once I have actually completed my goal with regards to Xplico. By then I will likely have even more knowledge about the tool than I do now, and would make for better documentation.


Tue Jul 27, 2010 5:09 pm
Profile

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
One particular constraint on output that I would like to inquire about.. it is important to keep track of where protocol information comes from with respect to particular input PCAP files. Does the PEI include a way for me to know which particular PCAP file the protocol element information is coming from? It is important, for my particular output purposes, that the output "remembers" which particular PCAP file the output comes from.


Tue Jul 27, 2010 5:41 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Quote:
Could you confirm my beliefs with regards to the timing of when DispPei() is called (for any dispatcher)?

Maybe you talk about DispInsPei of a dispatcher. This function is the 'main' function of dispatcher.
At this function (int DispInsPei(pei *ppei)) is passed a pointer of PEI, then the capture time of the data "inside" the PEI is: ppei->time_cap
And every component of an PEI has its capture-time.

Quote:
Are you saying that, since Xplico is multi-threaded, the DispPei() functions will be called in no particular order?

Yes, any thread (~ dissectors) call this function asynchronously than others threads. If you have some atomicity constraints, these should be handled by the dispatcher (see lite and cli dispatchers mutex).
But if you want that all PEI insertion request are serialized (not time capture based) then it suffices that in the configuration file is set: DISPATCH_PARALLEL=1
In this case the accumulation of PEI is handled by the system that will avoid running out of memory because of PEI.

Quote:
This would mean that I should use the pei's timing information (from inside the DispPei() function) for my ordering purposes?

Yes: ppei->time_cap

Quote:
I will likely document my findings once I have actually completed my goal with regards to Xplico. By then I will likely have even more knowledge about the tool than I do now, and would make for better documentation.

The policy of the wiki is interesting because it uses evolution. Any information can be improved, then you can start anytime.

Ciao.
Gianluca


Tue Jul 27, 2010 5:44 pm
Profile WWW
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Quote:
Does the PEI include a way for me to know which particular PCAP file the protocol element information is coming from?

Yes every PEI has a stack frame: ppei->stack in this stack you can find all informations (see pol, sol, ip extraction -in a dispatcher-).
To print (in the shell) the stack: ProtStackFrmDisp(ppei->stk, TRUE);
To save (in a buffer) a XML representation of a stack: ProtStackFrmXML(ppei->stack);
This XML is the info-xml file of XI.

Ciao.
Gianluca


Tue Jul 27, 2010 5:53 pm
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Yes, the entire time, I meant the DispInsPei() function - not DispPei(). Thank you.


Tue Jul 27, 2010 5:56 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
a mistake, I meant:
Code:
 DISPATCH_PARALLEL=0

The dispatcher is used also by all manipulators (see configuration file of manipulators).
Ciao.
Gianluca


Tue Jul 27, 2010 6:02 pm
Profile WWW
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 41 posts ]  Go to page 1, 2, 3, 4, 5  Next


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by Vjacheslav Trushkin for Free Forums/DivisionCore.