View unanswered posts | View active topics It is currently Mon Jun 17, 2019 9:40 pm



Post new topic Reply to topic  [ 41 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next
 Questions about PEI format/purpose; extracting info from PEI 
Author Message
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Xplico is not a protocol analysis and therefore does not separate the various messages (request-response). We do not want (FTP) a proliferation of PEI for each message, otherwise we should do it for all protocols and is not the goal of the project.
With the FTP comes a file (commands file) with all the commands, and in the XI is made available the pcap file with only the FTP traffic, therefore if someone were to serve to deepen the analysis, then he can use this pcap with Wireshark.
The FTP commands file is useful because in future there will be a Lucene-based search tool that will analyze all the (text based) data decoded.

Rapid alternative to your problem is to put in the file the needed information to you.
If you seek in the FTP dissector:
Quote:
if (clnt) {

you can add both in the 'if' an in to the 'else' if the data is a 'request' or 'response'. You can also select the data by moving the code:
Code:
/* write cmd-response */
fwrite(pkt->data, 1, pkt->len, fp_cmd);

Probably you'll need to add a new field structure ftp_priv (only if you do not want a proliferation of messages 'response' in the file).
Obviously, you can add all the components you want to PEI of FTP (but it is the solution with more work).

Not knowing your goal may be that I did not give you the right advice.


Sat Aug 07, 2010 6:24 am
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
gianluca.costa wrote:
Xplico is not a protocol analysis and therefore does not separate the various messages (request-response). We do not want (FTP) a proliferation of PEI for each message, otherwise we should do it for all protocols and is not the goal of the project.

Hmm, this difference is interesting to me - the difference between Xplico and Wiresharks's functionality and purpose. I am still not clear as to exactly what Xplico does differently from Wireshark, and why. Would you mind expanding on the purpose of Xplico (particularly in respect to the functionality/goals of Wireshark)? "Protocol analysis" seems to mean "helping a human look at packets" (and this sounds more like what Wireshark does and is meant for), while Xplico just "dissects and extracts information from traffic, structures that information, and dispatches it somewhere. This is the way that I would divide how these tools operate - would you agree, or am I missing something? Thanks.

gianluca.costa wrote:
Rapid alternative to your problem is to put in the file the needed information to you.
If you seek in the FTP dissector:
Quote:
if (clnt) {

you can add both in the 'if' an in to the 'else' if the data is a 'request' or 'response'. You can also select the data by moving the code:
Code:
/* write cmd-response */
fwrite(pkt->data, 1, pkt->len, fp_cmd);

Probably you'll need to add a new field structure ftp_priv (only if you do not want a proliferation of messages 'response' in the file).
Obviously, you can add all the components you want to PEI of FTP (but it is the solution with more work).

Not knowing your goal may be that I did not give you the right advice.

I have debugged to the FtpConnec() function where your mentioned code resides, and I now understand more about what is happening. Thank you. I will see what I can do with this.


Mon Aug 09, 2010 9:10 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Quote:
Hmm, this difference is interesting to me - the difference between Xplico and Wiresharks's functionality and purpose. I am still not clear as to exactly what Xplico does differently from Wireshark, and why.

"Wireshark is the world's most popular network protocol analyzer", with Wireshark you can analyze in detail every single packet, and it provides all the information contained in a protocol, packet by packet.
"The goal of Xplico is extract from an internet traffic capture the applications data contained." Even with huge volumes of raw data (except bugs ;) ). It does not provide an analysis packet by packet.
Quote:
Would you mind expanding on the purpose of Xplico (particularly in respect to the functionality/goals of Wireshark)?

Wireshark is not designed to extract, for example, the webmail, or rebuild a file exchanged with emule. Of course could do it.

Quote:
"Protocol analysis" seems to mean "helping a human look at packets" (and this sounds more like what Wireshark does and is meant for), while Xplico just "dissects and extracts information from traffic, structures that information, and dispatches it somewhere. This is the way that I would divide how these tools operate - would you agree, or am I missing something?

More or less. By comparison exaggerated: it is like taking a picture of an apple or to analyze with microscopes an apple. It is always information about an apple.

Moreover, why you chose one of two for FTP? ;)

Ciao.
Gianluca


Tue Aug 10, 2010 7:21 am
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
How do I get information about when the beginning of a TCP connection is established (when the SYN is sent)? Can I extract that information from the TCP PEI from inside a dispatcher? I have not done any debugging to find out yet in depth, but I don't immediately see that information being easily derivable from the TCP PEI ("xplico -i tcp" says the only 4 fields are srcport, dstport, clnt (a packet, I assume), and lost (a packet, I assume)). Ideally, I would want a new TCP PEI component "pei_tcp_syn_id"/"tcp.syn" that would be a string indicating the time of when a syn was sent. Would that be (the best)/(a good) route?

Could the desired information be extracted from the tcp.clnt PEI component? I am thinking it might be since that data looks like a packet that can be parsed for such information. I imagine that that kind of parsing should have been done in the TCP dissector though.

So would making modifications to the TCP dissector be the right next step towards getting the time of when a SYN was sent somewhere? If so, would you mind pointing out any functions already present in the TCP dissector (or elsewhere) that could be useful in detecting/storing when the SYN of a TCP handshake occurs, or if this is even the right route? Thank you.


Thu Aug 12, 2010 10:26 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Quote:
How do I get information about when the beginning of a TCP connection is established (when the SYN is sent)? Can I extract that information from the TCP PEI from inside a dispatcher?

Not all dissector generate PEI. The TCP (as IP, UDP, Ethernet, ...) do not generate PEI.

Quote:
Ideally, I would want a new TCP PEI component "pei_tcp_syn_id"/"tcp.syn" that would be a string indicating the time of when a syn was sent. Would that be (the best)/(a good) route?

This information (such as TCP-PEI) is not there. If you want to know the time of the first packet (that is SYN, if there is no loss of data) then the first packet sent to FTP dissector (or any disserctor over TCP) contains this information. For example line 1006 FTP:
Code:
    /* first tcp packet */
    pkt = FlowGetPkt(flow_id);

the time is:
Code:
pkt->cap_sec

This time is in the first PEI of FTP, in the time_cap filed of the PEI that containing the cmd componet.

Quote:
Could the desired information be extracted from the tcp.clnt PEI component?

No,
  • tcp.clnt is not a PEI component but a TCP stack frame information (or packet information)
  • tcp.clnt is not a time marker

Quote:
So would making modifications to the TCP dissector be the right next step towards getting the time of when a SYN was sent somewhere?

This type of information (starting time, and end time) is already provided to all dissectors based/over on TCP dissector. Each packet comes with the arrival time.

Quote:
If so, would you mind pointing out any functions already present in the TCP dissector (or elsewhere) that could be useful in detecting/storing when the SYN of a TCP handshake occurs, or if this is even the right route?

This informations are already all present, and all dissectors (over TCP) can use them, there is also the reporting of data loss (quantified) for TCP (tcp.lost: Lost packet).
From packet structure:
Code:
    time_t cap_sec;          /**< capture time sec */
    time_t cap_usec;         /**< capture time usec */

If you explain me what you do or what is your goal then it is easier for me to answer.
What is your goal?

Ciao.
Gianluca


Fri Aug 13, 2010 5:51 am
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Thank you again - that was informative. I will work on this more later after this post, but I thought I might give you more information about what I'm trying to do..

For my purposes, it is important to extract information about when a host attempts an active TCP connection to another host - any time a TCP open/connect/SYN is made. And I am particularly interested in extracting the port number that was connected to, and the time.

One particular example is FTP. FTP uses two channels to operate: a command channel and a data channel. The command channel (I believe) stays connected and stays the same throughout a whole FTP session, while a new data channel is created for each data request. In passive FTP, the client sends the PASV command to the server; the server responds with the IP and port number for the client to connect to; the client, after receiving this address information, attempts a TCP connect to the sent address - it is this port number that I am trying extract at the moment.

If that helps much. Thank you.


Fri Aug 13, 2010 4:19 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Well.
To avoid major problems of synchronization between flows (FTP, FTP-data) I suggest you use the TCP dissector called tcp_soft.
From configuration file:
Code:
#MODULE=dis_tcp.so        LOG=FEWS
MODULE=dis_tcp_soft.so   LOG=FEWITDS

We have developed two separate TCP dissectors for two different needs. Both provide the same data to dissectors higher (FTP, POP, SMTP), but with different time constraints (between different flows). Our "application" dissectors (= dissectors over TCP) are designed to work properly with both TCP dissectors, but if you modify the code for FTP then I suggest you change the TCP dissector.

The next two weeks I'm on vacation ... I do not know if I can read and respond.

Ciao.
Gianluca


Fri Aug 13, 2010 6:01 pm
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
gianluca.costa wrote:
Well.
To avoid major problems of synchronization between flows (FTP, FTP-data) I suggest you use the TCP dissector called tcp_soft.
From configuration file:
Code:
#MODULE=dis_tcp.so        LOG=FEWS
MODULE=dis_tcp_soft.so   LOG=FEWITDS

We have developed two separate TCP dissectors for two different needs. Both provide the same data to dissectors higher (FTP, POP, SMTP), but with different time constraints (between different flows). Our "application" dissectors (= dissectors over TCP) are designed to work properly with both TCP dissectors, but if you modify the code for FTP then I suggest you change the TCP dissector.

The next two weeks I'm on vacation ... I do not know if I can read and respond.

Ciao.
Gianluca

I'll check out that tcp_soft dissector and see if it can help.

Thank you for your help - I appreciate your time. I will be back in school after next week (and no longer working) so you won't likely see too much from me by the time you get back (I'll have another job). Have a good time though, and thanks again.


Fri Aug 13, 2010 8:18 pm
Profile

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
I am having difficulty in finding out how to extract when a TCP open occurs in a traffic stream. From previous advice, I was able to extract the time of the first TCP SYN occurring (via debugging until dissectors/ftp/ftp.c:1006 and printing the contents of pkt->cap_sec, and then stopping at the first occurrence of my dispatcher's DispFtp(pei *ppei) function to print ppei->time_cap, and recognizing that they are the same value).

But a TCP connect is more than just that SYN packet - it is 3 packets (SYN, SYN/ACK, ACK). Is there any way to detect a TCP open/connect in this way? I have so far only seen the code operate on a per packet basis - it seems like everything is done per packet and there is no (obvious) way to process a group of packets. I have tried utilizing the various ID-like fields of PEI (id, pid, serial) to see if I could get an order out of these PEIs, but the ppei->id field always appears to be 0. The ppei->serial field has varying numbers in it, but some of them even repeat themselves ("sqlite> select count(serial) from peis where serial = 0;" prints '15'). I was hoping that whenever my DispFtp(ppei) function gets called (it is called from DispInsPei() for each appropriate PEI), the first ppei->id would be 0 (and would be the SYN), the second ppei->id would be 1 (and would be the SYN/ACK), and the third ppei->id would be 2 (and would be the ACK).

And what about TCP opens/connects later on in the traffic stream? How do you detect those? My current solution seems to only tell me when the first SYN occurred - not any later ones, if any. Maybe that's something I'll figure out after I figure out the above problem.


Tue Aug 17, 2010 11:30 pm
Profile

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
What exactly does a PEI represent? What does it "mean"? Oddly enough, this was a question in the first post, but somehow never got answered (probably because I made several distracting posts just afterward, or maybe I didn't catch the answering reply). Do they exist on a per packet basis (a PEI per packet)? But maybe there are multiple PEIs generated per packet, since a packet is a layered bunch of protocols, and each protocol can generate a PEI.

One definition I propose: A PEI is constructed from whatever data a dissector chooses to dissect from each packet given to it. For each piece of data that the dissector considers to be important about a packet, it will represent it as a PEI component of the overall PEI for this packet.

There is no comment above the definition in dispatch/include/pei.h, other than comments describing the fields, but those arn't too informative (IMO) without a higher level description of what the fields are describing.

Along with the wiki documentation, I plan to provide API-level documentation as well, for select functions and data structures. I will provide a comment description of what I interpret a PEI to be, but that's just my thoughts though - this post is asking for a definitive answer from someone with more knowledge than I do. Thanks.


Wed Aug 18, 2010 10:10 pm
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 41 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by Vjacheslav Trushkin for Free Forums/DivisionCore.