View unanswered posts | View active topics It is currently Wed Oct 16, 2019 4:48 am



Post new topic Reply to topic  [ 41 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next
 Questions about PEI format/purpose; extracting info from PEI 
Author Message

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
gianluca.costa wrote:
Quote:
I really appreciate your help - without it, I would have to find out all of this information myself, costing time. Documentation is, however, not my current priority - my priority is results. I will document things once I have accomplished my goal(s) with Xplico - I need to get information exported first. Thank you though.

Exactly, I'm saving you some time.
I understand your priorities. I work for a living and certainly help you is not my priority during the week.
Maybe you should wait until I have time to write documentation on the wiki.
I do not believe in the promises, especially if they come from companies.

Ciao.
Gianluca

True. We'll see if I contribute or not : )


Fri Jul 30, 2010 9:03 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Each packets and every PEI in Xplico have a stack. The stack collects all information on protocols (precisely dissectors) concerning the packet or the PEI. In reality the stack is not a stack but (for complex protocols with multiple streams) a tree.
The PEI has the components. The stack has frames and frames have the attributes.
The pcap file name is only in these dissectors: pol, pcapf. These dissectors are the root of decoding:
Code:
------------------------------------------                                                   
------------- Protocol Graph -------------                                                   
------------------------------------------                                                   
 pcapf                                                                                       
   |                                                                                         
   |--->eth                                                                                   
   |     |                                                                                   
   |     |--->pppoe                                                                           
   |     |     |                                                                             
   |     |     `--->ppp                                                                       
   |     |           |                                                                       
   |     |           |--->ip                                                                 
   |     |           |     |                                                                 
   |     |           |     |--->ipv6                                                         
   |     |           |     |     |                                                           
   |     |           |     |     |--->tcp                                                     
   |     |           |     |     |     |                                                     
   |     |           |     |     |     |--->http                                             
   |     |           |     |     |     |     |                                               
   |     |           |     |     |     |     |--->httpfd                                     
   |     |           |     |     |     |     |--->ipp                                         
   |     |           |     |     |     |     |--->mms                                         
   |     |           |     |     |     |     |--->fbwchat                                     
   |     |           |     |     |     |     `--->webmail                                     
   |     |           |     |     |     |--->pop                                               
   |     |           |     |     |     |--->imap                                             
   |     |           |     |     |     |--->smtp                                             
   |     |           |     |     |     |--->sip                                               
   |     |           |     |     |     |--->ftp                                               
   |     |           |     |     |     |--->nntp                                             
   |     |           |     |     |     |--->irc                                               
   |     |           |     |     |     |--->pjl                                               
   |     |           |     |     |     |--->telnet                                           
   |     |           |     |     |     |--->paltalk                                           
   |     |           |     |     |     `--->tcp-grb                                           
   |     |           |     |     `--->udp                                                     
   |     |           |     |           |                                                     
   |     |           |     |           |--->sip                                               
   |     |           |     |           |--->rtp                                               
   |     |           |     |           |--->l2tp                                             
   |     |           |     |           |     |                                               
   |     |           |     |           |     `--->ppp                                         
   |     |           |     |           |--->dns                                               
   |     |           |     |           |--->tftp                                             
   |     |           |     |           `--->udp-grb                                           
   |     |           |     |--->tcp                                                           
   |     |           |     |--->udp                                                           
   |     |           |     `--->icmp                                                         
   |     |           |--->ipv6                                                               
   |     |           `--->llc                                                                 
   |     |                 |                                                                 
   |     |                 |--->eth                                                           
   |     |                 |--->ip                                                           
   |     |                 `--->ipv6                                                         
   |     |--->ip                                                                             
   |     |--->ipv6                                                                           
   |     |--->vlan                                                                           
   |     |     |                                                                             
   |     |     |--->ip                                                                       
   |     |     |--->ipv6                                                                     
   |     |     `--->arp                                                                       
   |     `--->arp                                                                             
   |--->ppp                                                                                   
   |--->ip                                                                                   
   |--->sll                                                                                   
   |     |                                                                                   
   |     |--->ip                                                                             
   |     `--->ipv6                                                                           
   |--->wlan                                                                                 
   `--->llc                                                                                   
------------------------------------------                                                   
 pol                                                                                         
   |                                                                                         
   |--->eth                                                                                   
   |--->ppp                                                                                   
   |--->ip                                                                                   
   |--->sll                                                                                   
   |--->wlan
   `--->llc
------------------------------------------

From a stack to extract the file name of pcap the code (written on the fly and not tested) is:
Code:
    const pstack_f *frame;
    int pol_id, pcapf_id,pol_filename_id, pcapf_filename_id;
    ftval val;

    pol_id = ProtId("pol");
    pcapf_id = ProtId("pcapf");
    pol_filename_id = ProtAttrId(pol_id, "pol.file"); // ./xplico -i pol
    pcapf_filename_id = ProtAttrId(pcapf_id, "pcapf.file"); // ./xplico -i pcapf

    frame = ProtStackSearchProt(ppei->stack, pol_id);
    if (frame) {
        ProtGetAttr(frame, pol_filename_id, &val);
        printf("name: $s\n", val.str);
    }
    else {
        frame = ProtStackSearchProt(ppei->stack, pcapf_id);
        if (frame) {
            ProtGetAttr(frame, pcapf_filename_id, &val);
            printf("name: $s\n", val.str);
        }
    }

Remember to use:
Code:
FTFree(&val, FT_STRING);

to free memory. It is necessary for strings.
You can find the pcap name extraction in DispInsPei function of Lite dispatcher. Search pol_filename_id .

Gianluca


Sat Jul 31, 2010 11:00 am
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Thank you for your reply - I will go over it Monday. I hope you got to notice some updates to the wiki regarding the components/code I'm currently interacting/working with.


Sat Jul 31, 2010 2:36 pm
Profile

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
gianluca.costa wrote:
Each packets and every PEI in Xplico have a stack. The stack collects all information on protocols (precisely dissectors) concerning the packet or the PEI. In reality the stack is not a stack but (for complex protocols with multiple streams) a tree.
The PEI has the components. The stack has frames and frames have the attributes.
The pcap file name is only in these dissectors: pol, pcapf. These dissectors are the root of decoding:
Code:
------------------------------------------                                                   
------------- Protocol Graph -------------                                                   
------------------------------------------                                                   
 pcapf                                                                                       
   |                                                                                         
   |--->eth                                                                                   
   |     |                                                                                   
   |     |--->pppoe                                                                           
   |     |     |                                                                             
   |     |     `--->ppp                                                                       
   |     |           |                                                                       
   |     |           |--->ip                                                                 
   |     |           |     |                                                                 
   |     |           |     |--->ipv6                                                         
   |     |           |     |     |                                                           
   |     |           |     |     |--->tcp                                                     
   |     |           |     |     |     |                                                     
   |     |           |     |     |     |--->http                                             
   |     |           |     |     |     |     |                                               
   |     |           |     |     |     |     |--->httpfd                                     
   |     |           |     |     |     |     |--->ipp                                         
   |     |           |     |     |     |     |--->mms                                         
   |     |           |     |     |     |     |--->fbwchat                                     
   |     |           |     |     |     |     `--->webmail                                     
   |     |           |     |     |     |--->pop                                               
   |     |           |     |     |     |--->imap                                             
   |     |           |     |     |     |--->smtp                                             
   |     |           |     |     |     |--->sip                                               
   |     |           |     |     |     |--->ftp                                               
   |     |           |     |     |     |--->nntp                                             
   |     |           |     |     |     |--->irc                                               
   |     |           |     |     |     |--->pjl                                               
   |     |           |     |     |     |--->telnet                                           
   |     |           |     |     |     |--->paltalk                                           
   |     |           |     |     |     `--->tcp-grb                                           
   |     |           |     |     `--->udp                                                     
   |     |           |     |           |                                                     
   |     |           |     |           |--->sip                                               
   |     |           |     |           |--->rtp                                               
   |     |           |     |           |--->l2tp                                             
   |     |           |     |           |     |                                               
   |     |           |     |           |     `--->ppp                                         
   |     |           |     |           |--->dns                                               
   |     |           |     |           |--->tftp                                             
   |     |           |     |           `--->udp-grb                                           
   |     |           |     |--->tcp                                                           
   |     |           |     |--->udp                                                           
   |     |           |     `--->icmp                                                         
   |     |           |--->ipv6                                                               
   |     |           `--->llc                                                                 
   |     |                 |                                                                 
   |     |                 |--->eth                                                           
   |     |                 |--->ip                                                           
   |     |                 `--->ipv6                                                         
   |     |--->ip                                                                             
   |     |--->ipv6                                                                           
   |     |--->vlan                                                                           
   |     |     |                                                                             
   |     |     |--->ip                                                                       
   |     |     |--->ipv6                                                                     
   |     |     `--->arp                                                                       
   |     `--->arp                                                                             
   |--->ppp                                                                                   
   |--->ip                                                                                   
   |--->sll                                                                                   
   |     |                                                                                   
   |     |--->ip                                                                             
   |     `--->ipv6                                                                           
   |--->wlan                                                                                 
   `--->llc                                                                                   
------------------------------------------                                                   
 pol                                                                                         
   |                                                                                         
   |--->eth                                                                                   
   |--->ppp                                                                                   
   |--->ip                                                                                   
   |--->sll                                                                                   
   |--->wlan
   `--->llc
------------------------------------------


I have noticed how protocols were stacked up like that, b/c when I was debugging earlier (a week or so ago), I ran into the ProtDissect() function, where various protocol-specific packet dissectors (PktDis) were called. I noticed how, in order, things were called - pcapf -> eth -> ...

gianluca.costa wrote:
From a stack to extract the file name of pcap the code (written on the fly and not tested) is:
Code:
    const pstack_f *frame;
    int pol_id, pcapf_id,pol_filename_id, pcapf_filename_id;
    ftval val;

    pol_id = ProtId("pol");
    pcapf_id = ProtId("pcapf");
    pol_filename_id = ProtAttrId(pol_id, "pol.file"); // ./xplico -i pol
    pcapf_filename_id = ProtAttrId(pcapf_id, "pcapf.file"); // ./xplico -i pcapf

    frame = ProtStackSearchProt(ppei->stack, pol_id);
    if (frame) {
        ProtGetAttr(frame, pol_filename_id, &val);
        printf("name: $s\n", val.str);
    }
    else {
        frame = ProtStackSearchProt(ppei->stack, pcapf_id);
        if (frame) {
            ProtGetAttr(frame, pcapf_filename_id, &val);
            printf("name: $s\n", val.str);
        }
    }

Remember to use:
Code:
FTFree(&val, FT_STRING);

to free memory. It is necessary for strings.
You can find the pcap name extraction in DispInsPei function of Lite dispatcher. Search pol_filename_id .

Gianluca


I remember trying this code out before, but now that I have seen how you would do it, I have found a mistake on my part (originating from this post - viewtopic.php?p=626&sid=2897698637ab59e66f2cf9e206c6e4ac#p626). Specifically, I called ProtPeiComptId() instead of ProtAttrId(). Not only did I not call the correct function, but also (would have) called it with bad parameters (namely, using "file" instead of "pcapf.file"). I have made the change and it works correctly now - I now have the filename I was looking for. Thanks for the help.

This does provide evidence of a hypothesis I had in a previous post (viewtopic.php?p=628&sid=2897698637ab59e66f2cf9e206c6e4ac#p628) though.


Mon Aug 02, 2010 5:02 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Thanks for the help with the Wiki documentation.


Tue Aug 03, 2010 4:54 am
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
How do you extract the FTP command that was used? In the lite dispatcher (the dispatcher I'm basing my custom dispatcher on), you see code like the following in the "while (cmpn != NULL)" loop of the DispFtp() function:

Code:
else if (cmpn->eid == pei_ftp_cmd_id) {
            ftp_filename = cmpn->name;
            path = cmpn->file_path;
}


I created a char *cmd variable and added "cmd = cmpn->strbuf;" just above the "ftp_filename = cmpn->name;" line. While running/debugging Xplico I noticed that whenever the code enters that branch of execution (when cmpn->eid == pei_ftp_cmd_id), the cmpn->strbuf variable is blank/garbage/NULL, and so the 'cmd' variable gets set as such as well. Why is this? Isn't cmpn->strbuf (at that point) supposed to be the FTP command that was used? There are all kinds of FTP commands like USER, PASS, and LIST... isn't this how I'm supposed to get that information?


Tue Aug 03, 2010 9:28 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Hi kizzo,
the pei_ftp_cmd_id component of FTP PEI is only the path of the file (created by dissectorr FTP) where you can find (inside the file) all commands (and all LIST from server).
If you search the user name and password you have two options:

  • the PEI components pei_ftp_user_id and pei_ftp_pswd_id (lite dispatcher)
  • parse the cmd file (cmpn->file_path of pei_ftp_cmd_id component)
Quote:
Why is this?

A component in a PEI can be only (at this time):
  • a string (populated with PeiCompAddStingBuff by dissector)
  • a file (populated with PeiCompAddFile by dissector)
Quote:
Isn't cmpn->strbuf (at that point) supposed to be the FTP command that was used?

No, in FTP cmpn->strbuf is used for containing the user name and the password.
Quote:
There are all kinds of FTP commands like USER, PASS, and LIST... isn't this how I'm supposed to get that information?

Xplico not work like Wireshark does not provide a list of commands but tries to provide the data and the information sent / received from interception. In FTP : file donwloaded/uploaded, user name, password and all FTP command (in a file).

I hope I have answered your questions.


Wed Aug 04, 2010 5:47 am
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
gianluca.costa wrote:
Hi kizzo,
the pei_ftp_cmd_id component of FTP PEI is only the path of the file (created by dissectorr FTP) where you can find (inside the file) all commands (and all LIST from server).
If you search the user name and password you have two options:

  • the PEI components pei_ftp_user_id and pei_ftp_pswd_id (lite dispatcher)
  • parse the cmd file (cmpn->file_path of pei_ftp_cmd_id component)
Quote:
Why is this?

A component in a PEI can be only (at this time):
  • a string (populated with PeiCompAddStingBuff by dissector)
  • a file (populated with PeiCompAddFile by dissector)
Quote:
Isn't cmpn->strbuf (at that point) supposed to be the FTP command that was used?

No, in FTP cmpn->strbuf is used for containing the user name and the password.
Quote:
There are all kinds of FTP commands like USER, PASS, and LIST... isn't this how I'm supposed to get that information?

Xplico not work like Wireshark does not provide a list of commands but tries to provide the data and the information sent / received from interception. In FTP : file donwloaded/uploaded, user name, password and all FTP command (in a file).

I hope I have answered your questions.

I have taken the time to examine the files you speak of - namely, the files given by the pei_ftp_cmd_id component. I saw the raw output of FTP conversations, and saw all of the commands (and their outputs) there as well (LIST, USER, PASS, etc.). Just to acknowledge.

If I were to ask "How do I extract the FTP commands from a PEI?", are you asserting that you would reply with "You don't, because you can't (currently). The best thing you have with regards to getting FTP commands is with the file indicated by pei_ftp_cmd_id ." Are you saying that the FTP dissector COULD (has the potential but is currently not implemented) extract the FTP commands from packets, and add those commands as components to a PEI, but it does'nt, and instead outputs that kind of info to a file - namely, the file indicated by pei_ftp_cmd_id ?

With this, it seams like I have two options for getting the FTP commands from the point of view of a dispatcher - I could either modify the dissector to add the commands as PEI components (so that when a dispatcher receives a PEI, it can directly access the FTP command(s) from it, without having to parse a file), OR the dispatcher could parse the file indicated by pei_ftp_cmd_id to get the commands. The second option seems more easy, and only involves more work with the dispatcher (I believe the first option would require modifying the FTP dissector, and that's just more learning/work).

However, I believe that the first option is a better option overall. I believe that, with regards to the role that PEI plays (particularly, sitting between the dissectors and the dispatchers), the PEI should itself contain everything interesting about a given protocol. In particular to FTP, it seems obvious that FTP commands should be directly (and easily) extractable from a PEI (the reason being that FTP commands are such an obvious part of the FTP protocol - they should be right there). I understand that the FTP commands, indeed, ARE accessible from a PEI, it's just that you have to parse a file to get them.

I'll sum up my questions. Could you confirm the two options described in the third paragraph from the top (either modifying the FTP dissector or parsing the indicated file). Also, why was a name like pei_ftp_cmd_id chosen (the output of "xplico -i ftp" says "cmd: User commands")? Is this just a shortcut for now? When a component of the FTP PEI says pei_cmd_id, it just seems too easy to assume it is referring to an FTP command (LIST, USER, etc.), and not a file full of data. Thanks for the help.


Wed Aug 04, 2010 5:24 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
Quote:
I'll sum up my questions. Could you confirm the two options described in the third paragraph from the top (either modifying the FTP dissector or parsing the indicated file)
.
Yes I confirm. The first option (add new components at the PEI) is not complicated to implement. In fact, as you can see from the FTP dissector (FtpCommand function), the FTP dissector recognizes many of the FTP ommands.
But, between the two options the easiest and fastest is the second.

Quote:
Also, why was a name like pei_ftp_cmd_id chosen (the output of "xplico -i ftp" says "cmd: User commands")?

Simple: I do not know English. You're lucky that there is the "-i" command. ;)
Sorry for the error.

Quote:
Is this just a shortcut for now?

No, the file (with comands and responses) is exactly what we wanted. It is not intended to provide a command at a time to dispathcer. In reality, the FTP is missing some important parts (see for example the function FtpParseLpasv).
Quote:
When a component of the FTP PEI says pei_cmd_id, it just seems too easy to assume it is referring to an FTP command (LIST, USER, etc.), and not a file full of data.

You're right, I made a mistake.


Wed Aug 04, 2010 6:36 pm
Profile WWW

Joined: Tue Jul 20, 2010 5:35 pm
Posts: 32
Reply with quote
Post Re: Questions about PEI format/purpose; extracting info from
gianluca.costa wrote:
Yes I confirm. The first option (add new components at the PEI) is not complicated to implement. In fact, as you can see from the FTP dissector (FtpCommand function), the FTP dissector recognizes many of the FTP ommands.
But, between the two options the easiest and fastest is the second.

It seems I have run into a problem with regards to the "meaning" of parsing the file - how do you know which IP is sending/receiving the commands and which one is sending/receiving the responses? The file is just the raw output of an FTP session - it says nothing about the source or destination of where any of the commands/replies are going to or coming from.

How do I associate that information correctly? Am I supposed to use the given PEI from which the filename came from, somehow (if so, I don't see clearly how)? Would this involve working with the dissector(s), perhaps making a new FTP PEI component? Thanks.


Fri Aug 06, 2010 8:34 pm
Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 41 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by Vjacheslav Trushkin for Free Forums/DivisionCore.