View unanswered posts | View active topics It is currently Wed Nov 14, 2018 1:43 pm



Post new topic Reply to topic  [ 9 posts ] 
 Problems decoding webmail and new version availability 
Author Message

Joined: Sat Jan 21, 2012 8:03 pm
Posts: 4
Reply with quote
Post Problems decoding webmail and new version availability
Hello,
Congratulations for this great project.

I've been testing xplico for a time now, but all the versions I've tested so far seem to fail decoding webmail messages (yahoo, and live)... I've tested versions 0.7 and 0.7.1, both of them with the debian packages and also with the virtual machines. I've also tested the DEFT distribution and also the new package recently released for backtrack... None of them worked with yahoo or hotmail... It seems (I'm not an expert at all -just guessing based on the errors I'm receiving) there might be some kind of problem decoding unicode characters...

However, I've seen that in the demo website you have online to test the new version of xplico (version 1.0) the application works great decoding webmail.

Do you have any idea about when the new version will be available for downloading?
Is there a way to solve the problems decoding webmail messages in versions 0.7 or 0.7.1?

Thanks in advance!
Best regards


Sat Jan 21, 2012 8:47 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Problems decoding webmail and new version availability
The Live/Hotmail webmail decoding depend to the language, the xplico version up to 0.7.1 can decode only Italian, English and France, but we are added (on the demo web, i.e. 1.0.0 version) the Spanish.
For the Yahoo webmail we are fixed some bugs in the decoding script.

If you can send us a your pcap (bug[@t]xplico.org) we can test it and we can send you the new two scripts (compatible with 0.7.1 version).

Ciao.
Gianluca


Sun Jan 22, 2012 9:18 am
Profile WWW
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Problems decoding webmail and new version availability
Hi,
you can use these two scripts scripts (CC BY-NC-SA 3.0) on 0.7.1.
Attachment:
wmail_update.tar.bz2

Ciao.
Gianluca


You do not have the required permissions to view the files attached to this post.


Mon Jan 23, 2012 1:35 pm
Profile WWW

Joined: Sat Jan 21, 2012 8:03 pm
Posts: 4
Reply with quote
Post Re: Problems decoding webmail and new version availability
Hello again,
thank you for posting the scripts.

Sorry to bother again, but I've been testing the scripts and I'm getting the same UnicodeEncode errors:

Traceback (most recent call last):
File "wbm_yahoo_v2.py", line 360, in <module>
File "wbm_yahoo_v2.py", line 356, in ymain
File "wbm_yahoo_v2.py", line 244, in mail_received
File "wbm_yahoo_v2.py", line 128, in save_msg
UnicodeEncodeError: 'ascii' codec can't encode character '\xa0' in position 159: ordinal not in range(128)

Traceback (most recent call last):
File "wbm_yahoo_v2.py", line 360, in <module>
File "wbm_yahoo_v2.py", line 356, in ymain
File "wbm_yahoo_v2.py", line 244, in mail_received
File "wbm_yahoo_v2.py", line 128, in save_msg
UnicodeEncodeError: 'ascii' codec can't encode character '\xbf' in position 354: ordinal not in range(128)
...

The messages (and the website showing them) are written in Spanish, and that seems to be the problem. It seems Xplico finds the webmail messages, but when detects a non ascii character xplico ignores the message.

I don't know much about python. I've searched the web trying to globally use "utf-8" instead of ascii and I changed main.py to read "encoding = utf-8" instead of "encoding = ascii", but it didn't helped either...

Is there some global configuration I should use in python or in xplico in order not to use ascii when using characters?
Is there a way to globally configure utf-8 instead of ascii? Or maybe a way to bypass the errors so xplico could continue ignoring just the "nos ascii" character but not the whole message?

As I said before, sorry to bother again, but I tested the demo website onlie and it works great with webmail messages even if they contain non ascii characters...

Thank you in advance.

Best regards


Wed Jan 25, 2012 6:43 am
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Problems decoding webmail and new version availability
Hi,
have you used the same pcaps that you have sent us?


Thu Jan 26, 2012 9:03 pm
Profile WWW

Joined: Sat Jan 21, 2012 8:03 pm
Posts: 4
Reply with quote
Post Re: Problems decoding webmail and new version availability
Hello,
yes, exactly the same. I've just tried them again in your demo site (demo.xplico.org) and the demo site decoded 8 messages in the yahoo pcap. When I try the same pcap in my virtual machine, xplico only decodes one message. It seems the message xplico was able to decode is completely written in English, I mean without non ascii characters... The same happens with hotmail: 1 decoded webmail message in the demo site and none in my virtual machines.

Right now I'm testing xplico with two virtual machines, one created with the DEFT iso image and the other one is the downloadable 0.7.1 virtual box machine. I copied the scripts you posted in "/opt/xplico/bin/" overwriting the ones that were in that folder before.

When I upload the pcaps to my xplico installations I receive several errors in console saying "UnicodeEncodeError: 'ascii' codec can't encode character". So my first thought was you might have some adjustments in your development environment to be able to handle "non ascci" characters, or to use another version of python or something similar.

Thank you for the time you are taking trying to solve this issue.

Best regards from Spain!


Fri Jan 27, 2012 10:25 am
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Problems decoding webmail and new version availability
The scripts are the same. The web demo has the checksum verification disabled and your pcaps have many TCP packets with checksum errors... then: have you disabled or not the checksum verification on your tests? if the answer is 'no'... this is the issue, and now you know the solution ;) .

You can access to the Xplico configuration pages from admin login (user:admin, password:xplico)

Ciao
Gianluca


Sat Jan 28, 2012 9:19 am
Profile WWW

Joined: Sat Jan 21, 2012 8:03 pm
Posts: 4
Reply with quote
Post Re: Problems decoding webmail and new version availability
Hello,
Thak you once more for your time trying to solve this issue.

Sorry, I guess I should have posted this in one of my messages before, but I also have the checksum verification disabled. That was one of the first things I tried ;)...

This afternoon as a new test I downloaded the new BackBox ISO image and created a virtual machine with it (installation language and keyboard = English US). After following the steps on the xplico website to install xplico, I started the application and disabled checksum verification. Then, I tested the application with my pcaps, getting (in console) the same UnicodeErrors I got with all the other distributions I've tested. Afer this first test, I copied the scripts you posted here (overwritting the ones previously installed and changing permissions), and repeated the test with the same result: UnicodeErrors.

In my humble opinion, the problem is not in my pcaps (I've tested xplico with several ones) but with a problem in xplico decoding webmail messages with non-ascii characters. The error says:

UnicodeEncodeError: 'ascii' codec can't encode character '\xa0' in position 103: ordinal not in range(128)

May I ask "'ascii' codec"? is that the correct "codec" xplico should be using?

Anyway, I can wait until you release the next version of xplico, hoping it solves this issue, as your demo site online works fine with the pcaps I upload.

Thank you again for your great project,

Best regards from Spain


Sat Jan 28, 2012 7:22 pm
Profile
Site Admin

Joined: Wed Sep 16, 2009 10:09 pm
Posts: 394
Reply with quote
Post Re: Problems decoding webmail and new version availability
Hi,
the scripts are the same of web demo but for python3.2 (the demo run with python3.1). I tested your pcaps with an Ubuntu 11.10 with xplico 0.7.1 and the scripts attached to this post, and all run well.
I'm not be able to identify the problem and then fix it.

For me it is important to identify the issue to understand it (and so fix it).
Now I have no idea how to proceed... I'll think of something

Ciao,
Gianluca


Sun Jan 29, 2012 10:19 am
Profile WWW
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by Vjacheslav Trushkin for Free Forums/DivisionCore.