Python: Detect Charset and convert to Unicode


Converting data to Unicode via chardet 
http://stackoverflow.com/questions/2686709/encoding-in-python-with-lxml-complex-solution

import chardet
from lxml import html

f = open('my.xml')

content = f.read()

f.close()

encoding = chardet.detect(content)['encoding']
if encoding != 'utf-8':
    content = content.decode(encoding, 'replace').encode('utf-8')

Converting Data to Unicode via UnicodeDammit 

http://lxml.de/elementsoup.html

from BeautifulSoup import UnicodeDammit

converted = UnicodeDammit(content)

   if not converted.unicode: 

    raise UnicodeDecodeError( 

       "Failed to detect encoding, tried [%s]", 

        ', '.join(converted.triedEncodings))  

        # print converted.originalEncoding

         return converted.unicode

Credit goes to Ian Bicking and others on the lxml team

Python Unicode Convert

Convert weird unicode string characters to string:
import unicodedata
unicodedata.normalize('NFKD',unicode_string).encode('ascii','ignore')

Django module for helping convert things to ascii strings:
from django.utils.encoding import smart_str
smart_str('weird string goes here')

Another hack for removing unicode errors:
unicode('my weird string', errors='ignore')


http://docs.python.org/howto/unicode.html#the-unicode-type