python编码解码单词_在使用w2v时python中的编码问题

我正在使用word2vec模型在python中编写我的第一个应用程序。这里是我的简单的代码在使用w2v时python中的编码问题

import gensim, logging

import sys

import warnings

from gensim.models import Word2Vec

logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

def main():

####LOAD MODEL

model = Word2Vec.load_word2vec_format('models/vec-cbow.txt', binary=False)

model.similarity('man', 'women')

if __name__ == '__main__':

with warnings.catch_warnings():

warnings.simplefilter("error")

#warnings.simplefilter("ignore")

main()

我得到这个以下错误：

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 96-97: invalid continuation byte

我试图通过增加这两条线解决它，但我仍然得到错误。

reload(sys) # Reload does the trick!

sys.setdefaultencoding('UTF8') #UTF8 #latin-1

该w2v模型是在英语句子训练。

编辑：以下是完整的堆栈：

**%run "...\getSimilarity.py"**

---------------------------------------------------------------------------

UnicodeDecodeError Traceback (most recent call last)

**...\getSimilarity.py in ()**

64 warnings.simplefilter("error")

65 #warnings.simplefilter("ignore")

---> 66 main()

**...\getSimilarity.py in main()**

30 ####LOAD MODEL

---> 31 model = Word2Vec.load_word2vec_format('models/vec-cbow.txt', binary=False) # C binary format

32 model.similarity('man', 'women')

**...\AppData\Local\Enthought\Canopy\User\lib\site-packages\gensim-0.12.4-py2.7-win-amd64.egg\gensim\models\word2vec.pyc in load_word2vec_format(cls, fname, fvocab, binary, encoding, unicode_errors)**

1090 else:

1091 for line_no, line in enumerate(fin):

-> 1092 parts = utils.to_unicode(line.rstrip(), encoding=encoding, errors=unicode_errors).split(" ")

1093 if len(parts) != vector_size + 1:

1094 raise ValueError("invalid vector on line %s (is this really the text format?)" % (line_no))

**...\AppData\Local\Enthought\Canopy\User\lib\site-packages\gensim-0.12.4-py2.7-win-amd64.egg\gensim\utils.pyc in any2unicode(text, encoding, errors)**

215 if isinstance(text, unicode):

216 return text

--> 217 return unicode(text, encoding, errors=errors)

218 to_unicode = any2unicode

219

**...\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.6.2.3262.win-x86_64\lib\encodings\utf_8.pyc in decode(input, errors)**

15 def decode(input, errors='strict'):

---> 16 return codecs.utf_8_decode(input, errors, True)

18 class IncrementalEncoder(codecs.IncrementalEncoder):

**UnicodeDecodeError: 'utf8' codec can't decode bytes in position 96-97: invalid continuation byte**

任何提示如何解决这个问题呢？在此先感谢。

2017-01-02