Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

cut words one by one

See original GitHub issue

hi @rkcosmos

when I ran the example, I got the a different result.

>>> import deepcut
Using TensorFlow backend.
2017-10-30 18:32:12.384678: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-30 18:32:12.384717: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-30 18:32:12.384726: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-10-30 18:32:12.384732: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-10-30 18:32:12.384739: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-10-30 18:32:12.679174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.6325
pciBusID 0000:02:00.0
Total memory: 10.91GiB
Free memory: 10.50GiB
2017-10-30 18:32:12.910366: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x5583d11dd3f0 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-10-30 18:32:12.911434: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 1 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.6325
pciBusID 0000:03:00.0
Total memory: 10.91GiB
Free memory: 10.75GiB
2017-10-30 18:32:12.912357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 1
2017-10-30 18:32:12.912376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y Y
2017-10-30 18:32:12.912415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 1:   Y Y
2017-10-30 18:32:12.912429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0)
2017-10-30 18:32:12.912439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0)
>>> print(" ".join(deepcut.tokenize(u'ตัดคำได้ดีมาก')))
ต ั ด ค ำ ไ ด ้ ด ี ม า ก
>>> print(deepcut.tokenize('ตัดคำได้ดีมาก'))
['\xe0', '\xb8', '\x95', '\xe0', '\xb8', '\xb1', '\xe0', '\xb8', '\x94', '\xe0', '\xb8', '\x84', '\xe0', '\xb8', '\xb3', '\xe0', '\xb9', '\x84', '\xe0', '\xb8', '\x94', '\xe0', '\xb9', '\x89', '\xe0', '\xb8', '\x94', '\xe0', '\xb8', '\xb5', '\xe0', '\xb8', '\xa1', '\xe0', '\xb8', '\xb2', '\xe0', '\xb8', '\x81']

I install deepcut by pip. Am I missing some configurations?

Issue Analytics

State:
Created 6 years ago
Comments:8 (4 by maintainers)

Top GitHub Comments

2reactions

tienlemanhcommented, Nov 12, 2017

@titipata I already fixed it. It works now. I changed code from str text to unicode text in everywhere. Should I create a pull request, then you can see?

1reaction

tienlemanhcommented, Nov 10, 2017

@titipata Python 2.7.9 tensorflow 1.2.1

>> pip show tensorflow
Name: tensorflow
Version: 1.2.1
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
License: Apache 2.0
Location: /usr/local/lib/python2.7/dist-packages
Requires: werkzeug, markdown, six, html5lib, protobuf, backports.weakref, wheel, numpy, mock, bleach