Hi!
I’ve tried to install KBQA model in Windows, but got the same unresolvable problem as mentioned here: Can't install KBQA due to package hdt installation error
So I’ve installed Ubuntu 18 in VirtualBox to try Linux environment with KBQA.
After installing deeppavlov, missing packages (BERT, python3-testresources, squad_zh_bert_zh, etc) my config was:
DeepPavlov: 0.12.0
Python 3.6.9
I tried to run sample code which is provided in your documentation here (Knowledge Base Question Answering (KBQA) — DeepPavlov 0.14.0 documentation):
from deeppavlov import configs, build_model
if name == ‘main’:
kbqa_model = build_model(configs.kbqa.kbqa_cq_rus, download = True)
kbqa_model([‘Когда родился Пушкин?’])
after downloading all data program terminated without any result (apart from downloaded files). Log was following:
2020-09-03 13:32:04.830 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/kbqa/datasets/entity_and_type_detection_rus.pickle download because of matching hashes
2020-09-03 13:32:36.132 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/deeppavlov_data/syntax_parser/syntax_ru_syntagrus_bert.tar.gz download because of matching hashes
2020-09-03 13:38:13.549 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/kbqa/wikidata/wikidata.hdt.index.v1-1 download because of matching hashes
2020-09-03 13:38:13.667 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/kbqa/wikidata/sparql_queries.json download because of matching hashes
2020-09-03 13:38:33.802 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/deeppavlov_data/bert/multi_cased_L-12_H-768_A-12.zip download because of matching hashes
2020-09-03 13:38:54.956 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/kbqa/models/rel_ranking_bert_rus.tar.gz download because of matching hashes
2020-09-03 13:48:37.406 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/kbqa/wikidata/wikidata.hdt download because of matching hashes
2020-09-03 13:48:45.136 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/kbqa/datasets/rel_ranking_bert_rus.tar.gz download because of matching hashes
2020-09-03 13:49:27.793 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/kbqa/models/ner_cq_rus.tar.gz download because of matching hashes
2020-09-03 13:49:45.785 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/deeppavlov_data/bert/rubert_cased_L-12_H-768_A-12_v1.tar.gz download because of matching hashes
2020-09-03 13:49:48.881 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/deeppavlov_data/morpho_tagger/UD2.3/ru_syntagrus.tar.gz download because of matching hashes
2020-09-03 13:50:10.0 INFO in 'deeppavlov.download'['download'] at line 132: Skipped http://files.deeppavlov.ai/kbqa/wikidata/wiki_rus_files.tar.gz download because of matching hashes
[nltk_data] Downloading package punkt to /home/va/nltk_data...
[nltk_data] Package punkt is already up-to-date!
[nltk_data] Downloading package stopwords to /home/va/nltk_data...
[nltk_data] Package stopwords is already up-to-date!
[nltk_data] Downloading package perluniprops to /home/va/nltk_data...
[nltk_data] Package perluniprops is already up-to-date!
[nltk_data] Downloading package nonbreaking_prefixes to
[nltk_data] /home/va/nltk_data...
[nltk_data] Package nonbreaking_prefixes is already up-to-date!
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/bert_dp/tokenization.py:125: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
2020-09-03 13:50:42.985 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 115: [loading vocabulary from /home/va/.deeppavlov/models/ner_ent_and_type_rus/tag.dict]
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/core/models/tf_model.py:37: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/core/models/tf_model.py:222: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/core/models/tf_model.py:222: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/core/models/tf_model.py:193: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/models/bert/bert_sequence_tagger.py:236: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
2020-09-03 13:50:43.547058: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3400030000 Hz
2020-09-03 13:50:43.547629: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5566ff0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-03 13:50:43.547695: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/models/bert/bert_sequence_tagger.py:314: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/bert_dp/modeling.py:178: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/bert_dp/modeling.py:418: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/bert_dp/modeling.py:499: The name tf.assert_less_equal is deprecated. Please use tf.compat.v1.assert_less_equal instead.
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
* https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/bert_dp/modeling.py:366: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/bert_dp/modeling.py:680: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.Dense instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/tensorflow_core/python/layers/core.py:187: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/bert_dp/modeling.py:283: The name tf.erf is deprecated. Please use tf.math.erf instead.
WARNING:tensorflow:Variable *= will be deprecated. Use `var.assign(var * other)` if you want assignment to the variable value or `x = x * y` if you want a new python Tensor object.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/models/bert/bert_sequence_tagger.py:75: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/models/bert/bert_sequence_tagger.py:571: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/core/models/tf_model.py:131: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/core/models/tf_model.py:131: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/core/models/tf_model.py:94: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/tensorflow_core/python/training/moving_averages.py:433: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/models/bert/bert_sequence_tagger.py:671: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/models/bert/bert_sequence_tagger.py:244: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.
2020-09-03 13:51:12.663763: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 367248384 exceeds 10% of system memory.
2020-09-03 13:51:13.223880: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 367248384 exceeds 10% of system memory.
2020-09-03 13:51:16.613835: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 367248384 exceeds 10% of system memory.
2020-09-03 13:51:17.167350: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 367248384 exceeds 10% of system memory.
2020-09-03 13:51:18.890928: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 367248384 exceeds 10% of system memory.
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/models/bert/bert_sequence_tagger.py:249: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2020-09-03 13:53:45.104 INFO in 'deeppavlov.core.models.tf_model'['tf_model'] at line 51: [loading model from /home/va/.deeppavlov/models/ner_ent_and_type_rus/model]
WARNING:tensorflow:From /home/va/.local/lib/python3.6/site-packages/deeppavlov/core/models/tf_model.py:54: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
Killed
=======================================================
I’ve tried to install/run this model through Docker but result was still the same.
Can you guide me or tell what I’m doing wrong? I’m fine with any solution (Windows/Linux, pip/Docker).
Feel free to answer in Russian, if it’s faster. I can understand it too 