ImportError: cannot import name 'etree'

[codecakes]

 $HOME/bin/bazel run --spawn_strategy=standalone :dance --copt --aspects=@bazel_tools//tools/python:srcs_version.bzl%find_requirements --verbose_failures=true --show_timestamps=true --python_version=PY3 --build_python_zip --sandbox_debug --color=yes --curses=yes --jobs=100 --loading_phase_threads=HOST_CPUS
(20:00:52) INFO: Current date is 2019-07-27
(20:01:40) INFO: Analysed target //:dance (17 packages loaded, 1500 targets configured).
(20:01:40) INFO: Found 1 target...
Target //:dance up-to-date:
  bazel-bin/dance-layer.tar
(20:01:42) INFO: Elapsed time: 50.227s, Critical Path: 2.20s
(20:01:42) INFO: 18 processes: 18 local.
(20:01:42) INFO: Build completed successfully, 25 total actions
(20:01:42) INFO: Build completed successfully, 25 total actions
ce7d757b71aa: Loading layer [==================================================>]  13.57MB/13.57MB
46ac83f79a96: Loading layer [==================================================>]  92.16kB/92.16kB
01cc2aecb2a3: Loading layer [==================================================>]  849.9kB/849.9kB
06bc60df9414: Loading layer [==================================================>]  40.96kB/40.96kB
9adbd1b6c94b: Loading layer [==================================================>]  10.24kB/10.24kB
3dcc846285f9: Loading layer [==================================================>]  10.24kB/10.24kB
2fc67700bf75: Loading layer [==================================================>]  30.72kB/30.72kB
7f5282e3470a: Loading layer [==================================================>]  10.24kB/10.24kB
77b5f1f80a39: Loading layer [==================================================>]  19.38MB/19.38MB
Loaded image ID: sha256:55918ef775fbb433dacb1dc6b381ae8c99fe181bef1c3ef435efbd25e87d10e2
Tagging 55918ef775fbb433dacb1dc6b381ae8c99fe181bef1c3ef435efbd25e87d10e2 as bazel:dance
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_lat0ql0u/runfiles/grow_rules_python/dance.py", line 1, in <module>
    from scrapy.crawler import CrawlerProcess
  File "/tmp/Bazel.runfiles_lat0ql0u/runfiles/pypi__Scrapy_1_6_0/scrapy/__init__.py", line 34, in <module>
    from scrapy.spiders import Spider
  File "/tmp/Bazel.runfiles_lat0ql0u/runfiles/pypi__Scrapy_1_6_0/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/tmp/Bazel.runfiles_lat0ql0u/runfiles/pypi__Scrapy_1_6_0/scrapy/http/__init__.py", line 11, in <module>
    from scrapy.http.request.form import FormRequest
  File "/tmp/Bazel.runfiles_lat0ql0u/runfiles/pypi__Scrapy_1_6_0/scrapy/http/request/form.py", line 11, in <module>
    import lxml.html
  File "/tmp/Bazel.runfiles_lat0ql0u/runfiles/pypi__lxml_4_4_0/lxml/html/__init__.py", line 53, in <module>
    from .. import etree
ImportError: cannot import name 'etree'

[codecakes]

Not exactly related to this and I'ven't looked under the hood but Is there a way pip_import() looks at dependencies and automatically installs missing dependencies?

[JRodDynamite]

@codecakes - Hey, I seem to be facing the same issue. Did you resolve this or have a workaround for it?

[thundergolfer]

@JRodDynamite I would suspect the etree module is a .so shared object file (or .dll on Windows I think) and there's a platform-mismatch issue between install and runtime.

Check under the equivalent of /tmp/Bazel.runfiles_lat0ql0u/runfiles/pypi__lxml_4_4_0/lxml/ for etree and check first if it's present and if so what platform tagging is present in the filename.

[JRodDynamite]

I do see the .so file present and since I'm running this on CentOS 7, it's tagging the correct OS in the filename (as well as architecture).

> ll bazel-bin/generateds/generateds_test.runfiles/generateds_deps/pypi__lxml/lxml/etree.cpython-36m-x86_64-linux-gnu.so
lrwxrwxrwx 1 jason developer 149 2021-03-08 11:37:31.304455246 +0530 bazel-bin/generateds/generateds_test.runfiles/generateds_deps/pypi__lxml/lxml/etree.cpython-36m-x86_64-linux-gnu.so -> /home/jason/.cache/bazel/_bazel_jason/f355dbcb22c82751e0ccf52a1cd8bb66/external/generateds_deps/pypi__lxml/lxml/etree.cpython-36m-x86_64-linux-gnu.so

[JRodDynamite]

@thundergolfer - I think you are right, my install and runtime python is different. I'm using a custom Python toolchain. I modified my .bazelrc to set the --action_env to the path of my custom Python. And that resolved my issue.

I don't think setting --action_env is the correct way to fix this. I'm trying to resolve that by passing the interpreter from the custom toolchain to pip_install. Once I can get that working, I should be able to resolve this as well. Thanks! 👍
https://bazelbuild.slack.com/archives/CA306CEV6/p1614939170006500

[thundergolfer]

I don't think setting --action_env is the correct way to fix this.

Yeah stick to configuring py_runtime for the toolchain and passing the same interpreter to pip_install via interpreter = attr.

Without setting up this, you're at the mercy of whatever the workspace rule or the py_* rules find on the $PATH, and some people have pyenv and brew causing all sort of shenanigans with interpreter setup via $PATH configuration.

Since you have solved you problem and @codecakes hasn't followed up in a while, I'll close this issue for now 👍

[PepeGonza]

Hello! I am having a similar issue to this. I have a docker container running on a centos:7 image. This image uses bazel to build a python zip. This zip is ran directly in an ec2 running centos 7. But having the following error:

+ python3 /var/lib/jenkins/tools/0.4_docker_publisher_wrapper.zip docker_publish --results '.ee/dockerPublisherResults*.txt' -vvv
Traceback (most recent call last):
  File "/tmp/Bazel.runfiles_36od4v7x/runfiles/__main__/src/publish/docker_publisher/bin/docker_publisher_wrapper.py", line 8, in <module>
    from src.publish.docker_publisher.docker_publisher import DockerPublisher
  File "/tmp/Bazel.runfiles_36od4v7x/runfiles/__main__/src/publish/docker_publisher/docker_publisher.py", line 7, in <module>
    from src.publish.docker_publisher.ecr_helper import ECRHelper
  File "/tmp/Bazel.runfiles_36od4v7x/runfiles/__main__/src/publish/docker_publisher/ecr_helper.py", line 3, in <module>
    from src.publish.docker_publisher.docker_publisher_context import DockerPublisherContext
  File "/tmp/Bazel.runfiles_36od4v7x/runfiles/__main__/src/publish/docker_publisher/docker_publisher_context.py", line 6, in <module>
    from src.pom_manager.pom.pom_worker import PomWorker
  File "/tmp/Bazel.runfiles_36od4v7x/runfiles/__main__/src/pom_manager/pom/pom_worker.py", line 8, in <module>
    from lxml.etree import Element, ElementTree, XML, parse, SubElement, Comment
ModuleNotFoundError: No module named 'lxml.etree'

Any advice on this? Thanks

[thundergolfer]

@PepeGonza it's most likely a platform mismatch problem. The .zip was built on platform OS X but was shipped into a centos:7 image running OS Y and the shared library file is not the right one.

[ahoyaharr]

Apologies for posting on an old/closed issue, but I am running into the a similar problem.

When I try to run my executable zip in a CentOS container, it seems like my dependencies that are not pure Python (LXML is a binding for some C libraries, and I'm having similar problems with CFFI) are breaking because only the Python pieces are being packaged.

The zip is being built in a different CentOS container, and all my other zips which contain only pure Python dependencies are running fine.

Does anyone have advice on this?