Error compiling latest yara-python code

[mrbill321]

Hi,

I'm trying to compile the latest yara-python code (version 4.2.3), and I'm getting a compilation error. I'm doing the build on a system running Rocky Linux 8.5. These are the commands I'm using to build the code:

git clone --recursive https://github.com/VirusTotal/yara-python
cd yara-python
python3 setup.py build

These are the compilation errors I'm getting:

gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -D_GNU_SOURCE=1 -DUSE_LINUX_PROC=1 -DHAVE_STDBOOL_H=1 -DHAVE_MEMMEM=1 -DHASH_MODULE=1 -DHAVE_LIBCRYPTO=1 -Iyara/libyara/include -Iyara/libyara/ -I. -I/usr/include/python3.6m -c yara/libyara/tlshc/tlsh.c -o build/temp.linux-x86_64-3.6/yara/libyara/tlshc/tlsh.o -std=c99
In file included from yara/libyara/tlshc/tlsh.c:3:
yara/libyara/tlshc/tlsh_impl.h:61:28: error: ‘TLSH_CHECKSUM_LEN’ undeclared here (not in a function)
unsigned char checksum[TLSH_CHECKSUM_LEN];
^~~~~~~~~~~~~~~~~
yara/libyara/tlshc/tlsh_impl.h:72:28: error: ‘CODE_SIZE’ undeclared here (not in a function); did you mean ‘FD_SETSIZE’?
unsigned char tmp_code[CODE_SIZE];
^~~~~~~~~
FD_SETSIZE
error: command 'gcc' failed with exit status 1

I successfully built yara 4.2.3 from source earlier; I'm only getting this error when building yara-python.

While looking into the error, I found the following lines in yara/configure.ac, which appear to set the proper variables in order to define the missing TLSH_CHECKSUM_LEN and CODE_SIZE variables:

Configure TLSH function

CFLAGS="$CFLAGS -DBUCKETS_128=1 -DCHECKSUM_1B=1"

These extra flags don't appear to be passed along to gcc, however.

Any ideas would be greatly appreciated.

Thanks,
-Bill

[GlennHD]

Same compilation error for me (v4.2.3) on Debian 11.

[wxsBSD]

I have a PR up which includes this fix (and some other major changes) that I need to update hopefully later this week.

[melihonem]

is there any update about the fix of this issue ?

[garanews]

same issue here

[wxsBSD]

You can make the changes from https://github.com/VirusTotal/yara-python/pull/210/files into your copy if you want. You only need the two lines in setup.py - everything else is not relevant for this issue.

The fix itself is ready for review but it mixes this build fix (the two lines from setup.py) with a rather intrusive change, so it may take some time. If you want to build 4.2.3 you can build the v4.2.x branch which should build fine (it doesn't have the authenticode changes).

[garanews]

@wxsBSD I'm building inside docker, compiling the v4.2.x it worked with any issue.
About compiling master branch, I forked it and modified the setup.py as you suggested but I am getting this error:

#0 8.250 yara/libyara/modules/pe/authenticode-parser/authenticode.c:22:10: fatal error: openssl/asn1.h: No such file or directory
#0 8.250  #include <openssl/asn1.h>
#0 8.250           ^~~~~~~~~~~~~~~~
#0 8.250 compilation terminated.
#0 8.252 error: command '/usr/bin/gcc' failed with exit code 1
------
failed to solve: executor failed running [/bin/sh -c python setup.py install]: exit code: 1

Relevant Dockerfile rows:

FROM common-base as base-builder
RUN apt-get update \
  # ldap support
  && apt-get install --no-install-recommends -y libsasl2-dev libldap2-dev libssl-dev \
  # utils
  curl unzip \
  # requirement to compile yara
  automake libtool make gcc pkg-config flex bison libssl-dev libjansson-dev libmagic-dev \
  # cleaning up unused files
  && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
  && rm -rf /var/lib/apt/lists/*
RUN pip install cryptography  #this added after attempts to compile with your modify
# Build yara and yara-python from sources
FROM base-builder as yara-builder
WORKDIR /tmp
RUN git clone --recursive https://github.com/VirusTotal/yara.git
WORKDIR /tmp/yara
RUN ./bootstrap.sh \
  && ./configure --enable-cuckoo \
  --enable-magic \
  --enable-dotnet \
  --with-crypto \
  && make \
  && make install \
  && echo "Install yara-python..."
WORKDIR /tmp
RUN git clone --recursive https://github.com/garanews/yara-python
WORKDIR /tmp/yara-python
RUN python setup.py build

Any idea?

[wxsBSD]

Looks like you need openssl-dev installed (or whatever provides the openssl headers and libraries for your system).

[alexander-pick]

Got the same issue and just reseted the sub module yara to another tag. Go into yara-python/yara/ and do a git reset --hard d5a7565. Given you have initialized the sub modules correctly it should build and work. I selected this tag because it had a green checkmark, other ones had a red cross and said something about failed checks.

[mobiusmalware]

@alexander-pick This worked for me. Phenomenal.

[doomedraven]

i have fixed that in this way sed -i "191 i \ \ \ \ # Needed to build tlsh'\n module.define_macros.extend([('BUCKETS_128', 1), ('CHECKSUM_1B', 1)])\n # Needed to build authenticode parser\n module.libraries.append('ssl')" setup.py

Here is my comment and solution #conda install sfe1ed40::yara-python or copy paste everything to yara-python's yara folder offical yara project (https://github.com/VirusTotal/yara this thing should be at yara-python's empty yara folder) I using 4.4.0 version