Creating RPM package of Elastic Filebeat

When You want to create custom RPM package of what ever software then follow these steps. In following guide we will try to create custom RPM package of elastic Filebeat.

Filebeat offers light way way of sending log with different providers (i.e Kafka).

Also I need to refer to StackOverflow answer on creating RPM packages. That helped me a lot in right direction.

First things first

If You already do not have access to some Linux distribution with RPM packaging system then our best option is to find Vagrant box from HashiCorp Atlas. From there search for CentOS. I found CentOS 7.2 box next copy the command from box page with appropriate provider. In my case it’s Parallels but by default in most cases is VirtualBox.

vagrant init boxcutter/centos72; vagrant up --provider parallels

After the command has completed we can just run vagrant ssh to login to machine.

Prepare CentOS for RPM package building

We need to check that rpm-build and required directories have been created.

Let’s install rpm-build and redhat-rpm-config

sudo yum install rpm-build -y
sudo yum install redhat-rpm-config.-y

Next we need to create directories for RPM package building.

mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cat <<EOF >~/.rpmmacros
%_topdir   %(echo $HOME)/rpmbuild
%_tmppath  %{_topdir}/tmp
EOF

Package Filebeat into RPM

Now we are ready to create our first RPM package. We will download and unpack Filebeat linux distribution. We also can remove the downloaded TAR.GZ file after unpacking we do not need it anymore.

mkdir ~/filebeat
wget -P ~/filebeat https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.4.0-linux-x86_64.tar.gz
cd ~/filebeat && tar xf filebeat-*.tar.gz && rm filebeat*.tar.gz

Next we will add configuration changes to filebeat.yml and add filebeat.sh file and package up the changed Filebeat to TAR again.

NOTE 1 The new configuration in this case adds Apache Kafka as output source.

NOTE 2 Will plan to write another post about how to setup Apache Kafka and Filebeat logging with Docker.

Updated filebeat.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
output.kafka:
  hosts: ["broker:9092"]
  topic: '%{[type]}'
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

Added filebeat.sh file.

#!/bin/bash

# Script to run filebeat in foreground with the same path settings that
# the init script / systemd unit file would do.

/usr/share/filebeat/bin/filebeat \
  -path.home /usr/share/filebeat \
  -path.config /etc/filebeat \
  -path.data /var/lib/filebeat \
  -path.logs /var/log/filebeat \
  $@

Package up the changes into .tar.gz archive and move to rpmbuild/SOUCES directory.

cd ~/filebeat
mv ~/filebeat/filebeat-5.4.0-linux-x86_64/ ~/filebeat/filebeat-5.4.0-acme_v1 
tar czf filebeat-5.4.0-acme_v1.tar.gz filebeat-5.4.0-acme_v1
mv filebeat-*.tar.gz ~/rpmbuild/SOURCES/ 

Create RPM build specification

IMPORTANT The packaged directory name must match {name}-%{version} combination. Also TAR file name must match the combination of %{name}-%{version}-%[release}` combination.

cat <<EOF > ~/rpmbuild/SPECS/filebeat.spec
# Don't try fancy stuff like debuginfo, which is useless on binary-only
# packages. Don't strip binary too
# Be sure buildpolicy set to do nothing
%define        __spec_install_post %{nil}
%define          debug_package %{nil}
%define        __os_install_post %{_dbpath}/brp-compress

Summary: Custom Filebeat log transfer system service
Name: filebeat
Version: 5.4.0
Release: acme_v1
License: Apache-2.0
Group: System/Monitoring
Source: %{name}-%{version}-%{release}.tar.gz
Url: https://www.elastic.co/products/beats/filebeat
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root
Requires: logrotate

%description
%{summary}

%prep
%setup -q -n %{name}-%{version}-%{release}

%build
# Empty section.

%install

## usr
## Add shell script 
%{__install} -d -m 755 %{buildroot}%{_bindir}/
mv filebeat.sh %{buildroot}%{_bindir}/filebeat
chmod +x %{buildroot}%{_bindir}/filebeat

## Add Filebeat binary to /usr/share
%{__install} -d -m 755 %{buildroot}/usr/share/%{name}/
%{__install} -d -m 755 %{buildroot}/usr/share/%{name}/bin
mv filebeat %{buildroot}/usr/share/%{name}/bin

## Add Filebeat modules
%{__install} -d -m 755 %{buildroot}/usr/share/%{name}/module
cp -rp module/* %{buildroot}/usr/share/%{name}/module/

## etc
%{__install} -d -m 755 %{buildroot}%{_sysconfdir}/%{name}/
%{__install} -m 644 filebeat.yml %{buildroot}%{_sysconfdir}/%{name}/

## var
%{__install} -d -m 755 %{buildroot}/var/lib/%{name}/

%files
%defattr(-,root,root)

%{_bindir}/%{name}
%dir /etc/%{name}/
%config(noreplace) /etc/%{name}/*
%dir /usr/share/%{name}
%doc /usr/share/%{name}/*
%dir /var/lib/%{name}

%changelog
* Wed May 31 2017  John Smith <john.smith@acme.com> 5.4.0-acme_v1
- Added custom filebeat.yml configuration

EOF

Build Filebeat project into RPM package

Now we are ready to build our custom Filebeat RPM package by running rpmbuild command.

rpmbuild -ba ~/rpmbuild/SPECS/filebeat.spec

Test Your built RPM file

Finally we are ready to test out our custom made filebeat. But first we need to install it.

cd ~/rpmbuidl/RPMS/x86_64
sudo rpm -ivh filebeat-5.4.0-acme_v1.x86_64.rpm

Now we can start the filebeat in standalone mode

filebeat

NOTE If there is no output everything works. Check logs under /var/log/filebeat/filebeat to see if there where errors starting the filebeat

Written on June 1, 2017