Threat Intelligence with Open Source Tools - Cornerstones of Trust 2014

Cornerstones of Trust 2014: http://cornerstonesoftrust.com.
Presenters: Jaime Blasco (@jaimeblascob) and Santiago Bassett (@santiagobassett).  Thank you Jaime.

Threat Intelligence has become increasingly important as the number and severity of threats is growing continuously. We live in an era where our prevention technologies are not enough anymore, antivirus products fail to detect new or sophisticated pieces of malware, our firewalls and perimeter defenses are easily bypassed and the attacker’s techniques are growing in complexity. In this new landscape, sharing threat intelligence has become a key component to mitigate cyber-attacks.

In this session we will define what Threat Intelligence is and discuss how to collect and integrate threat intelligence from public sources. In addition, we’ll demonstrate how to build your own Threat Intelligence data using Open Source tools such as sandboxes, honeypots, sinkholes and other publicly available tools.

The industry’s reticence to share information about attack vectors gives the adversary a huge advantage. Using Threat Intelligence we can reduce this advantage and enable preventative response. We will guide you through the different standards (OpenIOC, STIX, MAEC, OTX, IODEF…) to describe and share cyber intelligence, as well as Open Source Frameworks such as CIF (Collective Intelligence Framework) that allows you to combine different threat sources.

One of the biggest problems with Threat Intelligence is finding out how to take advantage of the data you have to actually improve the detection/prevention capabilities in your environment. We will describe how to leverage Threat Intelligence to detect threats and provide defenses, and we will focus on how to use Open Source Tools (Suricata, OSSIM, OSSEC, Bro, Yara…) to get the most of your Threat Intelligence.

OSSEC CON 2014 - Malware Detection with OSSEC

Happy to share my presentation from the OSSEC CON, which took place on September 16th in Cork, Ireland. Here you can find a brief explanation of different malware collection and analysis techniques, as well as a good example of how to use some IOCs to create a rootcheck signature.

And, for those interested, here is the video as well:

Updated OSSEC debian packages

Just published new versions ossec-hids_2.8-2 and ossec-hids-agent_2.8-2, for the different Debian distributions. Those can be found here: http://ossec.alienvault.com/repos/apt/debian/pool/main/o/

Here are the changelogs:

 ossec-hids (2.8-2) stable; urgency=low  
  * Fixed Makefile to use ossec-hids-debian.init instead of ossec-hids.init (fixes LSB headers warning).  
  * Fixed CVE-2014-5284. Patch included.  
  * Included debconf and templates for initial package configuration (email_to, email_from and smtp).  

 ossec-hids-agent (2.8-2) stable; urgency=low  
  * Fixed Makefile to use ossec-hids-debian.init instead of ossec-hids.init (fixes LSB headers warning).  
  * Fixed CVE-2014-5284. Patch included.  
  * Included debconf and templates for initial package configuration (server_ip).  

And the link to the github repo: https://github.com/santiago-bassett/ossec-debian.git

I also took the opportunity to update generate_ossec.sh script, which now accepts a few different arguments:

 santiago@debian-package:~# ./generate_ossec.sh -h  
 USAGE: Command line arguments available:  
 -h | --help Displays this help.  
 -u | --update Updates chroot environments.  
 -b | --build Builds debian packages.  
 -s | --sync Synchronizes with the debian repository.  

Files to create OSSEC HIDS Debian packages

Just published, in Github, the files I used to create OSSEC-HIDS version 2.8 debian packages, the ones included both in ossec.net website and in AlienVault repository. 

https://github.com/santiago-bassett/ossec-debian

You can find these packages at: http://www.ossec.net/?page_id=19

or directly at: http://ossec.alienvault.com/repos/apt/debian/pool/main/o/

There are two different packages that can be built with these files:

• ossec-hids: Package that includes both the server and the agent.
• ossec-hids-agent: Package that includes just the agent.

Each one of the subdirectories includes:

• Makefile
• Debian control files: changelog, compat, control, copyright, lintian-overrides, postinst, postrm, preinst, rules

Additionally a script, generate_ossec.sh, is included to generate the Debian packages for Jessie, Sid and Wheezy Debian distributions, both for i386 and amd64 architectures. This script uses Pbuilder to build the packages, and uploads those to an APT repository, setup with Reprepro.

For more details on how to create Debian Packages and an APT repository you can check my post at:

http://santi-bassett.blogspot.com/2014/07/setting-up-apt-repository-with-reprepro.html

Please don't hesitate to contribute (preferably via pull requests) to improve these packages.

Scripts to inject sample data to AlienVault / OSSIM SIEM

I just published a few scripts I wrote to inject sample data to AlienVault or OSSIM (Open Source Version) Unified SIEM. Those can be found in Github:

https://github.com/santiago-bassett/Alienvault-Demo_scripts

The scripts are ready to emulate Syslog data coming from these sources: Aruba Wireless, Cisco ASA, Cisco PIX, ClamAV, Oracle Database, OSSEC HIDS, Sonicwall and SSH.

As well, the scripts are ready to inject malicious network traffic in a dummy interface so it can be analyzed by Snort NIDS. Some of the traffic injected is related to: botnets, C&C communications, Zeus, spambot or spyware. The pcap files can be found in this directory:

https://github.com/santiago-bassett/Alienvault-Demo_scripts/tree/master/pcaps

Setting up an APT repository with Reprepro and Apache

This post can be considered a continuation of my previous ones:

• How to create a Debian package
• Debian packaging with Pbuilder

Following next steps we will set up a signed Debian repository, using Reprepro and Apache2.

1.- Installing the tools

 root@server:~# apt-get install apache2 dpkg-sig reprepro  

2.- Configuring Apache2 with reprepro

 root@server:~# mkdir -p /var/www/repos/apt/debian  

Then we need to edit /etc/apache2/apache2.conf file, adding the following line:

 ServerName localhost  

Now we need to create /etc/apache2/conf-availabe/repos.conf with this content:

 <Directory /var/www/repos/ >  
     # We want the user to be able to browse the directory manually  
     Options Indexes FollowSymLinks Multiviews  
     Order allow,deny  
     Allow from all  
 </Directory>  
 # This syntax supports several repositories, e.g. one for Debian, one for Ubuntu.  
 # Replace * with debian, if you intend to support one distribution only.  
 <Directory "/var/www/repos/apt/*/db/">  
     Order allow,deny  
     Deny from all  
 </Directory>  
 <Directory "/var/www/repos/apt/*/conf/">  
     Order allow,deny  
     Deny from all  
 </Directory>  
 <Directory "/var/www/repos/apt/*/incoming/">  
     Order allow,deny  
     Deny from all  
 </Directory>  

At last, we need to enable repos site configuration in Apache:

 root@server:/etc/apache2/conf-available# a2enconf repos  
 root@server:/etc/apache2/conf-enabled# apache2ctl configtest  
 Syntax OK  
 root@server:/etc/apache2/conf-enabled# service apache2 restart  

3.- Configuring Reprepro

 root@server:~# mkdir -p /var/www/repos/apt/debian/conf  

Now we create the file /var/www/repos/apt/debian/conf/distributions, with the following content:

 Origin: Debian  
 Label: Sid apt repository  
 Codename: sid  
 Architectures: i386 amd64  
 Components: main  
 Description: Apt repository for Debian unstable - Sid  
 DebOverride: override.sid  
 DscOverride: override.sid  
 SignWith: 870B8E2D  

 Origin: Debian  
 Label: Jessie apt repository  
 Codename: jessie  
 Architectures: i386 amd64  
 Components: main  
 Description: Apt repository for Debian testing - Jessie  
 DebOverride: override.jessie  
 DscOverride: override.jessie  
 SignWith: 870B8E2D  

 Origin: Debian  
 Label: Wheezy apt repository  
 Codename: wheezy  
 Architectures: i386 amd64  
 Components: main  
 Description: Apt repository for Debian stable - Wheezy  
 DebOverride: override.wheezy  
 DscOverride: override.wheezy  
 SignWith: 870B8E2D  

SingWith value, included above, needs to be taken from gpg --list-keys. This will be the key used to sign our packages.

 root@server:~# gpg --list-keys  
 pub  2048R/489CD644 2014-07-15  
 uid         Your Name <your_email_address@domain.com>  
 sub  2048R/870B8E2D 2014-07-15  

At last, we create the options file at /var/www/repos/apt/debian/conf/options:

 verbose  
 basedir /var/www/repos/apt/debian  
 ask-passphrase  

Note: We can also override Debian packages control file fields, by creating override files. In our case, for example, we could create /var/www/repos/apt/debian/conf/override.jessie with this content (used for our package named hello):

 hello         Priority     extra  
 hello         Section      admin  

4.- Adding a package to the repository

 root@server:/var/www/repos/apt/debian# reprepro includedeb jessie /opt/hello_0.1-1_i386.deb  

This will add our package hello_0.1-1_i386.deb to the APT repository. The command above will ask to insert a password, which is the same we used to sign the package. And, in case we want to remove the package from the repository we can use the following command:

 root@server:/var/www/repos/apt/debian# reprepro remove jessie hello  

5.- Providing the public key to repository users

We need to export our public GPG key to a file, and make it available at the web server so users can download it:

 root@server:~# gpg --armor --output public.gpg.key --export Your Name  
 root@server:~# mkdir /var/www/repos/apt/conf  
 root@server:~# cp public.gpg.key /var/www/repos/apt/conf/  

6.- Installing packages using the repository

 wget -O - http://YOUR_REPO_SERVER/repos/apt/conf/public.gpg.key | apt-key add -  
 echo "deb http://YOUR_REPO_SERVER/repos/apt/debian jessie main" >> /etc/apt/sources.list  
 apt-get update && apt-get install YOUR_PACKAGE  

7.- Useful commands

• List packages of a specific distribution: reprepro list DISTRIBUTION_CODENAME
• List packages matching a specific name: reprepro ls PACKAGE_NAME

As well, in case we want to remove a distribution from the repository, we need to:

• Remove it from /var/www/repos/apt/debian/conf/distributions
• Run reprepro delete clearvanished

References

https://wiki.debian.org/SettingUpSignedAptRepositoryWithReprepro

http://www.debian-administration.org/articles/286

https://wikitech.wikimedia.org/wiki/Reprepro

http://blog.jonliv.es/2011/04/26/creating-your-own-signed-apt-repository-and-debian-packages/

Debian packaging with Pbuilder

This post explains how to create chroot environments, for different Debian distributions and system architectures, to build Debian packages. It also includes a description of the packages signing process, so those can later be uploaded to reprepro, an apt-get repository. I decided to write this article as a continuation of the one I wrote explaining how to create Debian packages:

http://santi-bassett.blogspot.com/2014/07/how-to-create-debian-package.html

1.- Installing the tools

 apt-get install pbuilder debian-archive-keyring debootstrap devscripts  

2.- Creating the configuration file at /root/.pbuilder

 # Codenames for Debian suites according to their alias. Update these when  
 # needed.  
 UNSTABLE_CODENAME="sid"  
 TESTING_CODENAME="jessie"  
 STABLE_CODENAME="wheezy"  
 STABLE_BACKPORTS_SUITE="$STABLE_CODENAME-backports"  
 # List of Debian suites.  
 DEBIAN_SUITES=($UNSTABLE_CODENAME $TESTING_CODENAME $STABLE_CODENAME  
   "unstable" "testing" "stable")  
 # List of Ubuntu suites. Update these when needed.  
 UBUNTU_SUITES=("saucy" "raring" "precise")  
 # Mirrors to use. Update these to your preferred mirror.  
 DEBIAN_MIRROR="ftp.us.debian.org"  
 UBUNTU_MIRROR="ftp.ubuntu.com"  
 # Optionally use the changelog of a package to determine the suite to use if  
 # none set.  
 if [ -z "${DIST}" ] && [ -r "debian/changelog" ]; then  
   DIST=$(dpkg-parsechangelog | awk '/^Distribution: / {print $2}')  
   DIST="${DIST%%-*}"  
   # Use the unstable suite for certain suite values.  
   if $(echo "experimental UNRELEASED" | grep -q $DIST); then  
     DIST="$UNSTABLE_CODENAME"  
   fi  
 fi  
 # Optionally set a default distribution if none is used. Note that you can set  
 # your own default (i.e. ${DIST:="unstable"}).  
 #: ${DIST:="$(lsb_release --short --codename)"}  
 : ${DIST:="stable"}  
 # Optionally change Debian release states in $DIST to their names.  
 case "$DIST" in  
   unstable)  
     DIST="$UNSTABLE_CODENAME"  
     ;;  
   testing)  
     DIST="$TESTING_CODENAME"  
     ;;  
   stable)  
     DIST="$STABLE_CODENAME"  
     ;;  
 esac  
 # Optionally set the architecture to the host architecture if none set. Note  
 # that you can set your own default (i.e. ${ARCH:="i386"}).  
 : ${ARCH:="$(dpkg --print-architecture)"}  
 NAME="$DIST"  
 if [ -n "${ARCH}" ]; then  
   NAME="$NAME-$ARCH"  
   DEBOOTSTRAPOPTS=("--arch" "$ARCH" "${DEBOOTSTRAPOPTS[@]}")  
 fi  
 BASETGZ="/var/cache/pbuilder/$NAME-base.tgz"  
 # Optionally, set BASEPATH (and not BASETGZ) if using cowbuilder  
 # BASEPATH="/var/cache/pbuilder/$NAME/base.cow/"  
 DISTRIBUTION="$DIST"  
 BUILDRESULT="/var/cache/pbuilder/$NAME/result/"  
 APTCACHE="/var/cache/pbuilder/$NAME/aptcache/"  
 BUILDPLACE="/var/cache/pbuilder/build/"  
 if $(echo ${DEBIAN_SUITES[@]} | grep -q $DIST); then  
   # Debian configuration  
   MIRRORSITE="http://$DEBIAN_MIRROR/debian/"  
   COMPONENTS="main contrib non-free"  
   DEBOOTSTRAPOPTS=("${DEBOOTSTRAPOPTS[@]}" "--keyring=/usr/share/keyrings/debian-archive-keyring.gpg")  
 elif $(echo ${UBUNTU_SUITES[@]} | grep -q $DIST); then  
   # Ubuntu configuration  
   MIRRORSITE="http://$UBUNTU_MIRROR/ubuntu/"  
   COMPONENTS="main restricted universe multiverse"  
   DEBOOTSTRAPOPTS=("${DEBOOTSTRAPOPTS[@]}" "--keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg")  
 else  
   echo "Unknown distribution: $DIST"  
   exit 1  
 fi  

3.- Creating the chroot environments for different distributions and architectures

For example, to create the chroot for distribution Jessie and architecture i386, we would need to run the following command:

 DIST=jessie ARCH=i386 pbuilder create --mirror ftp://ftp.us.debian.org/debian/ --debootstrapopts "--keyring=/usr/share/keyrings/debian-archive-keyring.gpg"  

The results, as defined in /root/.pbuilder, will be stored at: /var/cache/pbuilder/jessie-i386-base.tgz

Example for Ubuntu:

 DIST=saucy ARCH=i386 pbuilder create --architecture i386 --distribution saucy --debootstrapopts "--keyring=/usr/share/keyrings/ubuntu-archive-keyring.gpg"  

Current possible values for DIST are “sid” (unstable), “jessie” (testing), “wheezy” (stable). Other values can be found at /usr/share/debootstrap/scripts/.

Possible values for ARCH are i386 or amd64.

4.- Updating chroot environments (recommended before building the package)

 DIST=jessie ARCH=i386 pbuilder update  

This command will extract the chroot, invoke "apt-get update" and "apt-get dist-upgrade" inside the chroot, and then recreate the base.tgz (in this case jessie-i386-base.tgz).

This is recommended to avoid errors while building the packages. For example when apt-get doesn’t find packages that are used as dependencies.

5.- Building the package inside the chroot environment

From your source code directory, the one that contains the Debian files (in our case /opt/hello-0.1/), run:

 /usr/bin/pdebuild --use-pdebuild-internal --architecture i386 --buildresult /var/cache/pbuilder/jessie-i386/result/ \  
 -- --basetgz /var/cache/pbuilder/jessie-i386-base.tgz --distribution jessie --architecture i386 --aptcache \  
 /var/cache/pbuilder/jessie-i386/aptcache/ --override-config  

The command pdebuild calls dpkg-source to build the source package, and then invokes pbuilder on the resulting source package. Results are stored in the --buildresult directory. 

Pbuilder, which is run as part of pdebuild command, extracts the base.tgz to a temporary working directory, enters the directory with chroot, satisfies the build-dependencies inside the chroot, and builds the package.

According to manual pdebuild should be called this way: 

pdebuild [pdebuild options] -- [pbuilder options]

For more info see:

• man pdebuild
• man pbuilder

6.- Generating GPG key and signing package

We need to have a GPG key (can be listed with gpg --list-keys command), or generate a new one, to sign the packages. Then we can sing the package with debsign:

 root@debian-package:/opt# gpg --gen-key  
 ...  
 pub  2048R/489CD644 2014-07-15  
    Key fingerprint = 39F8 7126 FC58 3272 9A8D 04AB B701 2A82 489C D644  
 uid         Your Name <your_email_address@domain.com>  
 sub  2048R/870B8E2D 2014-07-15  

 root@debian-package:/opt# debsign -k870B8E2D hello_0.1-1_i386.changes   
  signfile hello_0.1-1.dsc 870B8E2D  
 You need a passphrase to unlock the secret key for  
 user: "Your Name <your_email_address@domain.com>"  
 2048-bit RSA key, ID 489CD644, created 2014-07-15  
  signfile hello_0.1-1_i386.changes 870B8E2D  
 You need a passphrase to unlock the secret key for  
 user: "Your Name <your_email_address@domain.com>"  
 2048-bit RSA key, ID 489CD644, created 2014-07-15  
 Successfully signed dsc and changes files  

7.- Testing the Debian package inside the chroot environment

We can easily get access to a shell inside the chroot environment using --login option:

 root@debian-package:/opt# pbuilder --login --basetgz /var/cache/pbuilder/jessie-i386-base.tgz --distribution jessie --architecture i386 --bindmounts "/var/cache/pbuilder/jessie-i386/result" --override-config  
 I: Building the build Environment  
 I: extracting base tarball [/var/cache/pbuilder/jessie-i386-base.tgz]  
 I: creating local configuration  
 I: copying local configuration  
 I: Installing apt-lines  
 I: mounting /proc filesystem  
 I: mounting /dev/pts filesystem  
 I: Mounting /var/cache/pbuilder/jessie-i386/result  
 I: policy-rc.d already exists  
 I: Obtaining the cached apt archive contents  
 I: entering the shell  
 File extracted to: /var/cache/pbuilder/build//27119  

 root@debian-package:/#  

Note: Using --safe-after-login option would save changes in the chroot environment after login out of it.

Now we can go to the results directory, specified with --bindmounts option, and install the package.

 root@debian-package:~# cd /var/cache/pbuilder/jessie-i386/result/  
 root@debian-package:/var/cache/pbuilder/jessie-i386/result# dpkg -i hello_0.1-1_i386.deb  
 Selecting previously unselected package hello.  
 (Reading database ... 12428 files and directories currently installed.)  
 Preparing to unpack hello_0.1-1_i386.deb ...  
 Unpacking hello (0.1-1) ...  
 Setting up hello (0.1-1) ...  
 root@debian-package:/var/cache/pbuilder/jessie-i386/result# hello_world   
 Hello World  

On the other hand, in case other packages are needed as dependencies or for testing, we can run apt-get inside the chroot to install those. As well, more variants and custom packages can be added to debootstrap, as explained here:
http://askubuntu.com/questions/168158/how-do-i-add-more-variants-or-custom-packages-to-debootstrap

8.- References

http://www.netfort.gr.jp/~dancer/software/pbuilder-doc/pbuilder-doc.html

https://wiki.ubuntu.com/PbuilderHowto

How to create a Debian package

I have actually found really useful documentation in the Internet (see references section below) that explains the package creation process in great detail. Nevertheless I thought it could make sense for me to put together a simplified tutorial, using a simple "hello world" program as an example.

As well, as a continuation of this post, I will write another article explaining how to use Pbuilder to compile the binary package for the different Debian distributions, currently wheezy, jessie and sid, using chroot environments.

For this purpose I've used a 64 bits server with Debian wheezy, current stable, version 7.4 (you can check yours at /etc/debian_version). This way I've been able to create packages for both i386 and amd64 architectures.

1.- The package creation process

There are several tools involved in this process, used to build, check and sign the package. Debuild is a wrapper that will call them appropriately, so we don't need to do it manually. Here is a brief description of the tools it invokes:

• dpkg-buildpackage: It creates a temporary directory with the package files, building later the .deb package with its content. To work properly, it heavily relies on the files in the special debian subdirectory: control, rules, changelog, etc.
• lintian: Dissects Debian packages trying to find bugs or policy violations.
• debsign: Signs packages (.dsc and .changes files) using GPG or PGP.

2.- Installing necessary software to build our packages

 apt-get install dh-make build-essential
 apt-get install devscripts fakeroot debootstrap pbuilder

3.- Setting up environment variables

 DEBEMAIL="your_email_address@domain.com"  
 DEBFULLNAME="Your Name"  
 export DEBEMAIL DEBFULLNAME  

4.- Uncompressing our source code (format of the tar.gz file is software-version.tar.gz)

In my case, for the purpose of this little how-to, I will build the Debian package for a simple "hello world" program written in C.

 root@debian-package:/opt# tar -xvzf hello-0.1.tar.gz   
 hello-0.1/  
 hello-0.1/Makefile  
 hello-0.1/hello_world.c  

hello_world.c:

 #include <stdio.h>  
 main ()  
 {  
      printf("Hello World");  
 }  

Makefile:

 DESTDIR=/  
 INSTALL_LOCATION=$(DESTDIR)/usr/  
 CFLAGS:=$(shell dpkg-buildflags --get CFLAGS)  
 LDFLAGS:=$(shell dpkg-buildflags --get LDFLAGS)  
 all: hello_world  
 hello_world: hello_world.o  
      cc $(CFLAGS) $(LDFLAGS) -o $@ hello_world.o  
 install: hello_world_install  
 hello_world_install:  
      mkdir -p $(INSTALL_LOCATION)/bin  
      cp hello_world $(INSTALL_LOCATION)/bin  
      chmod 755 $(INSTALL_LOCATION)/bin/hello_world  
 clean:  
      rm -f *.o hello_world   

There are a two interesting things that we can see in our Makefile:

• It uses DESTDIR variable to support the DESTDIR convention. 
• Dpkg-buildflags is used to get C compiler options (CFLAGS) as well as linker options (LDFLAGS). This complies with the hardening requirements described in Debian documentation.

On the other hand, if your software has any external dependencies, you would need to install those, so you can compile it successfully. Typically you would be able to install them using apt-get.

5.- Building the Debian files skeleton

 root@debian-package:/opt# cd hello-0.1  
 root@debian-package:/opt/hello-0.1# dh_make -f ../hello-0.1.tar.gz   
 Type of package: single binary, indep binary, multiple binary, library, kernel module, kernel patch?  
  [s/i/m/l/k/n] s  
 Maintainer name : Your name
 Email-Address  : your_email_address@domain.com   
 Date       : Tue, 24 Jun 2014 21:50:02 +0000  
 Package Name   : hello  
 Version     : 0.1  
 License     : blank  
 Type of Package : Single  
 Hit <enter> to confirm:   
 Done. Please edit the files in the debian/ subdirectory now. You should also  
 check that the hello Makefiles install into $DESTDIR and not in / .  

As we only want to build a single binary package, I chose that option. Multiple binary package option would in fact, build multiple .deb packages.

Now we have a new directory, called "debian", with all the necessary Debian files that we need to build our package, including examples. This includes important files like:

•  control: includes meta data about the package
•  rules: specifies how the package is going to be built
•  changelog: history of the debian package
•  copyright: copyright information

As well, some other example files are created by dh_make, that we won't use at this point and can be deleted safely.

 root@debian-package:/opt/hello-0.1/debian# rm -f *.ex *.EX README.*  
 root@debian-package:/opt/hello-0.1/debian# ls  
 changelog compat control copyright docs rules source  

6.- Control file

The control file has two sections, the first part refers to the source package and the second to the binary one. More information about the different fields can be found in deb-control manual page.

 Source: hello  
 Maintainer: Your Name <your_email_address@domain.com>   
 Build-Depends: debhelper (>= 8.0.0)   
 Standards-Version: 3.9.3   
 Section: utils
  
 Package: hello  
 Priority: extra  
 Architecture: any   
 Depends: ${shlibs:Depends}, ${misc:Depends}  
 Description: Test package for hello world  
  This software literally prints "hello world".   

The variable ${shlibs:Depends} will be substituted by the shared library dependencies needed to build our binary package. Those are calculated automatically by dh_shlibdeps, one of the tools of the debhelper suite.

7.- Changelog file

 root@debian-package:/opt/hello-0.1# cat debian/changelog   
 hello (0.1-1) unstable; urgency=low  
  * Initial release (Closes: #100) # there was no previous ITP  
  -- Your Name <your_email_address@domain.com> Wed, 25 Jun 2014 19:50:08 +0000  

ITP stands for Intend to Package and, for our package to be included in a Debian distribution, the changelog file should close an existing bug. For our example we closed bug #100, this way we won't see lintian warnings requiring for this number later.

Note: We can use "dch -i" command to edit our changelog file.

8.- Copyright file

 root@debian-package:/opt/hello-0.1# cat debian/copyright   
 Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/  
 Upstream-Name: hello  
 Files: *  
 Copyright: 2014 Your Name <your_email_address@domain.com>  
 License: GPL-2  
  This package is free software; you can redistribute it and/or modify  
  it under the terms of the GNU General Public License as published by  
  the Free Software Foundation; either version 2 of the License, or  
  (at your option) any later version.  
  .  
  This package is distributed in the hope that it will be useful,  
  but WITHOUT ANY WARRANTY; without even the implied warranty of  
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the  
  GNU General Public License for more details.  
  .  
  You should have received a copy of the GNU General Public License  
  along with this program. If not, see <http://www.gnu.org/licenses/>  
  .  
  On Debian systems, the complete text of the GNU General  
  Public License version 2 can be found in "/usr/share/common-licenses/GPL-2".  

9.- Rules file

The rules file invokes the original software Makefile script, as well as the debhelper suite of tools (with the prefix "dh_). These tools handle different tasks, including the creation of the .deb file (dh_builddeb).

 #!/usr/bin/make -f  
 # -*- makefile -*-  
 # Uncomment this to turn on verbose mode.  
 #export DH_VERBOSE=1  
 %:  
     dh $@ 

The rules file can be run with different targets: clean (invokes make clean), build (invokes make) and binary (invokes make install). Usage of fakeroot command is recommended so you don't need to build your packages as root.

 root@debian-package:/opt/hello-0.1# fakeroot debian/rules clean  
 dh clean   
   dh_testdir  
   dh_auto_clean  
 make[1]: Entering directory `/opt/hello-0.1'  
 rm -f *.o hello_world   
 make[1]: Leaving directory `/opt/hello-0.1'  
   dh_clean  
 root@debian-package:/opt/hello-0.1# fakeroot debian/rules build  
 dh build   
   dh_testdir  
   dh_auto_configure  
   dh_auto_build  
 make[1]: Entering directory `/opt/hello-0.1'  
 cc -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security  -c -o hello_world.o hello_world.c  
 cc -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wl,-z,relro -o hello_world hello_world.o  
 make[1]: Leaving directory `/opt/hello-0.1'  
   dh_auto_test  
 root@debian-package:/opt/hello-0.1# fakeroot debian/rules binary  
 dh binary   
   dh_testroot  
   dh_prep  
   dh_installdirs  
   dh_auto_install  
 make[1]: Entering directory `/opt/hello-0.1'  
 mkdir -p /opt/hello-0.1/debian/hello/usr//bin  
 cp hello_world /opt/hello-0.1/debian/hello/usr//bin  
 chmod 755 /opt/hello-0.1/debian/hello/usr//bin/hello_world  
 make[1]: Leaving directory `/opt/hello-0.1'  
   dh_install  
   dh_installdocs  
   dh_installchangelogs  
   dh_installexamples  
   dh_installman  
   dh_installcatalogs  
   dh_installcron  
   dh_installdebconf  
   dh_installemacsen  
   dh_installifupdown  
   dh_installinfo  
   dh_pysupport  
 dh_pysupport: This program is deprecated, you should use dh_python2 instead. Migration guide: http://deb.li/dhs2p  
   dh_installinit  
   dh_installmenu  
   dh_installmime  
   dh_installmodules  
   dh_installlogcheck  
   dh_installlogrotate  
   dh_installpam  
   dh_installppp  
   dh_installudev  
   dh_installwm  
   dh_installxfonts  
   dh_installgsettings  
   dh_bugfiles  
   dh_ucf  
   dh_lintian  
   dh_gconf  
   dh_icons  
   dh_perl  
   dh_usrlocal  
   dh_link  
   dh_compress  
   dh_fixperms  
   dh_strip  
   dh_makeshlibs  
   dh_shlibdeps  
   dh_installdeb  
   dh_gencontrol  
 dpkg-gencontrol: warning: File::FcntlLock not available; using flock which is not NFS-safe  
   dh_md5sums  
   dh_builddeb  
 dpkg-deb: building package `hello' in `../hello_0.1-1_amd64.deb'.  

At this point we have already our .deb file created!. We can see the shared libraries needed to build our program (in this case libc6, because of the stdio.h include) listed in our debian/hello.substvars file:

 root@debian-package:/opt/hello-1.0# cat debian/hello.substvars   
 shlibs:Depends=libc6 (>= 2.2.5)  
 misc:Depends=  

10.- Inspecting package contents

 root@debian-package:/opt/hello-0.1# find debian/hello  
 debian/hello  
 debian/hello/DEBIAN  
 debian/hello/DEBIAN/control  
 debian/hello/DEBIAN/md5sums  
 debian/hello/usr  
 debian/hello/usr/bin  
 debian/hello/usr/bin/hello_world  
 debian/hello/usr/share  
 debian/hello/usr/share/doc  
 debian/hello/usr/share/doc/hello  
 debian/hello/usr/share/doc/hello/copyright  
 debian/hello/usr/share/doc/hello/changelog.Debian.gz  
 root@debian-package:/opt/hello-0.1# dpkg --contents ../hello_0.1-1_amd64.deb   
 drwxr-xr-x root/root     0 2014-07-02 02:07 ./  
 drwxr-xr-x root/root     0 2014-07-02 02:07 ./usr/  
 drwxr-xr-x root/root     0 2014-07-02 02:07 ./usr/bin/  
 -rwxr-xr-x root/root   6160 2014-07-02 02:07 ./usr/bin/hello_world  
 drwxr-xr-x root/root     0 2014-07-02 02:07 ./usr/share/  
 drwxr-xr-x root/root     0 2014-07-02 02:07 ./usr/share/doc/  
 drwxr-xr-x root/root     0 2014-07-02 02:07 ./usr/share/doc/hello/  
 -rw-r--r-- root/root    940 2014-07-02 01:12 ./usr/share/doc/hello/copyright  
 -rw-r--r-- root/root    174 2014-06-30 23:51 ./usr/share/doc/hello/changelog.Debian.gz  

11.- Package maintenance scripts

It is possible to supply scripts that will run when the package is installed, upgraded or removed. These scripts are the control information files: preinst, postinst, prerm, postrm. And in some cases, may prompt the user if necessary, typically through a program such as debconf. More information on how to create this scripts can be found at the Debian Policy Manual.

12.- Debuild

As mentioned before, we can use debuild to build the Debian binary and source packages, check it with lintian, and sign it with debsign. We can use "debuild -us -uc" to build the packages without signing the .changes file. More information can be found with "man debuild".

 root@debian-package:/opt/hello-0.1# debuild -us -uc  
  dpkg-buildpackage -rfakeroot -D -us -uc  
 dpkg-buildpackage: warning: using a gain-root-command while being root  
 dpkg-buildpackage: source package hello  
 dpkg-buildpackage: source version 0.1-1  
 dpkg-buildpackage: source changed by Your Name <your_email_address@domain.com>  
  dpkg-source --before-build hello-0.1  
 dpkg-buildpackage: host architecture amd64  
  fakeroot debian/rules clean  
 dh clean   
   dh_testdir  
   dh_auto_clean  
 make[1]: Entering directory `/opt/hello-0.1'  
 rm -f *.o hello_world   
 make[1]: Leaving directory `/opt/hello-0.1'  
   dh_clean  
  dpkg-source -b hello-0.1  
 dpkg-source: info: using source format `3.0 (quilt)'  
 dpkg-source: info: building hello using existing ./hello_0.1.orig.tar.gz  
 dpkg-source: info: building hello in hello_0.1-1.debian.tar.gz  
 dpkg-source: info: building hello in hello_0.1-1.dsc  
  debian/rules build  
 dh build   
   dh_testdir  
   dh_auto_configure  
   dh_auto_build  
 make[1]: Entering directory `/opt/hello-0.1'  
 cc -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security  -c -o hello_world.o hello_world.c  
 cc -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wl,-z,relro -o hello_world hello_world.o  
 make[1]: Leaving directory `/opt/hello-0.1'  
   dh_auto_test  
  fakeroot debian/rules binary  
 dh binary   
   dh_testroot  
   dh_prep  
   dh_installdirs  
   dh_auto_install  
 make[1]: Entering directory `/opt/hello-0.1'  
 mkdir -p /opt/hello-0.1/debian/hello/usr//bin  
 cp hello_world /opt/hello-0.1/debian/hello/usr//bin  
 chmod 755 /opt/hello-0.1/debian/hello/usr//bin/hello_world  
 make[1]: Leaving directory `/opt/hello-0.1'  
   dh_install  
   dh_installdocs  
   dh_installchangelogs  
   dh_installexamples  
   dh_installman  
   dh_installcatalogs  
   dh_installcron  
   dh_installdebconf  
   dh_installemacsen  
   dh_installifupdown  
   dh_installinfo  
   dh_pysupport  
 dh_pysupport: This program is deprecated, you should use dh_python2 instead. Migration guide: http://deb.li/dhs2p  
   dh_installinit  
   dh_installmenu  
   dh_installmime  
   dh_installmodules  
   dh_installlogcheck  
   dh_installlogrotate  
   dh_installpam  
   dh_installppp  
   dh_installudev  
   dh_installwm  
   dh_installxfonts  
   dh_installgsettings  
   dh_bugfiles  
   dh_ucf  
   dh_lintian  
   dh_gconf  
   dh_icons  
   dh_perl  
   dh_usrlocal  
   dh_link  
   dh_compress  
   dh_fixperms  
   dh_strip  
   dh_makeshlibs  
   dh_shlibdeps  
   dh_installdeb  
   dh_gencontrol  
 dpkg-gencontrol: warning: File::FcntlLock not available; using flock which is not NFS-safe  
   dh_md5sums  
   dh_builddeb  
 dpkg-deb: building package `hello' in `../hello_0.1-1_amd64.deb'.  
  dpkg-genchanges >../hello_0.1-1_amd64.changes  
 dpkg-genchanges: warning: missing Priority for source files  
 dpkg-genchanges: including full source code in upload  
  dpkg-source --after-build hello-0.1  
 dpkg-buildpackage: full upload (original source is included)  
 Now running lintian...  
 warning: the authors of lintian do not recommend running it with root privileges!  
 W: hello: binary-without-manpage usr/bin/hello_world  
 Finished running lintian.  

Both source and binary packages have been built now. One of Lintian checks warned us about our package not including a manual page for hello_world binary. Although it is just a reminder, more information about this warning can be displayed using "lintian-info -t lintian_tag" command. In our case lintian_tag is "binary-without-manpage".

 root@debian-package:/opt/hello-0.1# ls -l ../  
 total 32  
 drwxr-xr-x 3 root root 4096 Jul 2 03:07 hello-0.1  
 -rw-r--r-- 1 root root 3020 Jul 2 03:09 hello_0.1-1_amd64.build  
 -rw-r--r-- 1 root root 1407 Jul 2 03:08 hello_0.1-1_amd64.changes  
 -rw-r--r-- 1 root root 3338 Jul 2 03:08 hello_0.1-1_amd64.deb  
 -rw-r--r-- 1 root root 1378 Jul 2 03:07 hello_0.1-1.debian.tar.gz  
 -rw-r--r-- 1 root root 735 Jul 2 03:07 hello_0.1-1.dsc  
 -rw-r--r-- 1 root root 441 Jun 30 23:43 hello_0.1.orig.tar.gz  
 -rw-r--r-- 1 root root 441 Jun 30 23:43 hello-0.1.tar.gz  

13.- Extracting sources

Finally we can use dpkg-source command to extract the sources from our package:

 root@debian-package:/opt# dpkg-source -x hello_0.1-1.dsc
 dpkg-source: warning: extracting unsigned source package (hello_0.1-1.dsc)  
 dpkg-source: info: extracting hello in hello-0.1  
 dpkg-source: info: unpacking hello_0.1.orig.tar.gz  
 dpkg-source: info: unpacking hello_0.1-1.debian.tar.gz  

References

https://www.debian.org/doc/debian-policy/
http://www.debian-administration.org/articles/336
https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial.en.pdf

https://www.debian.org/doc/manuals/developers-reference/

https://www.debian.org/doc/debian-policy/ch-controlfields.html