Monday, May 25, 2009

Tool of the Day: Lambda Probe

keywords: tomcat5, memory usage, thread usage

This great little java app runs on Apache Tomcat 5 and allows memory/thread/process monitoring for all installed applications/contexts. Very small meaning low system impact. It's been really helpful tracing down Java code errors in one of my deployed systems.

Download link: http://www.lambdaprobe.org/d/download.htm

Friday, May 22, 2009

Setting up a multi-gateway FreeBSD firewall/Router

Keywords: Multi-homed, source based routing, FreeBSD, ADSL gateways.

The reason for this article is that due to strange and sometimes entertaining DSL solutions in South Africa, we must use a bit of technical ability to provide proper services for small to medium enterprises, where the prohibitive cost of true leased lines would sink such a small company.

Basic Premise: 2 x 1Mbps Uncapped Fibre ADSL Lines from ISP1, plus a bonded ADSL solution (bonding 3 x 4mbps ADSL Lines into a 12Mbps line). Problem is that on a standard Linux router, you get ONE gateway, meaning that all traffic coming in on interface1 does not return to sender through interface1, but rather interface2 if it is the default gateway.

The solution here will allow Incoming traffic to be routed correctly back through the orginating link, making multi-homing a reality.

Refer to the Network Diagram:

em0 - 1mbps Uncapped through ISP1
em1 - 1mbps Uncapped through ISP1
em2 - 12Mbps Capped through ISP2
fxp0 - interface to local LAN



First, we'll need to add some kernel compilation options (I'll do a followup article on creating custom FreeBSD Kernels).

In your kernel config, add:

# Firewalling and NAT
options IPFIREWALL (IPFW firewalling, better than IPTables IMHO)
options IPDIVERT (The Divert class in IPFW)
options IPFIREWALL_FORWARD (The Forward class in IPFW)
options IPFIREWALL_DEFAULT_TO_ACCEPT (create a default allow all rule)
options IPFIREWALL_VERBOSE (Verbose logging to stdout)

you'll need to compile your kernel, install it. Add the following line to /etc/sysctl.conf:

net.inet.ip.sourceroute=1

This will activate source based routing. Now reboot the box. IP information for the system is as follows:

em0: 196.1.0.2/29 em0 gateway: 196.1.0.1 (default gateway for server)
em1: 196.2.0.2/29 em1 gateway: 196.2.0.1
em2: 196.3.0.2/29 em2 gateway: 196.3.0.1
fxp0: 10.0.0.1/24 fxp0 gateway: none (self)

NAT Deamons are setup as follows:

em0: port 8668
em1: port 8669
em2: port 8670

You need to activate the source based forward rules in your firewall, so add the following lines to IPFW:

ipfw add 00059 divert 8668 ip from any to any recv em0
ipfw add 00060 fwd 196.1.0.1 ip from 196.1.0.2 to not 196.1.0.2 out recv em0

Rule 59 does not END traffic, but rule 60 does, so further processing of packets is not needed.

ipfw add 00061 divert 8669 ip from any to any recv em1
ipfw add 00062 fwd 196.2.0.1 ip from 196.2.0.2 to not 196.2.0.2 out recv em1

ipfw add 00063 divert 8670 ip from any to any recv em2
ipfw add 00064 fwd 196.3.0.1 ip from 196.3.0.2 to not 196.3.0.2 out recv em2

This will activate source based routing, so if you have a NAT rule on em1 saying redirect all port 25 traffic to internal IP 10.0.0.2, then it will return traffic back to the source over em1, instead of em0 (the default gateway).

Thus ends the horrible congestion of one line, as well as allowing incoming static NAT for multiple interfaces, really expanding possibilities for your business.

Wednesday, May 20, 2009

Tool of the Day: MTR

MTR is a great little tool for running sequential traceroutes and presenting a readable format for analysis. MTR stands for My TraceRoute and I just used it to prove the "expert" engineers at one of the top 3 ISP's in South Africa where problems on our ADSL system were.

A sample output from running MTR produces:

[root@nas ~]# mtr -c 25 -r 66.212.227.1
HOST: nas Loss% Snt Last Avg Best Wrst StDev
1. nas 0.0% 25 0.7 0.8 0.7 2.7 0.4
2. 196.213.232.169 0.0% 25 15.8 15.8 14.5 24.7 2.0
3. cdsl1-rba-vl144.isdsl.net 0.0% 25 15.1 15.7 14.4 18.2 1.1
4. core1b-rba-gi1-0-5.ip.isnet. 0.0% 25 15.8 25.5 14.5 196.5 37.5
5. mi-za-rba-p6-gi0-1-6.ip.isne 0.0% 25 188.3 189.6 188.3 196.7 2.0
6. mi-uk-dock-p2-po2-3.ip.isnet 0.0% 25 188.4 191.4 187.9 248.8 12.0
7. core1a-dock-gi1-0-19-24.ip.i 0.0% 25 188.3 195.2 188.1 272.8 22.5
8. cp1-dock-gi0-1.ip.isnet.net 4.0% 25 363.7 266.3 188.7 379.1 64.0
9. 0.ge-2-1-0.rt1.lthn.mohawk.c 12.0% 25 435.6 442.4 378.1 517.7 39.9
10. 0.so-0-0-2.cr2.kdca.mohawk.c 12.0% 25 525.0 525.0 468.1 587.4 34.2
11. t13-2.dr2.kdca.mohawk.ca 20.0% 25 512.1 505.8 486.0 524.4 10.3
12. pix 4.0% 25 556.2 528.2 466.6 597.5 30.9

Basically it pings the hops sequentially so you can spot at which point packet loss occurs, or high latency kicks in. Our ISP engineers were squawking about upgrading router IOS, etc.. but this conclusively proved that the problem as at the far end in the USA landing sites.

If you aren't lucky enough to use Linux or FreeBSD (Its in the ports system) there is a Windows 2000/XP/2003 implementation here: http://winmtr.sourceforge.net/

Anyways this saved me a lot of time and effort in fighting with my ISP, hope it helps you...

Thursday, May 14, 2009

HTML2Image web frontend

For those interested in the technical details, Guangming Software distributes a mini web server with their program that only talks to their program. The program works from the command line or as a CGI from their web server (and no other, AFAIK). The first thing you need to do is start the background X imager like this (use the 64 Bit Ubuntu version, not the one they supply):

Download Location : http://www.guangmingsoft.net/htmlsnapshot/html2image.i386.RHEL3.tar.gz

Install the package into /usr/local/html2image, you'll also need to install some software thus:

apt-get install ia32-libs
apt-get install ia32-libs-gtk
apt-get install xserver-xorg
apt-get install xhost

Make sure the following is in /etc/environment:

LD_LIBRARY_PATH="/usr/lib:/usr/local/lib:/usr/local/lib/php/extensions:/usr/local/html2image"

Run XHOST to allow Screen access:

DISPLAY=:1
xhost +

Then you can start with running the Xvfb server:

LOG=/var/log/xvfb.log
cd /usr/local/html2image
Xvfb :1 -screen 0 1024x768x16 -nolisten tcp -auth X1.cfg >$LOG 2>&1 &

Then, you need to start their mini web server, running as the same user as the regular web server:

su www-data
/usr/local/html2image/nweb 8181 /usr/local/html2image

Then you can use a url to get an image:

http://localhost:8181/para=www.google.com?/tmp/image.jpg

The only problem is what you get back is the text output of the html2image program itself, not the image. The image goes to a file on the local server's hard drive (/tmp/image.jpg in this case) which isn't very helpful unless it runs on the same server you're working from. Create the html2image.php file in the www root:

$HOST = "http://localhost:8181";
$OPT = "/para=-H&768&-W&1024&-Q&100&-t&8000&";
$URL = $_GET["url"];
$OUTFILE = "/tmp/image".getmypid().".jpg";
$TMPFILE = "/tmp/image".getmypid().".txt";
#
# This should work, but you get some sort of
# a stream creation error that screws the image up.
#
#$lines = file($HOST.$OPT.$URL."&".$OUTFILE);
#
# This works, but isn't pretty!
#
exec("wget -O ".$TMPFILE." '".$HOST.$OPT.$URL."&".$OUTFILE."'");
header("Content-Type: image/jpeg");
$img = @imagecreatefromjpeg($OUTFILE);
imagejpeg($img);
unlink($OUTFILE);
unlink($TMPFILE);
?>

Execute the Image grab with the URL:

http://SERVERNAME_OR_IP/html2image.php?url=www.google.com

Jails (VM's) in FreeBSD 7

==Building your jail for the first time==

===Creating an appropriate make.conf===
You'll need to run make world (or make installworld) to create your jail.
If you don't want to install the whole kitchen sink you can use the make.conf
below. You can put it in your jail for future use and it'll be used by future
port builds inside your jail. One thing I've noticed is that make installworld
doesn't seem to respect and MAKE_CONF or __MAKE_CONF variables passed to it so
we'll just put it in /etc/make.conf for now.

Lets first back our current make.conf up:
cp /etc/make.conf /etc/make.conf.bak

And new one in there. Keep in mind, depending on what you want to use
this jail for you may want to modify this make.conf. For me this has worked
on building a variety of services from ports (inside the jail).
I like to name the below file make.conf.jail and copy it to make.conf,
then copy make.conf.bak back to make.conf when I'm done building the jail.

NO_ACPI= true # do not build acpiconf(8) and related programs
NO_BOOT= true # do not build boot blocks and loader
NO_BLUETOOTH= true # do not build Bluetooth related stuff
NO_FORTRAN= true # do not build g77 and related libraries
NO_GDB= true # do not build GDB
NO_GPIB= true # do not build GPIB support
NO_I4B= true # do not build isdn4bsd package
NO_IPFILTER= true # do not build IP Filter package
NO_PF= true # do not build PF firewall package
NO_AUTHPF= true # do not build and install authpf (setuid/gid)
NO_KERBEROS= true # do not build and install Kerberos 5 (KTH Heimdal)
NO_LPR= true # do not build lpr and related programs
NO_MAILWRAPPER=true # do not build the mailwrapper(8) MTA selector
NO_MODULES= true # do not build modules with the kernel
NO_NETCAT= true # do not build netcat
NO_NIS= true # do not build NIS support and related programs
NO_SENDMAIL= true # do not build sendmail and related programs
NO_SHAREDOCS= true # do not build the 4.4BSD legacy docs
NO_USB= true # do not build usbd(8) and related programs
NO_VINUM= true # do not build Vinum utilities
NO_ATM= true # do not build ATM related programs and libraries
NO_CRYPT= true # do not build any crypto code
NO_GAMES= true # do not build games (games/ subdir)
NO_INFO= true # do not make or install info files
NO_MAN= true # do not build manual pages
NO_PROFILE= true # Avoid compiling profiled libraries

# BIND OPTIONS
NO_BIND= true # Do not build any part of BIND
NO_BIND_DNSSEC= true # Do not build dnssec-keygen, dnssec-signzone
NO_BIND_ETC= true # Do not install files to /etc/namedb
NO_BIND_LIBS_LWRES= true # Do not install the lwres library
NO_BIND_MTREE= true # Do not run mtree to create chroot directories
NO_BIND_NAMED= true # Do not build named, rndc, lwresd, etc.

===Building the Jail===
Now for actually building your jail...

I'm defining JAILDIR here because I'm going to use it in a shellscript style example throughout the rest of this howto.

# Let's first make some directories
JAILDIR=/home/jail
mkdir -p $JAILDIR/dev
mkdir -p $JAILDIR/etc
mkdir -p $JAILDIR/usr/tmp
chmod 777 $JAILDIR/usr/tmp

cd /usr/src/

# You can replace the below with make installworld if you've built your
# world previously
make buildworld
make installworld DESTDIR=$JAILDIR
cd /usr/src/etc
cp /etc/resolv.conf $JAILDIR

make distribution DESTDIR=$JAILDIR NO_OPENSSH=YES NO_OPENSSL=YES
cd $JAILDIR

# At this point we'll mount devfs, and then hide the unneeded devs
mount_devfs devfs $JAILDIR/dev
devfs -m $JAILDIR/dev rule -s 4 applyset

# Create a null kernel
ln -s dev/null kernel

# Quell warnings about fstab
touch $JAILDIR/etc/fstab

# Use our existing resolv.conf
cp /etc/resolv.conf $JAILDIR/etc/resolv.conf

# Copy our settings for ssl
mkdir -p $JAILDIR/etc/ssl
mkdir -p $JAILDIR/usr/local/openssl
cp /etc/ssl/openssl.cnf $JAILDIR/etc/ssl
cd $JAILDIR/usr/local/openssl/
ln -s ../../../etc/ssl/openssl.cnf openssl.cnf

Make a decent rc.conf:

hostname="jail.example.com" # Set this!
ifconfig_em0="inet 10.0.0.20 netmask 255.255.255.255"
defaultrouter="10.0.0.1" # Set to default gateway (or NO).
clear_tmp_enable="YES" # Clear /tmp at startup.
# Once you set your jail up you may want to consider adding a good securelevel:
# Same as sysctl -w kern.securelevel=3
kern_securelevel_enable="YES" # kernel security level (see init(8)),
kern_securelevel="3"


You'll also want to make an alias on your interface for the ip above so we'll
do something like:

ifconfig em0 10.0.0.20 netmask 255.255.255.255 alias

Now you'll want to have devfs inside your jail, so to get it working for
the first time do this:
mount_devfs devfs $JAILDIR/devfs

And finally, copy your original make.conf back.
cp /etc/make.conf.bak /etc/make.conf

==Starting the jail for the first time==
OPTIONAL (but probably necessary):
You'll want to mount /usr/ports and /usr/src so you can install ports inside
your jail, unless you have another way you want to do this (such as downloading
packages).
mount_nullfs /usr/ports $JAILDIR
mount_nullfs /usr/src $JAILDIR

Now we can start our jail
jail $JAILDIR jail.example.com 10.0.0.20 /bin/sh

Once inside the jail you'll want to start services:
/bin/sh /etc/rc

While you're here you'll want to edit your password file since if someone
breaks into your jail, and starts cracking it you won't want them to have
the same passwords as your root system has. Also remove all users you
don't need in the jail:
vipw
passwd root

From here, assuming all went well you can do something like:
cd /usr/ports/security/openssh
make install clean

And build your port(s) inside your jail. Once you're finished be sure to
unmount the directories so a compromised jail can't build more ports.

I've verifed that truss works correctly in a jail
so between ldd and truss you should be set.

Also note that if you try to start your jail with just:
jail $JAILDIR jail.example.com 10.0.0.20 /bin/sh /etc/rc
but you have no services/daemons/programs set to run, the jail will simply start and then exit since there's nothing running inside.

==Getting it to start automatically==
You'll now need to put your settings in /etc/rc.conf
First put the alias you jail has in there:

ifconfig_em0_alias0="inet 10.0.0.20 netmask 0xffffffff"

===Editing the rc.conf===
For those of you that are looking to make your own rc script, I don't recommend it.
I've found issues getting devfs rules to be applied with the a script, and really this way is much easier. It's also the standard way and you can attach to jails later on quite easily
without using screen (read below).

Here's the standard rc.conf way of getting your jail to run at startup:

jail_enable="YES" # Set to NO to disable starting of any jails
jail_list="pk" # Space separated list of names of jails
jail_set_hostname_allow="NO" # Allow root user in a jail to change its hostname
jail_socket_unixiproute_only="YES" # Route only TCP/IP within a jail

jail_pk_rootdir="/usr/home/prison/pk"
jail_pk_hostname="pk.example.com"
jail_pk_ip="10.0.0.20"
jail_pk_exec_start="/bin/sh /etc/rc"
jail_pk_devfs_enable="YES"
jail_pk_devfs_ruleset="devfsrules_jail"

==Jail maintenance==
Of course from time to time you may have to upgrade ports in your jail, or the world in the jail itself. This isn't a big deal either. Instead of using jail (which makes its own IP address and everything) we can use chroot instead which is similar since all we're using is a simple shell
and then we'll be done with it.

First mount the dirs so they're accessable in the chroot:
mount_nullfs /usr/ports $JAILDIR
mount_nullfs /usr/src $JAILDIR

Connect to your jail:
find the jail id of the jail you are running with jls:
#jls
JID IP Address Hostname Path
1 10.0.0.20 pk.example.com /usr/home/prison/pk

Now connect to it using the JID:
jexec 1 /bin/sh

To upgrade your world:
cd /usr/src
make buildworld
make installworld
NOTE: If you've just done make buildworld previously you can do make installworld and install all the newly compiled binaries again.

To build a port:
cd /usr/ports/sysutils/example
make install clean
NOTE: You may also want to install portupgrade to make port management easier.

When you're done just exit:
exit

Solaris 10 installing Postfix, Spamassassin and DNS Blocklists

== Install Postfix ==

Install postfix with our pkg-get program (http://www.opencsw.org/pkg-get):

pkg-get -U
pkg-get -i postfix

== Disable Sendmail and enable Postfix ==

First disable sendmail:

/usr/sbin/svcadm disable svc:/network/smtp:sendmail

Then clear maintenance and enable postfix:

/usr/sbin/svcadm clear svc:/network/smtp/postfix:default
/usr/sbin/svcadm enable svc:/network/smtp/postfix:default
/usr/sbin/svcadm restart svc:/network/smtp/postfix:default

Test that postfix is now the MTA:

root@Mitsubishi-S10-T1SMTP - />telnet 0 25
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 Mitsubishi-S10-T1SMTP.localdomain ESMTP POSTFIX
quit
221 2.0.0 Bye
Connection to 0 closed by foreign host.

Notice the POSTFIX on the 220 reply line - this means postfix is installed and working

== Enabling Postfix to deliver and receive mail ==

exit the /etc/opt/csw/postfix/main.cf file and make sure the following lines are unhashed and configured (current values included are for demonstration purposes only) :

inet_interfaces = $myhostname '''(specify which interfaces to listen on)'''
mynetworks_style = host '''(specify which type of access is allowed)'''
mynetworks = 10.0.0.0/24, 127.0.0.0/8 '''(specify which networks are allowed to relay)'''
relay_domains = $mydestination '''(domains that this mail server accepts mail for)'''
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain '''(list of acceptable domain suffix on addresses)'''
myhostname = Mitsubishi-S10-T1SMTP '''(defined hostname, make sure it matches the system hostname)'''
mydomain = kaizania.co.za '''(specify the domain that will be this mail servers mail suffix)'''

== Test the server ==

You can test the server to see that it relays mail:

root@Mitsubishi-S10-T1SMTP - /etc/opt/csw/postfix>telnet Mitsubishi-S10-T1SMTP 25
Trying 10.0.0.107...
Connected to Mitsubishi-S10-T1SMTP.
Escape character is '^]'.
220 Mitsubishi-S10-T1SMTP ESMTP Postfix
helo test
250 Mitsubishi-S10-T1SMTP
mail from: lionel.bisschoff@kaizania.co.za
250 2.1.0 Ok
rcpt to: someone@somewhere.com
250 2.1.5 Ok
data
354 End data with .
Subject : Test
Test
.
250 2.0.0 Ok: queued as 2404EDDD1B

And we see the message in the delivery queue:

root@Mitsubishi-S10-T1SMTP - /opt/csw/bin>./mailq
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
2404EDDD1B* 378 Tue May 27 10:16:23
lionel.bisschoff@kaizania.co.za
someone@somewhere.com

There we go - working now !!!

== Adding Spamassassin anti-spam ==

First, installing spamassassin:

pkg-get install spamassassin

Then the easiest way to integrate postfix and spamassassin is to use spamd in an "after-queue" inspection. This configuration does not allow rejecting messages within the SMTP transaction, so it unfortunately contributes to backscatter email. On the other hand, it has important performance advantages over "before-queue" inspection.

First, edit /etc/postfix/master.cf, find the

# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (50)
# ==========================================================================
...
smtp inet n - n - - smtpd
...

line and just add " -o content_filter=spamassassin" to the end of the line:

# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (50)
# ==========================================================================
...
smtp inet n - n - - smtpd -o content_filter=spamassassin
...

## you can change the maxproc to 400 if you have a strong server (dual xeon 2.8 nd above) and the performance will be much better.

Then, go to the end of the file, and add this:


# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (50)
# ==========================================================================
...
spamassassin
unix - n n - - pipe
user=nobody argv=/path/to/spamc -e /path/to/postfix/sendmail -oi -f ${sender} ${recipient}
# make sure it's all on one line or
# start the consecutive lines with a whitespace
# like I did here

Make sure that you have adjusted the path to the spamc and sendmail commands above! (Please note that the path required is Postfix's sendmail and not the standalone package, Sendmail the SMTP server. It will not work if you're not careful about which one is installed). Then, setup spamd to start with the system, and you are ready to go. If you wish to provide spamassassin preferences, change "user=nobody" to a valid system user (except for root, since Postfix will NOT call external programs as root), and add .spamassassin into that user's home directory.

If you use user preferences stored in SQL, you should change "spamassassin" service in master.cf to following:

# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (50)
# ==========================================================================
...
spamassassin
unix - n n - - pipe
flags=Rq user=nobody argv=/path/to/spamc -u ${recipient} -e /path/to/postfix/sendmail -oi -f ${sender} ${recipient}

Notice "-u ${recipient}" added. Otherwise "username" field in database will always appear as user which postfix is invoking spamc(in this example it is 'nobody').

Only mail received by SMTP will be scanned with this method, i.e. mail injected with sendmail(1) will not be fed to SpamAssassin.

== Adding a Blacklist to Postfix ==

to add a Blacklist to block certain senders, do the following:

Create a blacklist file in the postfix /etc directory (i.e. /etc/opt/csw/postfix/blacklist) and add the addresses you wish to block within:

support@connect2casino.com REJECT

With this file created, make it a DB file with the following command:

/opt/csw/sbin/postmap hash:/etc/opt/csw/postfix/blacklist

Then at the end of the postfix /etc/opt/csw/postfix/main.cf file, add the sender check line:

smtpd_sender_restrictions = reject_unknown_sender_domain, check_sender_access hash:/etc/opt/csw/postfix/blacklist

The added "reject_unknown_sender_domain" will also reject domains that do not exist (a DNS Lookup is performed). Restart the Postfix service to effect the changes:

svcadm disable /network/smtp/postfix; svcadm enable /network/smtp/postfix

And test that the sender address is now rejected:

Iridium-S10-SMTP(root)/var/log# telnet 10.1.2.24 25
Trying 10.1.2.24...
Connected to 10.1.2.24.
Escape character is '^]'.
220 iSMTP.actionpokernetwork.com ESMTP Postfix
helo dorks
250 iSMTP.network.com
mail from: support@connect.com
250 2.1.0 Ok
rcpt to: operations@network.com
554 5.7.1 : Sender address rejected: Access denied

Network link aggregation in Solaris

Aggregation is better than failover in the fact that it provides two or more datapaths, effectively doubling/tripling/etc... bandwidth between the server and the switch. Failover is still enabled, as if any network cable is removed the system will continue to function correctly.

To setup Aggregation:

Make sure the ports to be added to the aggregated port are unplumbed, create the aggregation thus:

 dladm create-aggr -d e1000g0 -d e1000g1 1

This will create an aggregated port "aggr1" that comprises e1000g0 and e1000g1, to show the the contents of the aggregated port:

 dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:14:4f:1:9c:98 (auto)
device address speed duplex link state
e1000g0 0:14:4f:1:9c:98 100 Mbps full up attached
e1000g1 0:14:4f:1:9c:99 100 Mbps full up attached

Bring up and configure the new port:

 ifconfig aggr1 plumb 10.0.0.10 netmask 255.255.255.0 up

and display it:

 ifconfig aggr1
aggr1: flags=1000843 mtu 1500 index 6
inet 10.0.0.10 netmask ffffff00 broadcast 10.0.0.255
ether 0:14:4f:1:9c:98

Edit the /etc/hosts and hostname.aggr1 file to make changes persistent.


Increasing the width of a Blogspot blog

Pretty east one this,

Under layout, click on Edit HTML and change the "width" elements in the following code:

/* Outer-Wrapper
----------------------------------------------- */
#outer-wrapper {
width: 1025px;
margin:0 auto;
padding:10px;
text-align:left;
font: $bodyfont;
}

#main-wrapper {
width: 710px;
margin-left: 20px;
float: left;
word-wrap: break-word; /* fix for long text breaking sidebar float in IE */
}


I set my outer wrapper width to 1025 pixels and then the main wrapper width to 710 pixels.

Wednesday, May 13, 2009

Increasing the speed of rTorrent downloads

I use rtorrent to get all my *cough* legal software downloads, and I found a way to optimize the source code of the libtorrent libraries to increase speeds in rtorrent.

I run this in FreeBSD, but its cross portable to most *nix'es...

BSD uses the ports system, and luckily all source files get dumped into /usr/ports/distfiles - I unzipped the libtorrent-0.12.2.tar.gz file into /usr/ports/distfiles/libtorrent-0.12.2.

I modified the following files:

in ./configure

set PEER_NAME to UT161

PEER_NAME "UT161"

this tells all trackers that your client is uTorrent 1.6.1 - which is widely accepted as a "good" version for private trackers and general public domain.

next, edit the ./src/protocol/peer_connection_leech.cc file:

old line #:114 if (cachedTime - m_timeLastRead > rak::timer::from_seconds(240))
new line #:114 if (cachedTime - m_timeLastRead > rak::timer::from_seconds(40))

This will drop an inactive client after 40 seconds instead of 4 minutes, allows faster uptake of seeds and peers.

next, edit ./src/download/download_wrapper.cc:

old line #:272 if (ticks % 4 == 0) {
new line #:272 if (ticks % 1 == 0) {

This will clear ncurses connected clients after 1 tick instead of 4 (1 tick equals 30 seconds)

next, edit ./src/torrent/tracker.cc:

old line #:49 m_normalInterval(1800),
new line #:49 m_normalInterval(600),

This will reconnect to the tracker every 10 minutes to check for new seeds/leechers instead of every 30 minutes.

next, edit ./src/protocol/peer_connection_leech.cc:

old line #:153 if (!download_queue()->canceled_empty() && m_downStall >= 6)
new line #:153 if (!download_queue()->canceled_empty() && m_downStall >= 2)

This will request new file pieces from peers every 2 ticks instead of 6

on BSD, I had to make sure openssl was installed (/usr/ports/security/openssl)

then the installation of libtorrent:
/usr/ports/distfiles/libtorrent-0.12.2>./configure --prefix=/usr/local; make && make install

once thats done, just install rtorrent itself: /usr/ports/net-p2p/rtorrent> make all install

for added performance, I setup my .rtorrent.rc file thus:

bind = xxx.xxx.xxx.xxx (Your local IP address that rtorrent binds to)
ip = xxx.xxx.xxx.xxx (the *Misreported* IP given to trackers he he!)
check_hash = yes (Hash check completed files for consistency)
directory = /root/.incoming (download directory)
session = /root/.session (Session directory, if rtorrent crashes, it will recover from torrents in here)
peer_exchange = yes (exchange tracker and peer info with connected systems)
upload_rate = 1 (I limit uploads to 1kb/sec)
download_rate = 60 (I limit downloads to 60kb/sec)
max_peers = 500 (maximum amount of connected peers per torrent)
dht = on (DHT for trackerless clients, helps a bit)

hash_read_ahead = 8 (FreeBSD Specific)
hash_max_tries = 5 (FreeBSD Specific)
hash_interval = 10 (FreeBSD Specific)

tos=lowdelay (Set TCP Type-Of-Service to lowdelay, best possible peers)
use_udp_trackers = yes (Useless if behind a firewall)

max_memory_usage = 2147483648 (2Gb maximum physical memory used)

A torrent that I was struggling with at about 1.1 Kb/sec shot up to 15Kb/sec after these changes - I can definately see much better speeds. These changes will also help on shaped bandwidth, where torrents are throttled by ISPS.

My First Blog !!!

I though I'd create my own technical blog, with the amount of problems and solutions I've come across (through highly convaluted and misleading articles out there), I've found solutions not publicly visible - I guess this is my contribution back to the net for all the times it's helped me...

Going to TRY and post one article per working day...