Wednesday, August 12, 2009

SPF and DKIM

Wow, has it been THAT long since my last post ? Been moving house and now I'm a certified Telecommuter, 100% of my Job done from my lovely home office at the beach.

Anyways, I had to do a large mailshot to about 1.4 million customers, and in the runup, to increase delivery rates, I instituted SPF and DKIM.

A quick explanation of each:

SPF - Sender Policy Framework
DKIM - Domain Keys Identified Mail

Both systems are a way of making sure that mail sent by a particular server is from the correct domain in question. Both are heavily used in the enterprise, and having them on your mailserver will increase hit rates.

SPF is easy, you just need to add a TXT DNS record to your domain, like so:

v=spf1 ip4:11.11.11.11 ip4:22.22.22.22 mx ptr mx:somedomain.com include:somedomain.com ~all

lets break it down:

v=spf1 Version (currently only v1 exists).
ip4:11.11.11.11 ip4:22.22.22.22 If source mail server is IP 11.11.11.11 or 22.22.22.22 then it is authorised.
mx ptr if source IP has reverse lookup to this domain then it is authorised.
mx:somedomain.com include:somedomain.com source domain authorised to send mails on behalf of this domain.

DKIM is a bit harder, first you need an MTA that supports it, I've used Merak 9 (which has it built in) and postfix (with the dkim-milter plugin).

The Logic is that you generate a keypair, the private part of which is integrated into your mailsystem, and the public part in a TXT DNS record on your domain.

The receiving MTA looks up the TXT record, finds the public key, and checks with the sending server through keypair handshake to confirm that it IS in fact the correct sender. If it is, the mail is accepted.

The source MTA adds headers like this:

X-DKIM: Sendmail DKIM Filter v2.8.2 sender1.yourdomain.com 75866730012
DKIM-Signature: v=1; a=rsa-sha1; c=simple/simple; d=yourdomain.com;
s=default; t=1239981026; bh=+NNkD6jOlYKtY2AIGNRToH2tkm0=;
h=Date:List-Help:List-Subscribe:List-Unsubscribe:List-Owner:
List-Post:From:Reply-To:To:Subject:MIME-Version:Content-Type:
Message-Id;
b=MrjXBShjNexWy62fC4Uu7xS3Hxav+cHtqIBzwMlcufadsffLtW9KmF5sO58+yHjyy
I3SiX0TNyEbvXtSHvRKm9z630zDiN0dxVXGqhgEfdklaj4jlkfhR6GrsRgzW2YOW6/9
sKFnz214AkhAPrFBD30hNmZfRfY75v5q94FnGDUo=


and the domain TXT record is setup thus:

v=DKIM1; g=*; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GWETBNiQKBgQC5KT1eN2lqCRQGDX+20I4liM2mktrtjWkV6mW9WX7q46cZAYgNrus53vgfl2z1Y/95mBv6Bx9WOS56OAVBQw62+ksXPT5cRUAUN9GkENPdOoPdpvrU1KdAMW5c3zmGOvEOa4jAlB4/wYTV5RkLq/1XLxXfTKNy58v+CKETLQS/eQIDAQAB


Where:

v=DKIM1 Version 1 of DKIM
k=rsa Keytype DSA or RSA
p=xxxxxx The actual public key

These two techniques, along with actually having proper forward and reverse lookups on your mailserver will increase delivery rates and decrease spam scores, making sure your mail ends up in the inbox, and not junkmail.

Wednesday, July 22, 2009

Tool of the Day: TCPMSSD

I've had issue with an ADSL multilink implementation over a FreeBSD firewall, that just wont seem to work with a particular PC's MTU. I've of course changed the pc MTU with TCPOptimizer, but now I've gone a level up and am actually mangling the packets with a tcp MTU/MSS Clamp.

The Clamp works by making sure that all packets are lower than or equal to a particular MTU, therefore making sure traffic flows correctly.

I installed TCPMSSD from FreeBSD ports /usr/ports/net/tcpmssd, and made sure the daemon starts from rc.local, with the following command line:

/usr/local/bin/tcpmssd -b -p 7777 -m 1300

-b = mangle both SYN and ACK packets
-p 7777 = run the Daemon on port 7777
-m 1300 = MTU size of 1300 Bytes

Then, you need to pass traffic to the daemon through IPFW:

add 00042 divert 7777 ip from any to 10.0.0.1 (mangle traffic to source)
add 00043 skipto 00047 ip from 10.0.0.1 to 10.0.0.0/20 (skip internal traffic)
add 00043 skipto 00047 ip from not 10.0.0.1 to any (skip anything else other than this host)
add 00044 divert 7777 ip from any to any (mangle traffic from source)
add 00045 divert 8670 ip from any to any (NATD traffic from source)
add 00046 fwd 192.168.0.1 ip from any to any (FWD and end traffic from source)

this is tested, working and in production on my live FreeBSD firewalls, a great little tool...

Thursday, July 16, 2009

Tool of the Day: TCP-Z Windows Vista Half-Open Connection Patch

To speed up Vista internet network speeds, you can now run a simplified tool, that modifies kernel values on the fly, without changing any settings permanently.

You can download it here: Softpedia

If you WOULD like to change the settings, follow microsofts article on how to HERE

Either way, look forward to increased network performance in Vista/Windows 7

Wednesday, July 8, 2009

Increasing performance of Static IP (L2TP) ADSL Lines

With most of the South African ISP's offering static IP ADSL, I Think there is some confusion or just plain not knowing how these offerings work. The machines connected to these Routers benefit from having correct MTU Size set, both to reduce packet fragmentation, and also allow packets to traverse if they have DF Bit set (DF = Do Not Fragment).

Standard Ethernet MTU is 1500 bytes, and ADSL PPPoE MTU is 1492 Bytes. What ISP's do is run an L2TP Tunnel from the ISP to the ADSL Router, presenting the user with static IP's, but reducing MTU size to 1472 Bytes (Standard L2TP Data segment).

To optimize your Linux/BSD machines, the easiest option is to just adjust the mtu thus:

ifconfig em0 mtu 1472

That will make sure all packets originating from that interface are the correct size. The issue arises if you use windows machines, as MTU Discovery does not always work correctly. This is tried and tested on a windows box I have here:

Ping Google with a 1492 Byte ICMP Packet:

C:\Users\kim.attree>ping -l 1492 www.google.com

Pinging www.l.google.com [74.125.45.103] with 1492 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 74.125.45.103:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

And Pinging Google with a 1472 Byte ICMP Packet:

C:\Users\kim.attree>ping -l 1472 www.google.com

Pinging www.l.google.com [74.125.45.103] with 1472 bytes of data:
Reply from 74.125.45.103: bytes=1472 time=354ms TTL=49
Reply from 74.125.45.103: bytes=1472 time=353ms TTL=49
Reply from 74.125.45.103: bytes=1472 time=354ms TTL=49
Reply from 74.125.45.103: bytes=1472 time=354ms TTL=49

Ping statistics for 74.125.45.103:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 353ms, Maximum = 354ms, Average = 353ms

The easiest way to fix this is by using a TCP Stack modifier, and I suggest using the freeware tool TCP Optimizer, which you can download here http://www.speedguide.net/files/TCPOptimizer.exe

Choose "Custom Options" and set your MTU to 1472 Bytes, and reap the rewards of improved performance and throughput.

Thursday, June 25, 2009

Tool of the Day: www.oldports.org

I had a problem installing Asterisk today, and had to rollback the zaptel port from 1.4.11 to 1.4.6 - the easiest way to do this was to get the port directory misc/zaptel from my old friend http://www.oldports.org

This site has years and years of port iterations, meaning it's painless and easy to get an older version of a port for yourself.

Wednesday, June 24, 2009

SSH host based authentication and security

I use SSH as my main form of connection, even with connecting to other boxes (I just create TCP Tunnels to get to RDP/VNC services). The problem with password based authentication is that social engineering and brute force CAN break through.

I prefer host based authentication - meaning that the server I am connecting to will only allow the server I'm connecting from to login, using DSA or RSA keys for the authentication process.

For clarity, Source means the server/workstation you are connecting FROM, Destination means the server you are connecting TO.

To setup the whole thing, you need to generate a private/public keypair on your Source server, do this with ssh-keygen:

ssh-keygen -t dsa

Follow the prompts and you DONT have to fill in a password if you dont want to, but it adds a level of security. The following files will be created in $HOME/.ssh :

id_dsa
id_dsa.pub

Cat the id_dsa.pub file, and copy the output, this is going to need to be placed on the Destination server. Login to your destination server as the user you normally connect as. Make sure there is a $HOME/.ssh directory and then create a file called "authorized_keys" (Case and spelling sensitive). Paste the previous output of the id_dsa.pub file into this file and save it.

Now you might need to edit your /etc/ssh/sshd_config file to allow host-based key authentication, make sure the following parameters are set:

PermitRootLogin yes (ONLY if you NEED root login, rather su to root)
RSAAuthentication yes (Allow RSA as well as DSA Keys)
PubkeyAuthentication yes (Allow Public key authentication)
AuthorizedKeysFile .ssh/authorized_keys (Location of the Server authorized keys)

Restart sshd, or HUP it (pkill -HUP sshd).

For added security, I disable PAM authentication in the /etc/ssh/sshd_config file:

ChallengeResponseAuthentication no

This prevents ANY type of password authentication, meaning bruteforce attacks are impossible to conduct against your SSH Server.

Should you wish to allow tunnelling through your SSH Server, set these parameters in your /etc/ssh/sshd_config:

AllowTcpForwarding yes

Again, restart SSHD.

Thats it - a pretty secure SSH system, which is usually the first point of attack for UNIX Hackers.

Thursday, June 11, 2009

Tool of the Day: Running Multiple versions of Internet Explorer

For testing websites/software on older versions of IE can be a pain in the posterier, but with this tool, you can run multiple versions from 3.0 to 6.0. This is for Windows XP ONLY !!! But its helped me with some dodgy client complaints in the past.

Link is here: http://tredosoft.com/Multiple_IE

Wednesday, June 10, 2009

Load Balanced SSL based Internet Casino Client System

My personally designed SSL-offloaded load-balanced casino system went live today across 49 Internet Casinos, it has increased performance immensely and is able to add multiple servers on the fly to alleviate stress on existing servers.

The SSL Offload does SSL encryption calculations on the Layer4 switch, using onboard floating-point processors, meaning NO SSL encode/decode on the servers themselves. The switches can handle 115,000 concurrent connections, so plenty space to grow.

I designed and implemented the entire system, barring the existing MSSQL implementation. Casino is now handling 200,000 spins per day.

Just cant say "which" casinos....

See my System Diagram here:

Squid Proxy server, Ads blocking

I added an ad-blocking component to squid, not a third-party product, just a few configs of my own. The way I've implemented it, you don't see any "AD BLOCKED" crap on your webpages, you just have a clear block where the ad was, with no error message, increasing the "look/feel" experience for your users.

First, in the ACL section, create the ACL for ads:

acl ads_block_list dstdomain -i "/usr/local/etc/squid/blocks/ads_block.list"

Above all your normal http_access/http_deny rules, place this one:

deny_info ERR_BLOCKED_ADS ads_block_list
http_access deny ads_block_list

Now, create the file /usr/local/etc/squid/blocks/ads_block.list and populate it (I've just shown a head from my file):

[root@nas /usr/local/etc/squid/blocks]# head ads_block.list
101com.com
101order.com
103bees.com
1100i.com
123banners.com
123found.com
123pagerank.com
180hits.de
180searchassistant.com
180solutions.com

You can get anti-ads lists such as mine from various locations, use google to search.

Now edit the custom error message for the ads_block_list ACL, which is: ERR_BLOCKED_ADS

[root@nas /usr/local/etc/squid/errors/English]# cat ERR_BLOCKED_ADS



That last line, you'll notice the ! character which means don't display the standard squid error message.

once all is complete, reload your configs with "squid - k reconfigure" and try access pages now. Ads are blocked, and all you see is the page background. This way you can save large amounts of bandwidth on your internet lines, without creating errors or graphical problems on user viewed pages.

Tool of the Day: pkg-get for Solaris 10

Package management in Solaris 10 is less than stellar, with no automated system such as apt-get, ports, portage or yum like linux distro's, but I found a FreeBSD-like portinstall package called pkg-get. It allows installation of packages from a web repository and is compatible with the Solaris pkgadd system.

To install pkg-get use the folowing command.

"pkgadd -d http://www.opencsw.org/pkg_get.pkg".

edit the mirror url using.

"vi /opt/csw/etc/pkg-get.conf" and changing the default site url to.

url=ftp://ftp.heanet.ie/pub/opencsw/stable (This is Ireland's repository - closest one for SA)

run "pkg-get -U" to update catalog.

pkg-get is now ready.

"pkg-get -a |grep package" to find the desired packages, then:

"pkg-get -i packagename" to install it.

Very simple and powerful, even allows upgrading of the complete CSW subsystem in Solaris.

Monday, June 8, 2009

Tool of the Day: DGTeam firmware for Netgear DG834 series routers

keywords: netgear, dg834, firmware, dgteam

Found a great tool which increased stability on my router (DG834PN), as well as including about 100 features not seen on the standard firmware - even SNR increase/reduction to affect speed/stability.

You can grab the firmware for most of the 834 series routers here: http://dgteam.ilbello.com

Update:

I did this on a DG834 straight router, connection speeds (synchronised ADSL) went from 2908Kbps to 3824Kbps - On the SAME LINE !! No connection drops - 34 hours on the same PPPoE session.

Highly Recommended

Hacked my XBOX 360

Microsoft say their new Lite-On DVD Rom is unhackable due to firmguard protection.

Well I hacked mine to the latest iXtreme firmware v1.6 - I'm now able to play the 65 XBOX backups I have. Quite a convaluted procedure involving drive door positioning, soldering the PCB to unlock the developer mode on the drive, then extracting the DVD keys and re-inserting them into the hacked firmware, erasing the drive (scary) and then flashing the hacked firmware back.

Been enjoying "Battlestations Pacific" over the weekend :)

Wednesday, June 3, 2009

Update

Been off having my Jaw broken to have some Wisdom teeth extracted, so I'll be back to posting articles in no time !

For personal note, i'll be firmware updating my XBOX360 tonight, as well as doing the technical drawings for my "PC in and XBOX 360" project...More on that later.

Seems I'm missed at work, 8 days off and systems crashing in flames always ensures job security when you're away :D

Monday, May 25, 2009

Tool of the Day: Lambda Probe

keywords: tomcat5, memory usage, thread usage

This great little java app runs on Apache Tomcat 5 and allows memory/thread/process monitoring for all installed applications/contexts. Very small meaning low system impact. It's been really helpful tracing down Java code errors in one of my deployed systems.

Download link: http://www.lambdaprobe.org/d/download.htm

Friday, May 22, 2009

Setting up a multi-gateway FreeBSD firewall/Router

Keywords: Multi-homed, source based routing, FreeBSD, ADSL gateways.

The reason for this article is that due to strange and sometimes entertaining DSL solutions in South Africa, we must use a bit of technical ability to provide proper services for small to medium enterprises, where the prohibitive cost of true leased lines would sink such a small company.

Basic Premise: 2 x 1Mbps Uncapped Fibre ADSL Lines from ISP1, plus a bonded ADSL solution (bonding 3 x 4mbps ADSL Lines into a 12Mbps line). Problem is that on a standard Linux router, you get ONE gateway, meaning that all traffic coming in on interface1 does not return to sender through interface1, but rather interface2 if it is the default gateway.

The solution here will allow Incoming traffic to be routed correctly back through the orginating link, making multi-homing a reality.

Refer to the Network Diagram:

em0 - 1mbps Uncapped through ISP1
em1 - 1mbps Uncapped through ISP1
em2 - 12Mbps Capped through ISP2
fxp0 - interface to local LAN



First, we'll need to add some kernel compilation options (I'll do a followup article on creating custom FreeBSD Kernels).

In your kernel config, add:

# Firewalling and NAT
options IPFIREWALL (IPFW firewalling, better than IPTables IMHO)
options IPDIVERT (The Divert class in IPFW)
options IPFIREWALL_FORWARD (The Forward class in IPFW)
options IPFIREWALL_DEFAULT_TO_ACCEPT (create a default allow all rule)
options IPFIREWALL_VERBOSE (Verbose logging to stdout)

you'll need to compile your kernel, install it. Add the following line to /etc/sysctl.conf:

net.inet.ip.sourceroute=1

This will activate source based routing. Now reboot the box. IP information for the system is as follows:

em0: 196.1.0.2/29 em0 gateway: 196.1.0.1 (default gateway for server)
em1: 196.2.0.2/29 em1 gateway: 196.2.0.1
em2: 196.3.0.2/29 em2 gateway: 196.3.0.1
fxp0: 10.0.0.1/24 fxp0 gateway: none (self)

NAT Deamons are setup as follows:

em0: port 8668
em1: port 8669
em2: port 8670

You need to activate the source based forward rules in your firewall, so add the following lines to IPFW:

ipfw add 00059 divert 8668 ip from any to any recv em0
ipfw add 00060 fwd 196.1.0.1 ip from 196.1.0.2 to not 196.1.0.2 out recv em0

Rule 59 does not END traffic, but rule 60 does, so further processing of packets is not needed.

ipfw add 00061 divert 8669 ip from any to any recv em1
ipfw add 00062 fwd 196.2.0.1 ip from 196.2.0.2 to not 196.2.0.2 out recv em1

ipfw add 00063 divert 8670 ip from any to any recv em2
ipfw add 00064 fwd 196.3.0.1 ip from 196.3.0.2 to not 196.3.0.2 out recv em2

This will activate source based routing, so if you have a NAT rule on em1 saying redirect all port 25 traffic to internal IP 10.0.0.2, then it will return traffic back to the source over em1, instead of em0 (the default gateway).

Thus ends the horrible congestion of one line, as well as allowing incoming static NAT for multiple interfaces, really expanding possibilities for your business.

Wednesday, May 20, 2009

Tool of the Day: MTR

MTR is a great little tool for running sequential traceroutes and presenting a readable format for analysis. MTR stands for My TraceRoute and I just used it to prove the "expert" engineers at one of the top 3 ISP's in South Africa where problems on our ADSL system were.

A sample output from running MTR produces:

[root@nas ~]# mtr -c 25 -r 66.212.227.1
HOST: nas Loss% Snt Last Avg Best Wrst StDev
1. nas 0.0% 25 0.7 0.8 0.7 2.7 0.4
2. 196.213.232.169 0.0% 25 15.8 15.8 14.5 24.7 2.0
3. cdsl1-rba-vl144.isdsl.net 0.0% 25 15.1 15.7 14.4 18.2 1.1
4. core1b-rba-gi1-0-5.ip.isnet. 0.0% 25 15.8 25.5 14.5 196.5 37.5
5. mi-za-rba-p6-gi0-1-6.ip.isne 0.0% 25 188.3 189.6 188.3 196.7 2.0
6. mi-uk-dock-p2-po2-3.ip.isnet 0.0% 25 188.4 191.4 187.9 248.8 12.0
7. core1a-dock-gi1-0-19-24.ip.i 0.0% 25 188.3 195.2 188.1 272.8 22.5
8. cp1-dock-gi0-1.ip.isnet.net 4.0% 25 363.7 266.3 188.7 379.1 64.0
9. 0.ge-2-1-0.rt1.lthn.mohawk.c 12.0% 25 435.6 442.4 378.1 517.7 39.9
10. 0.so-0-0-2.cr2.kdca.mohawk.c 12.0% 25 525.0 525.0 468.1 587.4 34.2
11. t13-2.dr2.kdca.mohawk.ca 20.0% 25 512.1 505.8 486.0 524.4 10.3
12. pix 4.0% 25 556.2 528.2 466.6 597.5 30.9

Basically it pings the hops sequentially so you can spot at which point packet loss occurs, or high latency kicks in. Our ISP engineers were squawking about upgrading router IOS, etc.. but this conclusively proved that the problem as at the far end in the USA landing sites.

If you aren't lucky enough to use Linux or FreeBSD (Its in the ports system) there is a Windows 2000/XP/2003 implementation here: http://winmtr.sourceforge.net/

Anyways this saved me a lot of time and effort in fighting with my ISP, hope it helps you...

Thursday, May 14, 2009

HTML2Image web frontend

For those interested in the technical details, Guangming Software distributes a mini web server with their program that only talks to their program. The program works from the command line or as a CGI from their web server (and no other, AFAIK). The first thing you need to do is start the background X imager like this (use the 64 Bit Ubuntu version, not the one they supply):

Download Location : http://www.guangmingsoft.net/htmlsnapshot/html2image.i386.RHEL3.tar.gz

Install the package into /usr/local/html2image, you'll also need to install some software thus:

apt-get install ia32-libs
apt-get install ia32-libs-gtk
apt-get install xserver-xorg
apt-get install xhost

Make sure the following is in /etc/environment:

LD_LIBRARY_PATH="/usr/lib:/usr/local/lib:/usr/local/lib/php/extensions:/usr/local/html2image"

Run XHOST to allow Screen access:

DISPLAY=:1
xhost +

Then you can start with running the Xvfb server:

LOG=/var/log/xvfb.log
cd /usr/local/html2image
Xvfb :1 -screen 0 1024x768x16 -nolisten tcp -auth X1.cfg >$LOG 2>&1 &

Then, you need to start their mini web server, running as the same user as the regular web server:

su www-data
/usr/local/html2image/nweb 8181 /usr/local/html2image

Then you can use a url to get an image:

http://localhost:8181/para=www.google.com?/tmp/image.jpg

The only problem is what you get back is the text output of the html2image program itself, not the image. The image goes to a file on the local server's hard drive (/tmp/image.jpg in this case) which isn't very helpful unless it runs on the same server you're working from. Create the html2image.php file in the www root:

$HOST = "http://localhost:8181";
$OPT = "/para=-H&768&-W&1024&-Q&100&-t&8000&";
$URL = $_GET["url"];
$OUTFILE = "/tmp/image".getmypid().".jpg";
$TMPFILE = "/tmp/image".getmypid().".txt";
#
# This should work, but you get some sort of
# a stream creation error that screws the image up.
#
#$lines = file($HOST.$OPT.$URL."&".$OUTFILE);
#
# This works, but isn't pretty!
#
exec("wget -O ".$TMPFILE." '".$HOST.$OPT.$URL."&".$OUTFILE."'");
header("Content-Type: image/jpeg");
$img = @imagecreatefromjpeg($OUTFILE);
imagejpeg($img);
unlink($OUTFILE);
unlink($TMPFILE);
?>

Execute the Image grab with the URL:

http://SERVERNAME_OR_IP/html2image.php?url=www.google.com

Jails (VM's) in FreeBSD 7

==Building your jail for the first time==

===Creating an appropriate make.conf===
You'll need to run make world (or make installworld) to create your jail.
If you don't want to install the whole kitchen sink you can use the make.conf
below. You can put it in your jail for future use and it'll be used by future
port builds inside your jail. One thing I've noticed is that make installworld
doesn't seem to respect and MAKE_CONF or __MAKE_CONF variables passed to it so
we'll just put it in /etc/make.conf for now.

Lets first back our current make.conf up:
cp /etc/make.conf /etc/make.conf.bak

And new one in there. Keep in mind, depending on what you want to use
this jail for you may want to modify this make.conf. For me this has worked
on building a variety of services from ports (inside the jail).
I like to name the below file make.conf.jail and copy it to make.conf,
then copy make.conf.bak back to make.conf when I'm done building the jail.

NO_ACPI= true # do not build acpiconf(8) and related programs
NO_BOOT= true # do not build boot blocks and loader
NO_BLUETOOTH= true # do not build Bluetooth related stuff
NO_FORTRAN= true # do not build g77 and related libraries
NO_GDB= true # do not build GDB
NO_GPIB= true # do not build GPIB support
NO_I4B= true # do not build isdn4bsd package
NO_IPFILTER= true # do not build IP Filter package
NO_PF= true # do not build PF firewall package
NO_AUTHPF= true # do not build and install authpf (setuid/gid)
NO_KERBEROS= true # do not build and install Kerberos 5 (KTH Heimdal)
NO_LPR= true # do not build lpr and related programs
NO_MAILWRAPPER=true # do not build the mailwrapper(8) MTA selector
NO_MODULES= true # do not build modules with the kernel
NO_NETCAT= true # do not build netcat
NO_NIS= true # do not build NIS support and related programs
NO_SENDMAIL= true # do not build sendmail and related programs
NO_SHAREDOCS= true # do not build the 4.4BSD legacy docs
NO_USB= true # do not build usbd(8) and related programs
NO_VINUM= true # do not build Vinum utilities
NO_ATM= true # do not build ATM related programs and libraries
NO_CRYPT= true # do not build any crypto code
NO_GAMES= true # do not build games (games/ subdir)
NO_INFO= true # do not make or install info files
NO_MAN= true # do not build manual pages
NO_PROFILE= true # Avoid compiling profiled libraries

# BIND OPTIONS
NO_BIND= true # Do not build any part of BIND
NO_BIND_DNSSEC= true # Do not build dnssec-keygen, dnssec-signzone
NO_BIND_ETC= true # Do not install files to /etc/namedb
NO_BIND_LIBS_LWRES= true # Do not install the lwres library
NO_BIND_MTREE= true # Do not run mtree to create chroot directories
NO_BIND_NAMED= true # Do not build named, rndc, lwresd, etc.

===Building the Jail===
Now for actually building your jail...

I'm defining JAILDIR here because I'm going to use it in a shellscript style example throughout the rest of this howto.

# Let's first make some directories
JAILDIR=/home/jail
mkdir -p $JAILDIR/dev
mkdir -p $JAILDIR/etc
mkdir -p $JAILDIR/usr/tmp
chmod 777 $JAILDIR/usr/tmp

cd /usr/src/

# You can replace the below with make installworld if you've built your
# world previously
make buildworld
make installworld DESTDIR=$JAILDIR
cd /usr/src/etc
cp /etc/resolv.conf $JAILDIR

make distribution DESTDIR=$JAILDIR NO_OPENSSH=YES NO_OPENSSL=YES
cd $JAILDIR

# At this point we'll mount devfs, and then hide the unneeded devs
mount_devfs devfs $JAILDIR/dev
devfs -m $JAILDIR/dev rule -s 4 applyset

# Create a null kernel
ln -s dev/null kernel

# Quell warnings about fstab
touch $JAILDIR/etc/fstab

# Use our existing resolv.conf
cp /etc/resolv.conf $JAILDIR/etc/resolv.conf

# Copy our settings for ssl
mkdir -p $JAILDIR/etc/ssl
mkdir -p $JAILDIR/usr/local/openssl
cp /etc/ssl/openssl.cnf $JAILDIR/etc/ssl
cd $JAILDIR/usr/local/openssl/
ln -s ../../../etc/ssl/openssl.cnf openssl.cnf

Make a decent rc.conf:

hostname="jail.example.com" # Set this!
ifconfig_em0="inet 10.0.0.20 netmask 255.255.255.255"
defaultrouter="10.0.0.1" # Set to default gateway (or NO).
clear_tmp_enable="YES" # Clear /tmp at startup.
# Once you set your jail up you may want to consider adding a good securelevel:
# Same as sysctl -w kern.securelevel=3
kern_securelevel_enable="YES" # kernel security level (see init(8)),
kern_securelevel="3"


You'll also want to make an alias on your interface for the ip above so we'll
do something like:

ifconfig em0 10.0.0.20 netmask 255.255.255.255 alias

Now you'll want to have devfs inside your jail, so to get it working for
the first time do this:
mount_devfs devfs $JAILDIR/devfs

And finally, copy your original make.conf back.
cp /etc/make.conf.bak /etc/make.conf

==Starting the jail for the first time==
OPTIONAL (but probably necessary):
You'll want to mount /usr/ports and /usr/src so you can install ports inside
your jail, unless you have another way you want to do this (such as downloading
packages).
mount_nullfs /usr/ports $JAILDIR
mount_nullfs /usr/src $JAILDIR

Now we can start our jail
jail $JAILDIR jail.example.com 10.0.0.20 /bin/sh

Once inside the jail you'll want to start services:
/bin/sh /etc/rc

While you're here you'll want to edit your password file since if someone
breaks into your jail, and starts cracking it you won't want them to have
the same passwords as your root system has. Also remove all users you
don't need in the jail:
vipw
passwd root

From here, assuming all went well you can do something like:
cd /usr/ports/security/openssh
make install clean

And build your port(s) inside your jail. Once you're finished be sure to
unmount the directories so a compromised jail can't build more ports.

I've verifed that truss works correctly in a jail
so between ldd and truss you should be set.

Also note that if you try to start your jail with just:
jail $JAILDIR jail.example.com 10.0.0.20 /bin/sh /etc/rc
but you have no services/daemons/programs set to run, the jail will simply start and then exit since there's nothing running inside.

==Getting it to start automatically==
You'll now need to put your settings in /etc/rc.conf
First put the alias you jail has in there:

ifconfig_em0_alias0="inet 10.0.0.20 netmask 0xffffffff"

===Editing the rc.conf===
For those of you that are looking to make your own rc script, I don't recommend it.
I've found issues getting devfs rules to be applied with the a script, and really this way is much easier. It's also the standard way and you can attach to jails later on quite easily
without using screen (read below).

Here's the standard rc.conf way of getting your jail to run at startup:

jail_enable="YES" # Set to NO to disable starting of any jails
jail_list="pk" # Space separated list of names of jails
jail_set_hostname_allow="NO" # Allow root user in a jail to change its hostname
jail_socket_unixiproute_only="YES" # Route only TCP/IP within a jail

jail_pk_rootdir="/usr/home/prison/pk"
jail_pk_hostname="pk.example.com"
jail_pk_ip="10.0.0.20"
jail_pk_exec_start="/bin/sh /etc/rc"
jail_pk_devfs_enable="YES"
jail_pk_devfs_ruleset="devfsrules_jail"

==Jail maintenance==
Of course from time to time you may have to upgrade ports in your jail, or the world in the jail itself. This isn't a big deal either. Instead of using jail (which makes its own IP address and everything) we can use chroot instead which is similar since all we're using is a simple shell
and then we'll be done with it.

First mount the dirs so they're accessable in the chroot:
mount_nullfs /usr/ports $JAILDIR
mount_nullfs /usr/src $JAILDIR

Connect to your jail:
find the jail id of the jail you are running with jls:
#jls
JID IP Address Hostname Path
1 10.0.0.20 pk.example.com /usr/home/prison/pk

Now connect to it using the JID:
jexec 1 /bin/sh

To upgrade your world:
cd /usr/src
make buildworld
make installworld
NOTE: If you've just done make buildworld previously you can do make installworld and install all the newly compiled binaries again.

To build a port:
cd /usr/ports/sysutils/example
make install clean
NOTE: You may also want to install portupgrade to make port management easier.

When you're done just exit:
exit

Solaris 10 installing Postfix, Spamassassin and DNS Blocklists

== Install Postfix ==

Install postfix with our pkg-get program (http://www.opencsw.org/pkg-get):

pkg-get -U
pkg-get -i postfix

== Disable Sendmail and enable Postfix ==

First disable sendmail:

/usr/sbin/svcadm disable svc:/network/smtp:sendmail

Then clear maintenance and enable postfix:

/usr/sbin/svcadm clear svc:/network/smtp/postfix:default
/usr/sbin/svcadm enable svc:/network/smtp/postfix:default
/usr/sbin/svcadm restart svc:/network/smtp/postfix:default

Test that postfix is now the MTA:

root@Mitsubishi-S10-T1SMTP - />telnet 0 25
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.
220 Mitsubishi-S10-T1SMTP.localdomain ESMTP POSTFIX
quit
221 2.0.0 Bye
Connection to 0 closed by foreign host.

Notice the POSTFIX on the 220 reply line - this means postfix is installed and working

== Enabling Postfix to deliver and receive mail ==

exit the /etc/opt/csw/postfix/main.cf file and make sure the following lines are unhashed and configured (current values included are for demonstration purposes only) :

inet_interfaces = $myhostname '''(specify which interfaces to listen on)'''
mynetworks_style = host '''(specify which type of access is allowed)'''
mynetworks = 10.0.0.0/24, 127.0.0.0/8 '''(specify which networks are allowed to relay)'''
relay_domains = $mydestination '''(domains that this mail server accepts mail for)'''
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain '''(list of acceptable domain suffix on addresses)'''
myhostname = Mitsubishi-S10-T1SMTP '''(defined hostname, make sure it matches the system hostname)'''
mydomain = kaizania.co.za '''(specify the domain that will be this mail servers mail suffix)'''

== Test the server ==

You can test the server to see that it relays mail:

root@Mitsubishi-S10-T1SMTP - /etc/opt/csw/postfix>telnet Mitsubishi-S10-T1SMTP 25
Trying 10.0.0.107...
Connected to Mitsubishi-S10-T1SMTP.
Escape character is '^]'.
220 Mitsubishi-S10-T1SMTP ESMTP Postfix
helo test
250 Mitsubishi-S10-T1SMTP
mail from: lionel.bisschoff@kaizania.co.za
250 2.1.0 Ok
rcpt to: someone@somewhere.com
250 2.1.5 Ok
data
354 End data with .
Subject : Test
Test
.
250 2.0.0 Ok: queued as 2404EDDD1B

And we see the message in the delivery queue:

root@Mitsubishi-S10-T1SMTP - /opt/csw/bin>./mailq
-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
2404EDDD1B* 378 Tue May 27 10:16:23
lionel.bisschoff@kaizania.co.za
someone@somewhere.com

There we go - working now !!!

== Adding Spamassassin anti-spam ==

First, installing spamassassin:

pkg-get install spamassassin

Then the easiest way to integrate postfix and spamassassin is to use spamd in an "after-queue" inspection. This configuration does not allow rejecting messages within the SMTP transaction, so it unfortunately contributes to backscatter email. On the other hand, it has important performance advantages over "before-queue" inspection.

First, edit /etc/postfix/master.cf, find the

# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (50)
# ==========================================================================
...
smtp inet n - n - - smtpd
...

line and just add " -o content_filter=spamassassin" to the end of the line:

# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (50)
# ==========================================================================
...
smtp inet n - n - - smtpd -o content_filter=spamassassin
...

## you can change the maxproc to 400 if you have a strong server (dual xeon 2.8 nd above) and the performance will be much better.

Then, go to the end of the file, and add this:


# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (50)
# ==========================================================================
...
spamassassin
unix - n n - - pipe
user=nobody argv=/path/to/spamc -e /path/to/postfix/sendmail -oi -f ${sender} ${recipient}
# make sure it's all on one line or
# start the consecutive lines with a whitespace
# like I did here

Make sure that you have adjusted the path to the spamc and sendmail commands above! (Please note that the path required is Postfix's sendmail and not the standalone package, Sendmail the SMTP server. It will not work if you're not careful about which one is installed). Then, setup spamd to start with the system, and you are ready to go. If you wish to provide spamassassin preferences, change "user=nobody" to a valid system user (except for root, since Postfix will NOT call external programs as root), and add .spamassassin into that user's home directory.

If you use user preferences stored in SQL, you should change "spamassassin" service in master.cf to following:

# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (50)
# ==========================================================================
...
spamassassin
unix - n n - - pipe
flags=Rq user=nobody argv=/path/to/spamc -u ${recipient} -e /path/to/postfix/sendmail -oi -f ${sender} ${recipient}

Notice "-u ${recipient}" added. Otherwise "username" field in database will always appear as user which postfix is invoking spamc(in this example it is 'nobody').

Only mail received by SMTP will be scanned with this method, i.e. mail injected with sendmail(1) will not be fed to SpamAssassin.

== Adding a Blacklist to Postfix ==

to add a Blacklist to block certain senders, do the following:

Create a blacklist file in the postfix /etc directory (i.e. /etc/opt/csw/postfix/blacklist) and add the addresses you wish to block within:

support@connect2casino.com REJECT

With this file created, make it a DB file with the following command:

/opt/csw/sbin/postmap hash:/etc/opt/csw/postfix/blacklist

Then at the end of the postfix /etc/opt/csw/postfix/main.cf file, add the sender check line:

smtpd_sender_restrictions = reject_unknown_sender_domain, check_sender_access hash:/etc/opt/csw/postfix/blacklist

The added "reject_unknown_sender_domain" will also reject domains that do not exist (a DNS Lookup is performed). Restart the Postfix service to effect the changes:

svcadm disable /network/smtp/postfix; svcadm enable /network/smtp/postfix

And test that the sender address is now rejected:

Iridium-S10-SMTP(root)/var/log# telnet 10.1.2.24 25
Trying 10.1.2.24...
Connected to 10.1.2.24.
Escape character is '^]'.
220 iSMTP.actionpokernetwork.com ESMTP Postfix
helo dorks
250 iSMTP.network.com
mail from: support@connect.com
250 2.1.0 Ok
rcpt to: operations@network.com
554 5.7.1 : Sender address rejected: Access denied

Network link aggregation in Solaris

Aggregation is better than failover in the fact that it provides two or more datapaths, effectively doubling/tripling/etc... bandwidth between the server and the switch. Failover is still enabled, as if any network cable is removed the system will continue to function correctly.

To setup Aggregation:

Make sure the ports to be added to the aggregated port are unplumbed, create the aggregation thus:

 dladm create-aggr -d e1000g0 -d e1000g1 1

This will create an aggregated port "aggr1" that comprises e1000g0 and e1000g1, to show the the contents of the aggregated port:

 dladm show-aggr
key: 1 (0x0001) policy: L4 address: 0:14:4f:1:9c:98 (auto)
device address speed duplex link state
e1000g0 0:14:4f:1:9c:98 100 Mbps full up attached
e1000g1 0:14:4f:1:9c:99 100 Mbps full up attached

Bring up and configure the new port:

 ifconfig aggr1 plumb 10.0.0.10 netmask 255.255.255.0 up

and display it:

 ifconfig aggr1
aggr1: flags=1000843 mtu 1500 index 6
inet 10.0.0.10 netmask ffffff00 broadcast 10.0.0.255
ether 0:14:4f:1:9c:98

Edit the /etc/hosts and hostname.aggr1 file to make changes persistent.


Increasing the width of a Blogspot blog

Pretty east one this,

Under layout, click on Edit HTML and change the "width" elements in the following code:

/* Outer-Wrapper
----------------------------------------------- */
#outer-wrapper {
width: 1025px;
margin:0 auto;
padding:10px;
text-align:left;
font: $bodyfont;
}

#main-wrapper {
width: 710px;
margin-left: 20px;
float: left;
word-wrap: break-word; /* fix for long text breaking sidebar float in IE */
}


I set my outer wrapper width to 1025 pixels and then the main wrapper width to 710 pixels.

Wednesday, May 13, 2009

Increasing the speed of rTorrent downloads

I use rtorrent to get all my *cough* legal software downloads, and I found a way to optimize the source code of the libtorrent libraries to increase speeds in rtorrent.

I run this in FreeBSD, but its cross portable to most *nix'es...

BSD uses the ports system, and luckily all source files get dumped into /usr/ports/distfiles - I unzipped the libtorrent-0.12.2.tar.gz file into /usr/ports/distfiles/libtorrent-0.12.2.

I modified the following files:

in ./configure

set PEER_NAME to UT161

PEER_NAME "UT161"

this tells all trackers that your client is uTorrent 1.6.1 - which is widely accepted as a "good" version for private trackers and general public domain.

next, edit the ./src/protocol/peer_connection_leech.cc file:

old line #:114 if (cachedTime - m_timeLastRead > rak::timer::from_seconds(240))
new line #:114 if (cachedTime - m_timeLastRead > rak::timer::from_seconds(40))

This will drop an inactive client after 40 seconds instead of 4 minutes, allows faster uptake of seeds and peers.

next, edit ./src/download/download_wrapper.cc:

old line #:272 if (ticks % 4 == 0) {
new line #:272 if (ticks % 1 == 0) {

This will clear ncurses connected clients after 1 tick instead of 4 (1 tick equals 30 seconds)

next, edit ./src/torrent/tracker.cc:

old line #:49 m_normalInterval(1800),
new line #:49 m_normalInterval(600),

This will reconnect to the tracker every 10 minutes to check for new seeds/leechers instead of every 30 minutes.

next, edit ./src/protocol/peer_connection_leech.cc:

old line #:153 if (!download_queue()->canceled_empty() && m_downStall >= 6)
new line #:153 if (!download_queue()->canceled_empty() && m_downStall >= 2)

This will request new file pieces from peers every 2 ticks instead of 6

on BSD, I had to make sure openssl was installed (/usr/ports/security/openssl)

then the installation of libtorrent:
/usr/ports/distfiles/libtorrent-0.12.2>./configure --prefix=/usr/local; make && make install

once thats done, just install rtorrent itself: /usr/ports/net-p2p/rtorrent> make all install

for added performance, I setup my .rtorrent.rc file thus:

bind = xxx.xxx.xxx.xxx (Your local IP address that rtorrent binds to)
ip = xxx.xxx.xxx.xxx (the *Misreported* IP given to trackers he he!)
check_hash = yes (Hash check completed files for consistency)
directory = /root/.incoming (download directory)
session = /root/.session (Session directory, if rtorrent crashes, it will recover from torrents in here)
peer_exchange = yes (exchange tracker and peer info with connected systems)
upload_rate = 1 (I limit uploads to 1kb/sec)
download_rate = 60 (I limit downloads to 60kb/sec)
max_peers = 500 (maximum amount of connected peers per torrent)
dht = on (DHT for trackerless clients, helps a bit)

hash_read_ahead = 8 (FreeBSD Specific)
hash_max_tries = 5 (FreeBSD Specific)
hash_interval = 10 (FreeBSD Specific)

tos=lowdelay (Set TCP Type-Of-Service to lowdelay, best possible peers)
use_udp_trackers = yes (Useless if behind a firewall)

max_memory_usage = 2147483648 (2Gb maximum physical memory used)

A torrent that I was struggling with at about 1.1 Kb/sec shot up to 15Kb/sec after these changes - I can definately see much better speeds. These changes will also help on shaped bandwidth, where torrents are throttled by ISPS.

My First Blog !!!

I though I'd create my own technical blog, with the amount of problems and solutions I've come across (through highly convaluted and misleading articles out there), I've found solutions not publicly visible - I guess this is my contribution back to the net for all the times it's helped me...

Going to TRY and post one article per working day...