Category Archives: sysadmin

security sysadmin

Quick SSH security tips

Just a quick post about a page I stumbled across, and I merely want to keep it in bookmarks. I was talking to some people to secure public facing SSH servers; and while we have the obvious:

  • Only allows SSH2
  • Disable root logins
  • Use keypairs instead of passwords
  • Implement fail2ban

When researching to make internet exposed SSH boxes more secure, I’ve been implementing recently MFA (Multi Factor Authentication). Usually found when you use something you know (password, PIN, …) and something you have (smartcard, keyfob, mobile, mobile app, …). In SSH, this can be achieved with the following; it forces you to authenticate with your keypair AND your password.

Match User johndoe
    AuthenticationMethods publickey,keyboard-interactive

This, and more tips were found on https://sysconfig.org.uk/two-factor-authentication-with-ssh.html, so check it out.

misc sysadmin

Error mongoDB under Ubuntu – can’t find libmozjs.so

When installing MongoDB under Ubuntu, one could stumble upon the following problem:

mongo: error while loading shared libraries:
libmozjs.so: cannot open shared object file: No such file or directory

This is caused because MongoDB uses XulRunner for it’s operations. (This is also why you see a bunch of X11 programs that are to be installed when using apt-get). XulRunner, however, contains this file, but is not loaded.

Solving the problem:
Make sure you have xulrunner installed (sudo apt-get install xulrunner-1.9.2) and add its path to ldconfig’s configuration files, which are in/etc/ld.so.conf.d. You could create a file, say, /etc/ld.so.conf.d/xulrunner.conf, containing the following line:

/usr/lib/xulrunner-1.9.2.10/

After this, you chould run ldconfig (as root) and that should do the trick. Let me know if you have any problems with this.

sysadmin

How to shrink Microsoft SQL log files

When having Microsoft SQL databases, its log files can grow quite a bit, potentially slow down the database server and eat up disk space.

To shrink a database, one can run the following line:

EXEC D_ShrinkDBLogs 0,100,1000,'with truncate_only','DB_NAME'
(change DB_NAME with the database’s name)

To shrink all databases, one can use “sp_MSforeachdb” which is an undocumented sql stored procedure:

EXEC sp_MSForEachDB 'D_ShrinkDBLogs 0,100,1000,''with truncate_only'',''?'''

To run this, the following stored procedure need to be installed. It can just be copy pasted from the code below (or download from this link):
read more »

sysadmin

Debian won’t boot on BL465 server

When installing Debian on a HP BL465 server, we sometimes run into the problems that after installing, GRUB won’t boot up anymore.

First off, it’s not a Debian specific problem, we had this problem with other Linux distributions also, but decided to standardize on Debian across the organization.

Installing to the MBR fails, and you have to install grub on /dev/cciss/c0d0, although upon rebooting after the installation, Grub throws out an error that it cannot boot off (hd1,0).

When you encounter this, you need to edit the configuration (by pressing e in the initial grub screen), and change the (hd1,0) to (hd0,0).

This did the trick for us,

Thanks,
Michael

sysadmin web

Reverse proxy for sharepoint on Linux using HAProxy

At Nakheel, we needed to load balance a new sharepoint instance. Our new sharepoint is single sign on, and was running on 2 web servers which needed to be load balanced. We played around with Apache for a while, and it’s awesome proxy balancer, but it gave us the problem that it was always asking for a username and password.

Apache was used, since I have a reasonable amount of experience with it load balancing servers such as Webrick, etc. After a few frustrating hours of messing with NTLM, Christian proposed a few alternatives for this.

Having this in mind, we decided to go for HAProxy, to provide load balancing and a reverse proxy for our sharepoint instance. The good this is that it is a very simple tool, it accept HTTP conenctions, and forward them.

Below is our simplified /etc/haproxy/haproxy.cfg file

global
        maxconn 4096
        user haproxy
        group haproxy
        daemon
        # debug

defaults
        mode    http
        option  forwardfor
        log     127.0.0.1 local0 notice
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

backend sharepoint
        balance roundrobin
        option redispatch
        cookie SERVERID insert nocache
        server sp1      172.30.16.11:80  cookie spsrv01 weight 30 check
        server sp2      172.30.16.12:80  cookie spsrv02 weight 30 check

frontend httpid
        bind *:80
        acl hosts_sharepoint hdr_end(host) -i intranet.domain.com
        acl hosts_sharepoint hdr_end(host) -i intranet.domain.com:80
        use_backend sharepoint if hosts_sharepoint
        default_backend sharepoint

The configuration is very straightforward, and it got rid of our continuous username/password boxes, especially under firefox.

Hope this helps,
Michael

sysadmin

Changing NSClient++ configurations in bulk

At work, we started using Nagios to monitor the hosts and their services for any issues, so we can, proactively, take actions when we see trouble coming (hard disk that fills up rapidly, restarting services, etc).

Recently we change the Nagios host to another IP address and found ourselves having to change a few dozen NSClient.ini files on the server. Even though we use DNS names as monitoring host (only nsclient access on port 12489/tcp is allowed from a certain host), NSClient by default caches the IP address (Using cache_allowed_hosts, which is set to “1” by default) so we were bound to change quite some files.

To do this, we have to do the following: read more »

sysadmin

Change the ILO server name under Linux

At work, we have a lot of HP Blade servers, and manage these through HP Onboard Administrator. By default, the server names are adapted from the Windows hostname, due to HP Insight Management agent tools. When running Linux, there doesn’t seem to be a option, or at least I couldn’t find one.

While questioned a few times on HP forums (such as this thread), questions rose to “installing windows on the server, then reinstall Linux”, it is possible in the web interface to be set. It’s not obviously placed, and hence might require some searching. read more »

sysadmin

Microsoft Virtual Server 2005 – network trouble?

Hi all,

In our intranet development environment we run Microsoft Virtual Server, it was setup by the consultants who developed our Intranet’s first phase. Now, the second phase is about the start, we dusted off those servers and found out that the virtual machines all had no network adapters installed anymore.

After cursing, and making a plan of migrating this to VMWare’s ESX (sorry consultant) next week, we found out that the only way of solving this was to remove the virtual adapters from the Virtual Server console, and adding them again. In the client machine’s, a static IP assigned adapter was trying to fetch an IP address (yeah, go figure). By just setting it to dynamic (dhcp), and then reassigning the static address, it was solved.

I guess Microsoft is about just re-trying it, rebooting, re-adding. But we’ll be switching to ESX anyways.

Thank you,
Michael

code misc sysadmin

nnscfgmaker.sh: a nagios/nsclient++ cfg maker

Dear all,

We are in the process of changing the monitoring system on part of our network from Zenoss to Nagios. This is not a Zenoss vs. Nagios debate, as both products are awesome and do the things they are designed for very well. We (Christian and myself) use a combination of Cacti for bandwith monitoring and Zenoss for server and device monitoring. Now, recently we decided to change the latter to Nagios. It is know for its “great deal of flexibility when integrating Nagios into their environment” (Galstad, 2005)

As Zenoss was configured using SNMP Informant to grab information about the MS Windows servers (available disk space, CPU load, etc) and Nagios uses NSClient++; one of the time consuming tasks was getting the client on the server (thank you domain 🙂 ) and configuring each and every server on the Nagios server. I guess there should be some sort of discovery tool for Nagios, but I couldn’t directly find one.

In order to tackle the copy-pasting for a few dozen config files, and to brush up my bash scripting again, I wrote the Nagios Nsclient++ cfg maker read more »

code sysadmin

Tar based incremental backups

A small bash script I wrote to have incremental backups done on a unix server, and then pushed to a Windows File system. On the fileserver, we add this directory to the normal backup.

This is on a mail server, where emails are stored in MailDir format. We create weekly full backups on sunday, and daily incremental. This script is called daily at night from a cron job. Gotta love the scripting abilities of bash.

It might help you out, so here goes:

#!/bin/bash
# backup script is doing following items
# dump all incremental email into a backup file, gzip the backup file and
# move the file to an external file server

START_TIME=`/bin/date`
echo "backup started at: ${START_TIME}"

DOW_N=`/bin/date +"%w"` # number, 0 (sun), 1 (mon)
DOW_T=`/bin/date +"%F"`

TO_BACKUP="/opt/maildata/"
TEMP_FILE="/tmp/${DOW_T}_mail_backup.tar"
BACKUP_LOG="/tmp/mail.backup"
FILE_SERVER="/mnt/fileserver/" # mounted over SMB

# if it's a sunday, delete the incremental file and take a full backup
if [ ${DOW_N} -eq "0" ]; then
  /bin/rm ${BACKUP_LOG}
fi

/bin/tar -c -f ${TEMP_FILE} --listed-incremental=${BACKUP_LOG} ${TO_BACKUP}
/bin/gzip -f ${TEMP_FILE}
FILE_SIZE=`/bin/ls -lah ${TEMP_FILE}.gz | awk '{ print $5 }'`
/bin/mv ${TEMP_FILE}.gz ${FILE_SERVER}

## report, this goes in an email through cron
END_TIME=`/bin/date`; export END_TIME
echo "backup ended at: ${END_TIME}"
echo "data moved: ${FILE_SIZE}"