I should write more Node.js posts

They are consistently my top rated posts from an SEO standpoint. Also, according to Google Analytics, they lead to an average of a good 5 minutes on the site.  I will come up with more, but I feel like I'm not doing anything noteworthy or extraordinary with Node.js at the moment. I've put a lot of time and coding into the foundation of my site and other sites I've written, and I've already written a lot about building up that foundation.  I will come up with some other stuff to write about, it's what the people want!

Fixed!

The latest update to run the webserver with nohup seems to have fixed my server issues. I did get a server crash soon after changing the script, but I couldn't track it down, but it's been running non-stop for a few days now.

Finally I can put this to rest. Also, out of it, I got more efficient tracking as well as more detailed error messages in some instances, and less detailed error messages in others (for files not found, for instance, it just spits out 'Not found: ' and the file name, instead of a whole stack trace.).

Moment of Truth #3

After adding some better logging and stuff in the webserver, I determined that no error was causing the server to close. While I was starting the service with node webserver &, I think that I still needed to do nohup.

We shall see. My nohup'ed webserver is now running, and will run always in the future, whether it dies randomly or not. It's just the better way to start it.

Moment of Truth #2

After upgrading to Node.js 0.10.21 and implementing domains in Node.js, I was still getting errors. Since then, I've added code to listen for the response 'close' event, as well as the server 'clientError' event, in an effort to track down the issue. I think I may be on to something. At least then I'll be able to go to stack overflow or github with the issue in hopes that it will be fixed, or the act of listening for those events and logging the errors will lead me to the fix myself. Here's hoping.

The Moment of Truth

Every morning I have an email from pingdom that my site went down overnight. I've upgraded to node.js 0.10.20 a few weeks ago to take advantage of some other bug fixes and optimizations, and it shows in the Google Analytics page load times (they're all 0 seconds)

I have had no luck tracking down the cause, but I read about how to prevent a server error from bringing down the site. I've implemented the suggested solution using node.js domains, and we will see what's happening tomorrow.

It might be the same thing because when I go into the server, node is still running my webserver, it just appears the socket was destroyed... We shall see. Wish me luck!

Flickr Integration Complete!

It really didn't take too long! What I outlined in the previous post is exactly what it does:

  • Check flickr for my photos that were tagged with "jtccom". Download the 2048 pixel wide version of it
  • Write JS + Node part of my website which shows, in a queue like manner, an image that hasn't been processed
  • Clicking on the image shows a box of where I clicked, where it would be cropped, according to the sizes defined for the header
  • Send request to the server, which uses GraphicsMagick to crop it and then create different scaled images based on the sizes defined for the site

Here are some screenshots:

At first, you are prompted with the next image which hasn't been processed


Next, you select the point where you want to crop it. Sizes are pre-determined, so there's no dragging and resizing a bounding box, it knows all the sizes and the size of the images, so it just does it for you


Click the process button when you've made your crop selection


Wait a second or two while Node.js and gm (GraphicsMagick) processes your photos.

And GraphicsMagick code in node.js, which is really helpful, and I was able to get it to work on Windows

this.handlePost = function(site, query, finishedCallback){ var tmpdir = path.normalize(site.path + site.config.tempDownloadFolder); var processeddir = path.normalize(site.path + site.config.processedFolder); var form = query.form; var sizes = site.config.imageWidths; var heights = site.config.imageHeights; var filename = form.image.substring(form.image.lastIndexOf("/")+1); var fileParts = filename.split("."); sizes.sort(function(a,b){ return b - a; }); heights.sort(function(a,b){ return b - a; }); var x1 = parseInt(form.x1, 10), x2 = parseInt(form.x2, 10), y1 = parseInt(form.y1, 10), y2 = parseInt(form.y2, 10); var w = x2 - x1, h = y2 - y1; // process first size, use that for base of resizes var croppedPath = processeddir + fileParts[0] + "-" + sizes[0] + "." + fileParts[1]; gm(tmpdir + filename).crop(w, h, x1, y1).write(croppedPath, function(err){ var sync = new SyncArray(sizes); sync.forEach(function(size, index, array, finishedOne){ if (index > 0){ var scaled = processeddir + fileParts[0] + "-" + size + "." + fileParts[1]; gm(croppedPath).resize(size, heights[index]).write(scaled, function(x){ finishedOne(); }); } else finishedOne(); }, function(){ finishedCallback({ content: JSON.stringify({ success: true }), headers: {"Content-Type": "application/json"} }); }); }); }

Again, it uses my custom built webserver and the SyncArray object that I also wrote.

Flickr code was pretty simple too. Here's that, accessing the Flickr API (no auth) with Node.js

var http = require("http"), querystring = require("querystring"), SyncArray = require("syncarray").SyncArray; this.getPhotosByTag = function(apiKey, user, tag, callback){ var self = this; var method = "flickr.photos.search"; var qs = { method: method, api_key: apiKey, user_id: user, tags: tag, format: "json", nojsoncallback: 1 }; var req = { host: "api.flickr.com", path: "/services/rest/?" + querystring.stringify(qs) }; http.get(req, function(res){ var json = ""; res.on("data", function(d){ json += d; }).on("end", function(){ var photos = JSON.parse(json).photos.photo; if (photos.length > 0){ var sync = new SyncArray(photos); sync.forEach(function(photo, index, array, finishedOne){ self.getPhotoSizes(apiKey, photo.id, function(sizes){ photo.url = sizes.filter(function(s){ return s.label == "Large 2048"; })[0].source; finishedOne(); }) }, function(){ console.log("url = " + photos[0].url) callback(photos); }) } else callback([]); }); }); } this.getPhotoSizes = function(apiKey, photoId, callback){ var method = "flickr.photos.getSizes"; var qs = { method: method, api_key: apiKey, photo_id: photoId, format: "json", nojsoncallback: 1 }; var req = { host: "api.flickr.com", path: "/services/rest/?" + querystring.stringify(qs) }; http.get(req, function(res){ res.setEncoding("utf8"); var json = ""; res.on("data", function(d){ json += d; }).on("end", function(){ var sizes = JSON.parse(json).sizes.size; console.log(sizes.length); callback(sizes); }); }); }

The next step is to update the front end css to include all sizes of a version of the image, and switch between them using the respond.js and media queries. That should be simple, but it's late and I'm going to bed!!

Enjoy! Leave a comment.

Tag List Added

I recently went about aggregating the tags used on my posts to create a sort of tag cloud. I never liked the display of tag clouds, so I just list them out in order of occurrence, with the most frequent showing first.

This should help me get some traffic. Node.js and MongoDB are super fast. It doesn't even stutter when loading the site, across 500+ posts. Actually, I have no idea how many there are.  Super fast.
Here's the code which pretty much finishes in -5 seconds
var db = require("../db"); this.tagCloud = []; this.initialize = function(site, callback){ var self = this; while (self.tagCloud.pop()); db.getPosts(site.db, {}, -1, -1, function(posts){ var tags = {}; posts.forEach(function(post){ if (post == null) return; for (var i = 0; i < post.tags.length; i++){ if (tags[post.tags[i]] == null) tags[post.tags[i]] = { tag: post.tags[i], count: 0 }; tags[post.tags[i]].count++; } }); for(var tag in tags){ if (tags[tag].count > 8) // arbitrary limit so we don't list like 200 tags with 1 post each self.tagCloud.push(tags[tag]); } self.tagCloud.sort(function(a,b){ return b.count - a.count; }); callback(); }); }

Brutal Node.JS errors

I haven't been paying much attention to the site, and at the worst time, there have been a series of crashes due to those things previously mentioned on this site. The site was down like 50% of July, sometimes going days without me restarting the node.js services. I assure you it is node.js' fault and not my code :)   Really, though, there's a socket issue on the OS that this is running on, which is Cent OS 5.5 or something. I will again try to upgrade to the latest one that's out, 0.10.15 and see if that fixes it.  Stay tuned.

Node.js Keeps Me Sane

Scenario:  We're rewriting the code for a site but it's pretty much using the same javascript and css currently. So I have a copy of the old version running and a copy of the new version. And something is not perfect in the new site. A Javascript problem.  To eliminate differences in javascript as a cause of the problem, here's what I did:


var http = require("http"), fs = require("fs"), SyncArray = require("syncarray").SyncArray, fshelp = require("filesystem"); require("strings"); var web1 = "http://localhost/"; var web2 = "http://oldsite.com/"; var name1 = "new"; var name2 = "old"; function processRequest(baseUrl, url, name, baseDir, finished){ fshelp.ensureFolder(baseDir, function(){ http.get(baseUrl + url, function(resp){ var file = url.lastIndexOf("/") == url.length - 1 ? "_index.html" : url.substring(url.lastIndexOf("/")); resp.setEncoding("utf8"); var data = ""; resp.on("data", function(d){ data += d; }); resp.on("end", function(){ fs.writeFile(baseDir + "\\" + file, data, "utf8", function(err){ if (err) console.log("error encountered writing file - " + JSON.stringify(err)); var files = findUrls(data); console.log(files.length); var sync = new SyncArray(files); sync.forEach(function(f, i, arr, finishedOne){ console.log("processing " + f); processRequest(baseUrl, f, name, baseDir, function(){ finishedOne(); }); }, function(){ finished(); }) }); }); }); }); } function findUrls(content){ var files = []; var reg = /(src|href)=["|']([a-zA-Z0-9\._\-\/]*?)["|']/g; var m = null; while ((m = reg.exec(content)) != null){ if (m[2].trim().length > 0 && m[2].indexOf(".js") != -1){ files.push(m[2]); } } return files; } processRequest(web1, "", name1, "c:\\webs\\" + name1, function(){ console.log("Done " + web1); }); processRequest(web2, "", name2, "c:\\webs\\" + name2, function(){ console.log("Done " + web2); });

After that was complete, I then used WinMerge.  It would find all javascript files in the html of the home page (what we were having trouble with), and save them all to a folder.  It turned out they were identical. But part of narrowing in on the problem is eliminating other possible causes. And I found the problem in the HTML, in where some JS files were being included. Some in head, some below.  Putting them all below fixed everything.

I write Node.js in work too.

Read my blog post on Node.js as a tool to get work done...

I'm an IDIOT!!

I spent a ton of time trying to write a syntax parser for a meta-language which would be parsed by Javascript.  I just replaced a lot of confusing logic with about 40 lines of code.

First, Javascript is dynamic and has stuff built in that can evaluate arbitrary code passed in via a string.

I'm not talking about eval().

Function!

However, my syntax looks like this:  Hello {{page.user == null ? 'Anonymous' : page.user.firstName }}

So, we need to know what "page" is in the context of the function.  I still have to parse out top level variables.  In the code above, it will get "page" as a top level variable.

Then, I build up the function:

var f = new Function(vars[0], body);

"body" is actually modified, I set it to "return " + body;  So that I can do {{ page.user == null ? 'Anonymous' : page.user.firstName }} and it will return the display name instead of undefined, the default behavior of a void function.

I have to count up the number of variables used, and build the function accordingly.  Currently, this is a switch statement.

switch (vars.length){
     case 0: f = new Function(body); break;
     case 1: f = new Function(vars[0], body); break;
     case 2: f = new Function(vars[0], vars[1], body); break;
}

Luckily in my code, there aren't more than 3-4 "top level" variables, including globals like "Array" and "String".

Here's the variable parsing part:


var shared = require("./shared"); require("strings"); var constants = { "true": true, "false": true, "null": true }; var lookup = {}; this.getVariables = function(body){ if (body in lookup) return lookup[body]; var vars = []; var instr = false; var instrch = null; var buf = ""; var toplevel = false; var result = null; for (var i = 0; i < body.length; i++){ var ch = body.charAt(i); switch (ch){ case "'": case "\"": instr = ch != instrch; instrch = instrch == null ? ch : (instr ? instrch : null); break; } if ((!instr && shared.tokenSeparator.test(ch)) || i == body.length-1){ if (i == body.length-1) buf+= ch; if (!toplevel && (result = shared.variable.exec(buf)) != null && !(result[1] in constants)){ if (!vars.some(function(d){ return d == result[1]})){ vars.push(result[1]); toplevel = true; } } buf = ""; } else if (instr) buf = ""; else buf += ch; if (toplevel && (instr || (shared.tokenSeparator.test(ch) && ch != "." && ch != "]"))) toplevel = false; } lookup[body] = vars; return vars; }

And here's the evaluation part:

var syntax = require("./syntax"); var shared = require("./shared"); var lookup = {}; var Evaluator = function(globals){ this.globals = globals; } Evaluator.prototype.evaluate = function(body, context){ body = shared.replaceComps(body); var vars = syntax.getVariables(body); var args = []; body = "return " + body; for (var i = 0; i < vars.length; i++){ if (context && vars[i] in context) args.push(context[vars[i]]); else if (this.globals && vars[i] in this.globals) args.push(this.globals[vars[i]]); } if (body in lookup){ return lookup[body].apply(null, args); } var f = null; switch (vars.length){ case 0: f = new Function(body); break; case 1: f = new Function(vars[0], body); break; case 2: f = new Function(vars[0], vars[1], body); break; case 3: f = new Function(vars[0], vars[1], vars[2], body); break; case 4: f = new Function(vars[0], vars[1], vars[2], vars[3], body); break; } var result = null; if (f != null){ result = f.apply(null, args); lookup[body] = f; } return result; } this.Evaluator = Evaluator;

shared.js has a regular expression, a map (for old syntax considerations), and a function to replace some old ways of doing things with the new, pure Javascript way of doing it.

this.replCompRegex = / (eq|ne|gt|lt|gte|lte) /; this.replCompMap = { eq: "==", ne: "!=", gt: ">", lt: "<", gte: ">=", lte: "<=" }; this.replaceComps = function(body){ var res = null; while ((res = this.replCompRegex.exec(body)) != null){ body = body.replace(res[1], this.replCompMap[res[1]]); } return body; }

Downgrade to Node.js v0.8.23!

Something's going on with 0.10.9 and 0.10.10, where it can't keep my server running. It appears to be related to streams or sockets. But 0.8.23 is working so I will just leave it be until the Node developers can get their acts together :)

Upgrade to Node.js 0.10.10!

I know the previous post was my upgrade to 0.10.9.  But then I started getting an EventEmitter error on Stream end.  So I downgraded to 0.8.23 in the meantime, until it was fixed. But then I kept reading, and found out it was fixed in 0.10.10.  So I did the git checkout v0.10.10 and make && make install && restart node webserver.  After this, it was still getting the error but it wasn't crashing node.js. At this time I saw my error logs flood with file not found (404) errors, and the node.js process was at a consistent 65%.  So, I figured it was working and I'll check again when I got home.  I'm happy to report, it was just a DDOS attack or something, and the CPU is back down to 2% for Node.js like it should be.

Close call.

Node 0.10.9

Upgraded!  I'm trying to get all of my computers up to it, so far it's just my mini (netbook) that I use on the couch for dev work, and the server. Left to go = home "real" laptop, and work machine. 

Solving [WordHero]

I took pretty much a brute force attack at coming up with this app that solves [WordHero]. It is pretty efficient and takes longer to type in the letters of the puzzle than for it to solve it.

Here's how I went about it.

So, you get a puzzle with 16 (or so, as you can see with the Qu block) letters arranged in a 4x4 grid.  Obviously this is a dictionary brute force attack, in the most literal sense.

What I do is start at the top left block, finding words that start there, then when all options are exhausted, move onto the next block and find all words that start there, and so on until we've covered all blocks. I wrote this solution in Node.js. First, there is the LetterBox class:

var LetterBox = function(letter, row, col){ this.letter = letter; this.row = row; this.col = col; this.used = false; } LetterBox.prototype.clone = function(){ var lb = new LetterBox(this.letter, this.row, this.col); return lb; }

The letter box can only be used once during a word solve. The next is the Board, which consists of LetterBoxes

var Board = function(str){ this.originalString = str; this.makeBoard(str); this.letterBoxLookup = {}; var self = this; this.letterBoxes.forEach(function(lb){ self.letterBoxLookup[lb.row + "" + lb.col] = lb; }); this.rows = 4; this.cols = 4; }

I implemented clone on both the Board and LetterBox classes.
The last piece is the BoardSolver class. We'll just look at two functions of it:
Solve:

BoardSolver.prototype.solve = function(max){ for (var i = 0; i < this.board.rows; i++){ for (var j = 0; j < this.board.cols; j++){ var boardinstance = this.board.clone(); this.getWords("", i, j, boardinstance, this.dictionary); } } var list = []; for (var w in this.foundWords) list.push(w); return list; }

And getWords:

BoardSolver.prototype.getWords = function(cur, row, col, boardinstance, dictlocal){ var letter = boardinstance.getLetter(row,col); if (letter == null || letter.used) return; var curlocal = cur + letter.letter; var potlocal = this.potential(curlocal, dictlocal); if (potlocal.length == 0) return; // no potentials down this path var wordlocal = curlocal.length > 2 ? this.matches(curlocal, potlocal) : []; var self = this; wordlocal.forEach(function(w){ if (!(w in self.foundWords)) self.foundWords[w] = w; }); letter.used = true; //var recurseClone = boardinstance.clone(); // n,s,e,w this.getWords(curlocal, row, col-1, boardinstance.clone(), potlocal); this.getWords(curlocal, row, col+1, boardinstance.clone(), potlocal); this.getWords(curlocal, row+1, col, boardinstance.clone(), potlocal); this.getWords(curlocal, row-1, col, boardinstance.clone(), potlocal); // diagonals this.getWords(curlocal, row-1, col-1, boardinstance.clone(), potlocal); this.getWords(curlocal, row+1, col+1, boardinstance.clone(), potlocal); this.getWords(curlocal, row+1, col-1, boardinstance.clone(), potlocal); this.getWords(curlocal, row-1, col+1, boardinstance.clone(), potlocal); }

So, while we're recursing, we'll get the current potential word (previous cur + this letter) and see if there are any potential words, words that start with the letters in curlocal. If there are no potential matches, stop recursion. If there are potential matches, keep going, also, store the current matched words at the current length (e.g. "break" is found on the way to finding "breaking")

To speed things up a bit, I only then search the potential list in the future attempts to narrow it down. Notice I'm passing the potlocal (potential local dictionary) into getWords on recursion.

To use the app currently, I just modify the string of characters. They are entered as follows (for example, for solving the puzzle attached in the screenshot)

[qu]aaevftmeimsckou

It will then store a file called 

[qu]aaevftmeimsckou.txt.

Here's the results of it running that:

teams
omits
ammos
steam
stick
stave
smoke
vitae
smite
items
meats

My dictionary is lacking words, it only has 81,536...

One thing to point out. Since Javascript does everything by reference, I have to clone the board for each level of recursion since they will mark letters as used and then future recursions will see them as used.  The clone board method clones the letter boxes and then correctly marks the currently used letters as used in the clone. This makes it correct.

The file uses a library I wrote for node.js to easily read / write files. You can download my [WordHero] solver here. You can fill in the missing fshelp methods with standard node file IO methods.


Enjoy!!