All Posts - Page 38 - Jason Connell

Node.js Keeps Me Sane June 26, 2013

Scenario: We're rewriting the code for a site but it's pretty much using the same javascript and css currently. So I have a copy of the old version running and a copy of the new version. And something is not perfect in the new site. A Javascript problem. To eliminate differences in javascript as a cause of the problem, here's what I did:

var http = require("http"),
	fs = require("fs"),
	SyncArray = require("syncarray").SyncArray,
	fshelp = require("filesystem");
require("strings");

var web1 = "http://localhost/";
var web2 = "http://oldsite.com/";

var name1 = "new";
var name2 = "old";


function processRequest(baseUrl, url, name, baseDir, finished){
	fshelp.ensureFolder(baseDir, function(){
		http.get(baseUrl + url, function(resp){
			var file = url.lastIndexOf("/") == url.length - 1 ? "_index.html" : url.substring(url.lastIndexOf("/"));
			resp.setEncoding("utf8");

			var data = "";
			resp.on("data", function(d){
				data += d;
			});

			resp.on("end", function(){
				fs.writeFile(baseDir + "\\" + file, data, "utf8", function(err){
					if (err) console.log("error encountered writing file - " + JSON.stringify(err));
					
					var files = findUrls(data);
					console.log(files.length);
					var sync = new SyncArray(files);
					sync.forEach(function(f, i, arr, finishedOne){
						console.log("processing " + f);
						processRequest(baseUrl, f, name, baseDir, function(){
							finishedOne();
						});
					}, function(){
						finished();
					})
				});

			});

		});
	});
}

function findUrls(content){
	var files = [];
	var reg = /(src|href)=["|']([a-zA-Z0-9\._\-\/]*?)["|']/g;

	var m = null;
	while ((m = reg.exec(content)) != null){
		if (m[2].trim().length > 0 && m[2].indexOf(".js") != -1){
			files.push(m[2]);
		}
	}
	return files;
}



processRequest(web1, "", name1, "c:\\webs\\" + name1, function(){
	console.log("Done " + web1);
});

processRequest(web2, "", name2, "c:\\webs\\" + name2, function(){
	console.log("Done " + web2);
});

After that was complete, I then used WinMerge. It would find all javascript files in the html of the home page (what we were having trouble with), and save them all to a folder. It turned out they were identical. But part of narrowing in on the problem is eliminating other possible causes. And I found the problem in the HTML, in where some JS files were being included. Some in head, some below. Putting them all below fixed everything.

I write Node.js in work too. June 21, 2013

Read my blog post on Node.js as a tool to get work done...

Credit Card Payment Plan June 14, 2013

I figured out this little trick. Now that banks offer automatic payments to credit cards, there's a pretty persistent way to make a dent.

Say your bill is $4,000 and your minimum payment is $150. Instead of paying $150 on the due date (a pretty sizable chunk to be missing during one pay period), do $75 every other week. Schedule it in a way that the 2nd payment is a day or two before the due date, so that they can't hit you with a late fee, as scumbag banks like to do.

My credit card bill just hit below $2,000. I've actually been paying $100 every two weeks, so I started it approximately 9 months ago. It's pretty nice. The problem is I have two other credit cards with outstanding balances!! I'm getting them down there using the same strategy. Give it a try, especially if you can auto-deduct it, then you don't even have to worry about remembering to do it!

I'm an IDIOT!! June 11, 2013

I spent a ton of time trying to write a syntax parser for a meta-language which would be parsed by Javascript. I just replaced a lot of confusing logic with about 40 lines of code.

First, Javascript is dynamic and has stuff built in that can evaluate arbitrary code passed in via a string.

I'm not talking about eval().

Function!

However, my syntax looks like this: Hello {{page.user == null ? 'Anonymous' : page.user.firstName }}

So, we need to know what "page" is in the context of the function. I still have to parse out top level variables. In the code above, it will get "page" as a top level variable.

Then, I build up the function:

var f = new Function(vars[0], body);

"body" is actually modified, I set it to "return " + body; So that I can do {{ page.user == null ? 'Anonymous' : page.user.firstName }} and it will return the display name instead of undefined, the default behavior of a void function.

I have to count up the number of variables used, and build the function accordingly. Currently, this is a switch statement.

switch (vars.length){

case 0: f = new Function(body); break;

case 1: f = new Function(vars[0], body); break;

case 2: f = new Function(vars[0], vars[1], body); break;

}

Luckily in my code, there aren't more than 3-4 "top level" variables, including globals like "Array" and "String".

Here's the variable parsing part:

var shared = require("./shared");
require("strings");

var constants = { "true": true, "false": true, "null": true };

var lookup = {};

this.getVariables = function(body){
	if (body in lookup)
		return lookup[body];

	var vars = [];
	var instr = false;
	var instrch = null;
	var buf = "";
	var toplevel = false;
	var result = null;

	for (var i = 0; i < body.length; i++){
		var ch = body.charAt(i);

		switch (ch){
			case "'": case "\"":
				instr = ch != instrch;
				instrch = instrch == null ? ch : (instr ? instrch : null);
				break;
		}

		if ((!instr && shared.tokenSeparator.test(ch)) || i == body.length-1){
			if (i == body.length-1) buf+= ch;
			if (!toplevel && (result = shared.variable.exec(buf)) != null && !(result[1] in constants)){
				if (!vars.some(function(d){ return d == result[1]})){
					vars.push(result[1]);
					toplevel = true;
				}
			}
			buf = "";
		}
		else if (instr) buf = "";
		else buf += ch;

		if (toplevel && (instr || (shared.tokenSeparator.test(ch) && ch != "." && ch != "]"))) toplevel = false;
	}

	lookup[body] = vars;
	return vars;
}

And here's the evaluation part:

var syntax = require("./syntax");
var shared = require("./shared");

var lookup = {};

var Evaluator = function(globals){
	this.globals = globals;
}

Evaluator.prototype.evaluate = function(body, context){
	body = shared.replaceComps(body);

	var vars = syntax.getVariables(body);
	var args = [];

	body = "return " + body;

	for (var i = 0; i < vars.length; i++){
		if (context && vars[i] in context)
			args.push(context[vars[i]]);
		else if (this.globals && vars[i] in this.globals)
			args.push(this.globals[vars[i]]);
	}

	if (body in lookup){
		return lookup[body].apply(null, args);
	}

	var f = null;
	switch (vars.length){
		case 0: f = new Function(body); break;
		case 1: f = new Function(vars[0], body); break;
		case 2: f = new Function(vars[0], vars[1], body); break;
		case 3: f = new Function(vars[0], vars[1], vars[2], body); break;
		case 4: f = new Function(vars[0], vars[1], vars[2], vars[3], body); break;
	}

	var result = null;
	if (f != null){
		result = f.apply(null, args);
		lookup[body] = f;
	} 

	return result;
}

this.Evaluator = Evaluator;

shared.js has a regular expression, a map (for old syntax considerations), and a function to replace some old ways of doing things with the new, pure Javascript way of doing it.

this.replCompRegex = / (eq|ne|gt|lt|gte|lte) /;
this.replCompMap = { eq: "==", ne: "!=", gt: ">", lt: "<", gte: ">=", lte: "<=" };

this.replaceComps = function(body){
	var res = null;

	while ((res = this.replCompRegex.exec(body)) != null){
		body = body.replace(res[1], this.replCompMap[res[1]]);
	}
	return body;
}

Downgrade to Node.js v0.8.23! June 10, 2013

Something's going on with 0.10.9 and 0.10.10, where it can't keep my server running. It appears to be related to streams or sockets. But 0.8.23 is working so I will just leave it be until the Node developers can get their acts together :)

Upgrade to Node.js 0.10.10! June 7, 2013

I know the previous post was my upgrade to 0.10.9. But then I started getting an EventEmitter error on Stream end. So I downgraded to 0.8.23 in the meantime, until it was fixed. But then I kept reading, and found out it was fixed in 0.10.10. So I did the git checkout v0.10.10 and make && make install && restart node webserver. After this, it was still getting the error but it wasn't crashing node.js. At this time I saw my error logs flood with file not found (404) errors, and the node.js process was at a consistent 65%. So, I figured it was working and I'll check again when I got home. I'm happy to report, it was just a DDOS attack or something, and the CPU is back down to 2% for Node.js like it should be.

Close call.

Node 0.10.9 June 4, 2013

Upgraded! I'm trying to get all of my computers up to it, so far it's just my mini (netbook) that I use on the couch for dev work, and the server. Left to go = home "real" laptop, and work machine.

Javascript Peculiarity May 30, 2013

I was trying to write a syntax parser, and everything was going just great until I came across this little tidbit. In parsing this syntax:

obj.prop1.prop2.substring(0,1) I'd parse and evaluate obj, then find properties in it like so:


   var currentContext = null;
   code.split(".");  
   for each (token in split string){
       if (currentContext == null) currentContext = evaluate(token);
       else if (token in currentContext) currentContext = currentContext[token];
   }

Evaluate executes code and returns objects within a context global to the method call, not the local context, which I call here, "currentContext"

But then I got an error trying to do stringProperty.substring(0,1). Can't call "IN" on a string? So I'm like, ok, let me try something else

if (token in Object.getPrototypeOf(currentContext)

And guess what... That shit don't work! Cannot call getPrototypeOf on a non-object. But it's a string?!

So I just ended up doing, if (token in String.prototype), just to say F YOU JAVASCRIPT!!! Otherwise I love Javascript. If you have any insight on this, PLEASE feel free to leave a comment.

I wrote up a JSFiddle for it, please link your own in the comments

This site now running on MongoDB 2.4.3 May 29, 2013

It does not take advantage of it in any way though... But I wanted to get it updated because a new site I'm writing takes advantage of some new features, like aggregate. Fun stuff!

MongoDB has an aggregate method May 28, 2013

And I'm writing a disc golf score keeping and statistics tracking application. I wanted to see what course a particular user played the most.

db.scores.aggregate([{$match: { user: userId } }, {$group: { _id: { course: "$course", variant: "$variant", tees: "$tees" }, played: { "$sum": 1 } } }]);

It shows that in the score keeping data, I've played the Blue Tees at the Front Nine at Sedgley Woods 6 times, and the Back Nine 5 times, since last year.

[

{"_id":{"course":"50acfe3b91b5581439a05ed2","variant":"back-9","tees":"blue"},"played":5},

{"_id":{"course":"50acfe3b91b5581439a05ed2","variant":"front-9","tees":"blue"},"played":6}

]

I will be able to use this to get a whole bunch of different statistics :) I will make an announcement when that goes live, I want to put it on the web, it'll be fun. It's written on my latest of the software I'm writing in Node.js.

This one is a C# post May 11, 2013

At work, I was working on cool stuff, but then my boss was like "I need this report and this report and this report. Thanks."

I'm not one to turn down such a politely worded and completely fictitious request. Reports are easy until the requests become stuff like "Include subtotal line for every Apple category and Orange category"

My data set was obviously not Apples and Oranges, but here's what I did to quickly and easily make subtotals for each of these

First, I made some C# Attributes, which are nice when you like to work in the meta.

public class MyReportItem {
  [SubtotalGroup(GroupName = "Fruit Type")]
  public string FruitType { get; set; }

  public string FruitName { get; set; }

  [SubtotalSum]
  public int Count { get; set; }

  [SubtotalAverage]
  public int SalesPerDay { get; set; }

  [SubtotalSummaryDesignator]
  public bool IsSubtotalLine { get; set; }

  [TotalDesignator]
  public bool IsTotalLine { get; set; }
}

Your SQL might look like this:

select FruitType, FruitName, StockQty, SalesPerDay from Fruits order by FruitType, FruitName

So your data looks like this

'Apple', 'Mcintosh', 12, 80
'Apple', 'Delicious Red', 22, 50
'Orange', 'Some Orange Name', 33, 90

The code I wrote allows that data to be quickly, easily, and automatically shown like this:

'Apple', 'Mcintosh', 12, 80
'Apple', 'Delicious Red', 22, 50
'Apple Subtotal', '', 34, 65
'Orange', 'Some Orange Name', 33, 90
'Orange Subtotal', '', 33, 90

Notice the "SalesPerDay" column has an average attribute on it, not a sum. Here's the meat of my code, after getting the attributes and the data all figured out.

	public List<T> PopulateSubtotalItems()
{
	List<T> withSubs = new List<T>();
	if (this.list.Count == 0) return withSubs;

	// allow multiple group by with subtotals. e.g. group by Fruit Name and say fruit type, like "Citrus"
	// to subtotal Oranges and subtotal Limes and then subtotal Citrus
	List<GroupSub<T>> subs = new List<GroupSub<T>>();  

	foreach (string key in this.groupBy.Keys)
	{
		T sub = new T();
		GroupSub<T> groupSub = this.groupBy[key];
		groupSub.SubRecord = sub;
		// sets the properties which designate the group. So this subgroup might set FruitType to "Apple"
		this.SetGroup(groupSub, this.list[0]);
		// sets the bool property which the subtotal designator is on to true.
		this.SetSummary(groupSub);  

		subs.Add(groupSub);
	}

	// if there's a bool property with the "TotalDesignator" attribute, include total
	GroupSub<T> totals = null;  
	if (this.includeTotal)
	{
		T sub = new T();
		totals = new GroupSub<T>();
		totals.SubRecord = sub;
		totals.IsTotal = true;
		this.SetTotal(totals);  // sets the property which the TotalDesignator is on to true
	}

	subs = subs.OrderBy(grp => grp.Sequence).ToList();

	int grpCount = 0;

	for (int i = 0; i < this.list.Count; i++)
	{
		bool added = false, last = i == this.list.Count - 1;
		foreach (GroupSub<T> grp in subs)
		{
			bool same = SameGroup(grp, this.list[i]);
			if (!same)
			{
				this.Average(grp, grpCount); // set the average properties to the sum / grpCount

				withSubs.Add(grp.SubRecord); // add the subtotal record to the group

				grpCount = 0;  // start afresh
				grp.SubRecord = new T();
				this.SetSummary(grp);
				SetGroup(grp, this.list[i]);
			}

			Increment(grp, this.list[i]);

			if (last) // special handling on the last one.
			{
				this.Average(grp, grpCount);
				if (!added)
				{
					Increment(totals, this.list[i]);
					withSubs.Add(this.list[i]);
					added = true;
				}
				withSubs.Add(grp.SubRecord);
				grpCount++;
			}
		}

		if (!added)
		{
			Increment(totals, this.list[i]);
			withSubs.Add(this.list[i]);
			added = true;
			grpCount++;
		}
	}

	// add the total line
	if (this.includeTotal)
	{
		this.Average(totals, this.list.Count); // average the total record
		withSubs.Add(totals.SubRecord);
	}

	return withSubs; // that's it!!
}

As you can see, I no longer have to dread doing subtotals on reports!

My Daughter is 11 Months Old April 2, 2013

And can't stand the sight of blocks stacked up.

Anti Code Generation March 19, 2013

I've been anti code generation ever since always. I have some good reasoning behind it. Recently, though, the company I work for has inherited code from another firm (always bad in my experience), and a lot of it was generated from an internal tool, and it made me think about why I have been against it, and made me more firm in my stance.

We actually use code generation for some projects, but we're smart / not wasteful about it.

Here's my selling point. If one becomes dependent on code generation, their data architecture can suffer grave consequences, in being inefficient, and generally not thought out. They will create any old architecture because it is of no cost to them, in terms of time. If you've been doing code generation against data structures, you have no reason to design an efficient one, reuse concepts across the entire structure, optimize for efficiency, and you might be stuck with certain data types and assumptions about past data that your code generates to, that you might even get stuck.

Some explanation. CRUD operations are generally easy to generate. You generate an insert stored procedure, an update (or combine them), a select and a delete. This is for one object in your database. So you have a customer object, with an address foreign key, now you have to get the Customer->AddressID and then do another call to get the Address data. But sometimes you don't want customer and address at the same time. So now you have two stored procedures. When I wrote an ORM, this was one thing I nipped in the bud. It would do the join and get all data for customer and address in one call, if you passed in "load references = true" to the load method.

A bit around the not-thought-out part... I take great care in designing a data architecture, and I'm just a lowly software developer! I kid, of course, the data structure is at the crux of what I do. If I have a shitty data structure, I can't work with it. One recent example was an application of sorts, and a wizard with steps 1-4. It was inherited code and data structure which we couldn't change. The main application part with the customer data didn't have a date field for created or updated date time. Those were stored in a log table. So to get the time that the application was created, you had to do this:

Select app.* from app join app_log on app.id = app_log.app_id where app_log.date = (select min(date) from app_log where app_id = app.id) order by app_log.date desc

That inner select is a killer. We'll hopefully get both a createDate and updateDate fields on the app table in the near future. This actually wasn't a product of code generation but more of an example of the bad code that we inherit.

More of the not thought out stuff, since that last one wasn't to do with code gen... Sometimes there will be repeated fields or concepts. Some people get an Address table and if it's going to be a slightly different address, add another address table with all of the fields from address, and then other fields that they needed. Hey, it's simple to just generate the new code!! But the structure is repeated. I will try my damnedest to not repeat code or a data structure. If something has an int and a string, and another thing has an int and a string, they will both inherit from a base class that has an int and a string. Goddamnit!

Copying and pasting code is worse, but we're not talking about that.

Moving on to the "you might get stuck with certain data types and assumptions" part of my thesis. By assumptions, I mean, assumptions that were made when the code gen tool was written. For instance, the ID field must be an int. -1 means a null int. A lookup type table (basically an enumeration) must have display value and an internal value (usually matching up with a code enumeration) and must have int ids. Our old code generation generated stored procedures, if a parameter was a bit field, 0 meant "don't care" where 1 meant "where this field == 1". You couldn't filter based on where that value was 0. If you wanted to say 0 was the more significant one, you would name the column in a negative way. For instance, a "Deleted" field, where 0 means not deleted. You couldn't get just not deleted records. So you would have to name the column "NotDeleted", which is crap.

I was recently looking at my old code from college, it's great :) I remember my professors and how they molded me into the programmer I am today. Then all the many many hours I spent honing my skills. I wanted REUSABLE code. I never wrote a code generator for personal use. I have modified the one we use at work to be better and more acceptable, according to my standards. I'm tired and I'm going to bed...

Baby Contests March 4, 2013

I'm sorry, random stranger on Facebook, if your baby is in a contest with my baby, I will refer to your baby as "The Ugly One".
Jason Connell (@jasontconnell) March 4, 2013

Contact Lenses January 10, 2013

I can see!! Yesterday I went to the eye doctor. I went to LensCrafters since it was easy and they let me make an appointment online. The people there were friendly and funny, the doctor was young and had a good attitude, and was fun.

They showed me how to put the contacts in, and without thinking that they, at some point, would have to come out again, thought "This is so simple, why didn't I do this years ago?!". Then they had to come out! I got through it in maybe 10-15 minutes. It was great help when someone, either the doctor or the assistant, would come in to see how I was doing / offer some advice. I got dilated, got the exams, put my contacts in for the second time (much easier this time), and drove home.

I was put on a schedule, to wear them for 4 hours the first day. I put them back in around 6:10, so around 9:40 I headed upstairs to begin the arduous process of taking them out. It took me 25 minutes!! This morning I woke up feeling like I had crabs crawling around in my eyeballs!

Anyway, I put them in again this morning, and it's amazing seeing everything with clarity. I can feel them in there, but that goes away and you get used to it. Today I'm supposed to only wear them for 6 hours, and I've passed that point. I might take them out here at work...

It is great for my photography!! I can focus! I do manual focus, manual everything actually. This will lead to a higher percentage of non-blurred pictures!!

I got glasses around age 12, but in the end I only wore them at night when driving. I lost them at some point so for a few years I haven't had anything to help me see. So I got used to blur. Hence why it's amazing that I can see again :)