Thursday, 7 November 2013

Footnote Summit 2013 (#footnotesummit)

Well, the FootnoteSummit has come and gone...


Yesterday, I had the good fortune to be one of those able to attend the FootNote summit 2013, my company sponsored me, of course, just before tickets ran out.

Those who had the good fortune to attend, were wowed with statistics and information regarding Digital Publishing both nationally and internationally. We were treated to an awesome lunch, a "free" touchscreen stylus, a "free" 3 month subscription to Getaway Magazine ... a "free" lunch, and an awesome opportunity to make friends.

Friends were made...

I had the rare opportunity to make friends with another "techie" named Arthur Atwell. We chatted a bit about his Paperight business, and I was able to shed some light for him on how to make EPUB3 documents... I have only recently discovered that it's possible to take an EPUB2 file and massage it into a valid EPUB3 file by a tedious process of manual editing... This is a post for another day though.

Cards were exchanged

If any of you have watched a romantic comedy (or romcom) as it is widely known, you will undoubtedly remember a scene of some young woman exchanging numbers with some your guy ... much to the guy's excitement. That is kind of what happened at the FootnoteSummit, but in this case everyone was exchanging business cards... I swear, I have never seen so many people swapping business cards at one time in my life...

Sadly the fun was over

At the end of the day, we had been wooed and wowed with presentation after fantastic presentation. There were global representatives from all facets of the digital publishing industry present to knock our socks off with demos of awesome new technologies...

But all the fun had to come to an end as the day closed at around 15:30, just enough time for us to beat the traffic in C.T 

On the plus side, I was pleased to discover that in South Africa, I am one of the few people who understands how to take an existing EPUB2 document and edit that file so that we end up with a valid EPUB3 eBook, it's quite astounding to know that most people don't understand how to go from version 2 to version 3 without paying exorbitant fees to have the book redone...

Well, I enjoyed myself and learned a lot, le't hope that we are able to attend the summit next year too.

Wednesday, 23 October 2013

So, after writing my several thousand lines of Python....

After I have written and DEMO'd the code...

I finally finished writing the several thousand lines of Python code, interspersed with HTML and CSS .. and provided my bosses with a very nice DEMO of what the app does... needless to say, the app was running on our intranet - off my machine, which was partially converted into a Server Node on out intranet...

The bosses loved the app, they loved it so much, in fact, that they wanted it "out there" for the world to use!

Writing a web app in PHP is hard...

So, after writing all that awesome code in Python, it was time to start hunting for a new web provider which allowed Python 3 code ... none were forthcoming, and my bosses didn't like the idea of moving our website to a different provider ... so it was back to the drawing board ... to rewrite the entire thing in PHP ...

Well, PHP is not that hard ... when you know it...

In the defense of PHP, it's not really a hard language to master ... sure, there are a lot of functions, but nothing that is too insane to handle .. mostly, it's quite easy to learn and master ... but to write several thousand lines of code is another thing entirely.

I started by learning the basics - how to get it running on my Apache installation, how to "secure" the feature set... how to write the actual code, then it was time to slice the workload into pieces...

Understanding the concept of PHP is not too bad...

It took me a while to get my head into the concept of PHP - calling functions right inside the HTML code, functions which return more HTML ... with Python, you formatted the code as you went ... with PHP you formatted your page, and plugged in the function calls you needed at the end ... 

Once I got this distinction into my head, it was easy to implement the functionality needed to get my code running the way I wanted it to...

The most useful function in PHP

The most useful function I have come across in PHP is the "include" statement... not only does it give you the option of including HTML source, it allows you to use one "header", "footer" or source file all over your site ... this has by far saved me the most time. Of course, my Python was as modular as I could make it, but the idea of including a "header" file, or a "common html" file eluded me because it required me to write code to get it done ... in PHP this is already done for you so when you become aware of this feature, you start using it as second nature.

In conclusion...

In conclusion, I have to say that PHP is awesome, it simplifies several issues I have been having with Python, and it is blazingly fast ... Python is also pretty nifty when used right, but it costs you more complex code and several optimizations to make it as fast as possible ... in PHP one can write lazy code and still get away with a huge speed benefit. I must admit, that the learning curve is not as simple as I make it sound, but I can vouch for how much simpler it is to learn PHP than it is to learn Python ... probably because the syntax is so similar to Javascript and C++ ... 

All in all, I have already ported 90% of the project in about a month ... and the original App took me three months to write. And, this time I'm not dedicating nearly as much "zone time" to it as I should.

Tuesday, 10 September 2013

So, python web apps are not that hard after all

So, I was saying earlier that writing web apps with Python is hard ...


I was sorely mistaken... Obviously, I am not using any frameworks to facilitate my project - which is a huge mistake, but a learning exercise for me (I'm  fairly new to python). When not using a framework, you are exposed to the internals of how CGI and Python all meld together, as well as how to use the FieldStorage object to respond to Ajax requests.

It's only hard if you do it the "wrong" way

Writing a web app with Python 3 is only hard if you do it the wrong way. I come from a background of coding C++ code using the most bleeding edge coding techniques - OOP, TDD, Templates, Multiple inheritance, Polymorphism, insert bleeding edge coding technique here, etc.

Now, this is all good and well when using a language like C++ where you only face the OS endpoints (Windows API, stdlib etc) at certain points in your code. But these techniques do not work too well when using a high level scripting language like Python. I started off by churning out dozens of classes which has get and set methods for any data members. I also tried to abstract away the basic Python functionality by writing classes which dealt with IO and repeated function calls...in short I ended up trying to write a framework on top of Python. This is all good, as awesome Python frameworks exist, but it was detracting from my main purpose - to get on with writing the app.

It gets even harder when you just write what you need

By this I mean that I stopped writing my framework when I realized my folly, but then had to start over by only writing the code I needed. This is the hardest part of coding - writing the least amount of code to accomplish the highest level of functionality. This is the zen of coding, and it took me a while to find my sweet spot.

Of course, you need Database wrappers, you don't want to instantiate a SQLite3 console in the middle of iterating through your FieldStorage object to find URL parameters ... but you do want to have a handy function on hand so that you can take those parameters and do something useful during that loop.

Getting it "just right"

I finally hit my stride when I had my fancy UML diagram all spec'd out and I was able to write only the classes I needed. Those classes, in turn, contained only the functions I needed, and after ironing out some kinks in my way of thinking about Python classes, I was able to bang out a decent web app in about 80 hours. Of course, I did not have the luxury of spending days on end adding classes all willy nilly, I had to squeeze in odd hours here and there to get it done.

The main thing I learned is that Python is awesomely powerful when it comes to manipulating data. Python 3 comes with a slew of features which allows one to have a working Ajax request handler in about 20 lines of code. I also learned that the true strength of Python shows through when you have finished your self-inflicted torture ie: "learning curve" and simply get on with the business of writing your app.

Python frameworks are there to abstract away the drudgery of having to code a "find key in dataset" algorithm for the 20th time, they save you from yourself when you do not know enough about the language to step around common pitfalls.

The current trend is micro-frameworks

Todays developers have gotten slightly lazy in that they all rely on some framework. These days micro-frameworks are all the rage, and developers become burnt out after spending maybe an hour trying to digest the documentation for the latest and greatest framework. I prefer to stay away from all that because these frameworks fall into disuse and are forgotten and if you have gone 98% of the way into building a business using one of these forgotten frameworks then you either have to hire rockstar developers to save you from the framework, or you have to add improvements and workarounds for the last version of the framework on the planet.

These options are not really too usable for startups, and I can see why many startups bank on either technologies which have already found a firm footing, or on creating their own frameworks and platforms on which they can build their latest and greatest product.

I'm not saying it's wrong to use a new micro-framework, just that you should really weigh your options when deciding on your development strategy. I spent about 4 hours searching the net and becoming aware of the "abandonment trend", which causes developers to furiously write code using a new library, only to become frustrated when something doesn't work the way they guessed ... and the documentation has not quite caught up to all the new features - developers write code, great developers write code which documents itself, the greatest developers don't write code - they write documentation which executes at runtime. Chew on that for a second.

Optional ranting

During my foray into writing my first web app, I discovered that many would-be Pythonistas are doing "the right thing" by writing everything themselves - excluding the kitchen sink. During their development they hit inevitable snags which could be solved by adding something to the language itself - like a way to say FieldStorage.findkey('key') ... which may or may not actually exist. But, the great part of these flaws is that patterns emerge, tried and tested functions become public knowledge after a developer bangs his head on his desk while solving a "simple" problem.

I ended up needing to send emails to users of my web app, I wanted a way to send the email, store it in the database, and store the resulting message off-site ... I spent a few hours trolling messageboards to find a solution when it suddenly hit me, like a slice of lemon wrapped around a brick - use gmail, which promptly led me down the rabbit hole in search of a way to send email... About an hour later I hit upon this little script, which must be the simplest way to send email which I have found: http://kutuma.blogspot.com/2007/08/sending-emails-via-gmail-with-python.html

If you have copied the script and are now frantically struggling to get the interpreter to recognize your efforts, bear in mind that things have been reorganised in Python 3, by replacing the import statements with these statements, you should be able to get up and running in no time:
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email.mime.text import MIMEText
from email import encoders
That little slice of code should save you lots of time when working with Python 3 and struggling to understand why seemingly innocuous import statements don't want to work..

Well, that's enough of my ranting, my web app should go live within the next month or so - at which time I will make it available for public use. Until then, happy coding... 

Tuesday, 30 July 2013

Writing a home-monitoring system using python + internet...

Hmmm, what's that, you say? A Python home monitoring system? Why would anyone want to subject themselves to that kind of abuse?

Well, I , for one, amd lazy ... it shows in my work, it shows in my lifestyle, it shows in my weight .. and it shows in the way I write code. I'm the guy that will spend 2 days writing an app (which is available for purchase on the Google App Store) because I'm too lazy to get up and get myself a credit card ... because I'm too lazy to get up and scratch in the drawer for my old credit card so I can purchase stuff from Google Play ...

Before you go ranting that I expend time and effort to write an app which costs $3 ... you must remember that I am a programmer .. code it like water to me - extremely bitter, and mostly salty water, but water nonetheless ...

I eat code, I think code, I dream code and I can churn out code if I want to because it has become second nature to me. So it's simpler for me to just write it myself than it is to actually expend some physical effort to go and find something ... you cannot grep in the real world. You cannot hit [Ctrl][F] after opening your drawer. Frankly, I should patent some kind of "Reality Search" function which would apply itself to real-world objects ... but I'm too lazy to go down to the patent office to get it filed in person... Even if I could file patents online in my country, I would be too lazy to get up off my ass to go and use my new function...

Which brings me to the crux of my post ... why would I use python for such a serious task? What I am trying to do is definitely non-trivial, but surely PYTHON?

Well, the truth is that I am too lazy to go back to coding things in C++, I am not an Environmentalist, nor am I a packer ... I am more of a "mapper" who likes to interpret and adapt information into some kind of "view" which can be applied to knowledge ... and my "view" of the matter is that if I need the blazing speed to shoot myself in the foot a trillion times while simultaneously being bitten into millimeter sized pieces, then C++ is the way to go for every project.

If I were to use C++, then I would have to either write little command-line programs which did something specific with some command line input ... and tie that into my server (I'm using Apache - it's easy to set up, and doesn't require any movement on my part) ... or I could churn out some high-level code using a high-level "interpreted", "dynamically typed" language like Python... which saves on compile time, debug time, the amount of keystrokes I have to bang out ... it saves on heart rate as there is less agitation, and it saves on breathing since there is less cursing...

It's kind of a simple choice, really. For anything web-related, use web technologies ... for industrial "blazing speed", specialized applications, then get closer to the hardware with C or C++ ... For the record, I prefer the straightforward nature of C (I have spent several years coding industrial apps using C - if it's not straightforward, then you misunderstood something) but I love the speed you get when you can bang out a couple of object, an interface between them and then set them loose on the world of C++ ... so it's a trade off really. You just need to know which hammer will beat which nail, then all should be fine.

So back to the app - it's a very simplistic application:


  • Set up a "service" running in apache which allows an authenticated user to receive a screenshot containing all visible web cams on the home network.
  • Send these screenshots at some interval to the authenticated user.

This is fairly simplistic - since I will not even get into the hair raising mess of authenticating a user...
The simplicity is in how one would collate the images and send them over the network ... it's fairly straightforward to get the images - leave the DVR software running all the time, and periodically take a screenshot using this function:


import win32gui
import win32ui

def screenshot(_filename):
  wDC = win32gui.GetWindowDC(HWND_DESKTOP)  #hwnd
  _w = win32ui.GetSystemMetrics(SM_CXSCREEN)
  _h = win32ui.GetSystemMetrics(SM_CYSCREEN)
  dcObj=win32ui.CreateDCFromHandle(wDC)
  cDC=dcObj.CreateCompatibleDC()
  dataBitMap = win32ui.CreateBitmap()
  dataBitMap.CreateCompatibleBitmap(dcObj, _w, _h)
  cDC.SelectObject(dataBitMap)
  cDC.BitBlt((0,0),(_w, _h) , dcObj, (0,0), win32con.SRCCOPY)
  dataBitMap.SaveBitmapFile(cDC, _filename)

  cDC.DeleteDC()
  dcObj.ReleaseDC()
  win32gui.ReleaseDC(HWND_DESKTOP, wDC)

This would give me some image with a predetermined name which I could look up on the drive when I needed it ... I would probably store the images in a folder which represents the images from a specific cam on a specific day ...

Loading up the image is easy ... displaying it to the user is equally easy ... write a little JS function which requests an updated image via Ajax at some interval... Embed an <Object> into the HTML, tie it to the JS and VIOLA! You have a bandwidth-friendly solution to a home-surveillance problem...

This is obviously a gross oversimplification of the whole app, but it would definitely work along these lines ... I would obviously write some code (using PIX or Pillow) to convert the images to JPEGS on the fly either before saving, or at request time ... but the basis is the same. Obviously I would have to port forward on my router to allow that to happen ... but luckily DynDNS has made that much simpler, and all the grunt work of getting it set up , and obtaining a DOMAIN with a dynamic IP has been taken care of .. I have even assigned a static IP to my "surveillance" box which runs the apache server .. so most of the setup work is done .. all I need to do is to tie in some way of getting screenshots at a specified interval - which is equally easy ... write a "backdoor" into the system whereby I can leave a browser window open on the same machine running some periodic Ajax function which would force the server to respond at some predetermined interval - when I get a specific Ajax request, then do the hard work ... very hacky, but sufficient for my purposes - I could hack this together in a day. Now, imagine doing all of that, AND writing the C/C++ code to back it up? Can you imagine the pain and agony and suffering you would go through simply debugging why X won't initialize before Y?

If I wanted ultra-secure 128bit encryption and blazing fast real-time RTSP streaming web app, then I would be forced to resort to a C/C++ HTTP server which relies on the live555 media streaming library somewhere... and then write a C/C++/wxWidgets-backed client side which decodes and displays using FFMPEG... but, thankfully, I don't need to go that route just yet :-)

Thanks to pyfunc over at StackOverflow for the screenshot function, which I am generally going to plunder from the SO site - as much as I plunder many lines of code from awesome coders who are not as lazy as I am.





On working with Python and Javascript .. and HTML and Ajax and SQLite

I have recently been creating a production management system. The system is not that complex, but it is taking an inordinate of time to write ... even though I am using rapid-prototyping languages - Python, SQLite3, HTML4/5, Javascript coupled with liberal Ajax sprinkles ...

I simply don't understand - but I will try to get to the root cause.

So far, I have constructed the tables and written a database backend wrapper - for the SQLITE3 DB which I manipulate on the server side using python cgi.

I have created the concept of "sessions", but not in the MOD_PYTHON way, no, in the more traditional - hacky, store-it-in-the-window_name way ... By allowing the user to send an Ajaxlogin request, which generates a session token on the server side, which is then "kept" or "remembered" on the client side by storing the token in the window name browser object variable... It's probably the most rubbish way I can do it, but it works - survives page reloads (F5), survives navigation buttons (back, forward) ... and means I end up writing less python code to deal with IP-address-based-sessions since our network is mostly DHCP based, and we frequently restart our routers in any case due to some other technical difficulty we have to live with due to a third party.

I have added support for individual "users" and am in the process of linking "projects" to users in a nice way.

I have created an HTMLFormatter python script which can generate nicely formatted links which call JavaScript functions ...

I have written JavaScript functions which append the session token to outgoing requests ...

I have written more python cgi code which checks for session tokens and discovers which user "owns" the token  and does something fancy with that ...


I think I'm starting to notice a trend here ...

Medium/Large frameworks hide all this detail from the programmer - allowing them to deal with the core of their code - the business logic. When writing something from scratch, we tend to spend an inordinate amount of time getting a trivial feature to work "just right", only to blow another week on it when we discover it bombs out in IE...

When you are a hobbyist programmer, it makes sense to write as much as you can on your own so that you can both sharpen skill and learn a diverse set of  code "katas". But when you are an experienced programmer who is simply trying to get a project "done", you don't want to spend 2 days learning how to handle sessions which do not use cookies ... you want to spend that time banging out the code that ties the project to the user and allows the bosses to draw up time sheets and plot graphs and all those other things that bosses do with production management systems....

Even for hobbyist programmers I would recommend using frameworks for things you are already comfortable with - know how to implement sessions already? Use MOD_PYTHON for the next project which requires sessions - you know how it works, now use some code which saves you from 3 weeks of implementing it yourself.

Keep it simple ... but simple is a very relative term - upon embarking on this project, my idea of simple was to have a simple "session" - based on the user IP, which displayed their own project queue to each user, and a more "admin" interface to the sysadmin. I spent about a day banging out some nice python/html/JS/Ajax code to get it all working nicely and discovered that my idea fell flat when the DHCP server (our router) was rebooted during a session ... so I spent some more time tweaking the idea to use ip +  machine name ... but that quickly fell flat when I discovered that users on our network didn't have unique machine names ... and I wan't up to the task of renaming each machine to make my code work ... which is when I stumbled on the "shared login" concept - log a request to the server to get a session token, then associate the token to a user id on server side, and store only the session token on the client side - that way we don't care if the transport mechanism goes offline between requests - we simply send off each request with the session token and let the server handle the rest...

This was by far the most robust approach I have found so far - now if only I could get it to work if the user suddenly starts working in a different tab .. hmmm ...

Well, the only thing left to say is that python/JS/HTML+Ajax are "rapid prototyping" tools for a simple reason - somebody has already built the prototype, whether is is a python recipe or a JS library, it has been built - if you are going to keep inventing the wheel it will be slow in any language ... and will probably end up gathering many patches along the way, until it ends up like my project - which now consists of more patch than actual wheel ....

Thursday, 23 May 2013

Writing a Windows Gadget is hard!

Windows Gadgets suck ... Microsoft has recently discontinued support for Windows Gadgets - in favour of Apps. So any Gadgets fans/freaks out there who might enjoy the challenge of writing a gadget ... like myself ... are out in the cold when it comes to sharing gadgets, or hoping for support on some Sidebar issues ...

Well, it's not like they dismantled Windows Sidebar in Windows 7 ... but it looks like there is no gadget support in Windows 8 ... which kind of sucks because gadgets filled the instinctive need of many computer users to clutter up their desktop with cool techno-looking bars and charts and dials and switches etc...


So, Microsoft, in their infinite wisdom, decided to integrate most of the popular gadget functionality into the actual OS, instead of these little unrelated boxes floating around causing havoc with system resources...

Well, disregarding everything I just wrote ... I decided to write my own gadget - it's still a work in progress, so don't laugh too hard... but I started by studying the gadget format - not too tough to master, basically a folder structure with some XML defining the main files...

Then there is the gadget syntax - specialized html tags to identify elements of the gadget .. not too bad...
Then there is the compulsory JavaScript - luckily I was already learning that... so not too bad there...

Then there is the docking functionality... let me start by telling you that all the examples on the internet have one or other subtle flaw which my SideBar doesn't like ... while debugging the docking code I suddenly "lost" my gadget and it would not appear in the SideBar anymore .. well, ok, I mistyped something - but it's still frustrating to not have anything to debug with besides the tools which come with Chrome ... Oh, and did I mention that gadgets rely on the ActiveX components built into Explorer?

That means that you can debug MOST of your gadget and Javascript code using Chrome ... but for all those Microsoft-specific things there is no actual way to DEBUG a gadget using Internet Explorer ... which is a shame because I rather enjoy playing with my TODOList gadget ... it's very simple, right now it can be docked - which shows a smaller list or undocked - which shows the same list in a larger view ...  fun times :-)
My main plan is to refine the todo list to the point where I enjoy using it - and it will be my default app, since I am a programmer and my mind is inordinately full and prone to dropping details after all these years of juggling crazy amounts of things around in short term memory - plus there are a lot of distractions all around me, since I am the only programmer IN THE COMPANY... which is kind of awesome, The Lone Coder lol!

Once the gadget has been refined to my liking, I would like to use it as a base for a gadget game I would like to write ... something like the old NES Star Force ... which is another mission on its own - but well worth the fun...

So, gadgets suck, but the joy they bring definitely makes it worth the time and effort that goes into creating one...

Building web apps with Python

Well, as much as coding sucks, I love it -so much in fact, that I spend an inordinate amount of time doing crazy things using the incorrect tools for the job at hand - just for the kick of getting it to work!

Well, recently, I have been prone to the urge to use different languages to augment my website - for those of you who don't know, I am a software developer, who does web development in his spare time... Actually, my new job involves copious amounts of web development - but that is another story.

Well, my current occupation allows me to do web development, and throw in "hardcore" programming languages into the mix - just for fun.

So, one day I decided that our company intranet needed a way of sharing files - not a Samba share (Shared folder on Windows), not an FTP server - because FTP causes too much back and forth network traffic.... and certainly not using anything "standard" like PHP or JavaScript!

No, I opted to use Python... So, I set up Apache, and got cracking ... mind you, it was not as simple as I thought it would be, but as it turns out, Python is quite capable of this task- with relatively little code ... It's quite simple, you edit your httpd.conf file to include python files to run as cgi scripts, which can then run in any folder... then you get cracking writing your python script...

Of course, you would need a fixed IP for the server to be of any use, but it is rewarding enough to crank up your browser and see a web page that your Python code spat out...

 I have included a screenshot of the web app running - along with the tools which were created with it:

*note that the image was doctored slightly to remove company information ... but the layout and look and feel are 100% correct. The left side shows the home page which lists all the services this app provides, and the right page is the "download" service running in my chrome browser window.

Well, let me tell you that Python is quite a dream for writing web apps - especially to a C/C++ programmer who is used to banging out 1-3000 lines of code before morning coffee... This entire web app, with all associated services comes in at under 3000 lines of code - without refactoring to remove common code!

Anyone interested in building web apps with Python should be aware that there are numerous security risks which Python is addressing as I type this. In the near future we might see python becoming the dominant web 2.0 language of choice for building awesome web apps.

Have fun ... my next project is to write a Windows Gadget Game using JavaScript and some clever DOM tricks I picked up along the way...

Wednesday, 22 May 2013

Coding Sucks ... I Love It!

Well, I have been coding since 2002 ... not too long, yes, but in that time empires have risen and fallen, billions of lines of code have been written, and I have learned one thing:

Coding Sucks ... I Love It!

Seriously, I can't get enough of it... I have written projects in Java, Python, C, C++, PASCAL, assembly (win32) ... most recently JavaScript ... each language (or scripting language) has it's dizzying highs and abysmal lows, but one thing is consistent:

They all suck

Yes, that is a cruel statement, but let's face it - all languages suck in one way or the other, templates in C++, no-classes in C, the ease with which Python allows you to destroy your project ... the inane simplicity of PASCAL (and lack of true graphics support + hardware accel.) ... There are pitfalls to any language - and yet I have coded in many of them, for one simple reason....

The feeling of triumph when I accomplish a milestone...

Or better yet - that feeling of relief when I complete a project.

So, I might rant, and I might rave, but I have been coding for a long time - several of those years was as a Senior Developer at a software production firm, and I loved every second of the code - not the politics...

I am a programmer, and this is my blog.